Elastic Agents

Extension Information


Availability	GoCD version 19.3.0 onwards
Extension Name	`elastic-agent`
Extension API Version	`5.0`

Background

In the current GoCD setup, all agents are fully provisioned and always available waiting for a job. When there are no jobs to be executed in that environment or for that specific resource that the agents have, the agents stay in an idle state waiting for jobs to be assigned to it, which is a waste of infrastructure resources. It could also lead to high costs if agents are running on cloud providers.

Elastic agents are on-demand agents which are created and provisioned by an elastic-agent plugin when there are jobs to be executed, and terminated when the agents are running idle. These agents can be in a data center or in the cloud or both, and may be physical or virtual.

Developers can start building their own elastic-agent plugins by forking the skeleton plugin and looking at a sample docker plugin as an example reference implementation.

How will it help you?

A feature like this can allow for more efficient use of agent machines, can allow flexible scaling and in many cases, can reduce the cost of running agents. Imagine an automated performance test which runs occasionally and needs a lot of machines. These machines can be started at the beginning of the performance test, possibly using some cloud service, and then brought down when not needed. This feature should enable a more flexible and dynamic build grid.

Current process of handling agents

Agents are currently started manually or automatically by scripts written by users. They are registered to the GoCD server manually, on the agents page or automatically, using auto-registration.

Idle agents, after registration with the server, continuously poll the server, asking for jobs to run. When a stage is triggered, the jobs that need to be started are identified by the server. Suitable jobs are assigned to agents (depending on resources and environments). The agents pick up the jobs and run them. Once finished, they start polling the server for the next job to run.

When a job is scheduled, it is considered for assignment to an agent, and it waits in the queue for an idle agent to ping the server.
When an idle agent pings the server for work, assuming that resources and environments match, the agent is assigned the scheduled and waiting job.
Once the agent finishes running that job, it goes back into the idle agent pool, waiting for another, suitable job to be scheduled and assigned to it.

Watch the video linked below to help make this a little more clear.

Process of handling elastic agents

Here is an example cluster profile, elastic profile configuration, and a Job configured to use the profile for the docker plugin

<server>
...
</server>
<elastic>
  <clusterProfiles>
    <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
      <property>
        <key>DockerURI</key>
        <value>https://docker-uri/</value>
      </property>
    </clusterProfile>
  </clusterProfiles>
  <agentProfiles>
    <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
      <property>
        <!--
          The plugin currently only supports the `Image` property,
          which allows you to select the docker image that the build should run with
        -->
        <key>Image</key>
        <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
      </property>
    </agentProfile>
  </agentProfiles>
</elastic>
...
<job name="defaultJob" elasticProfileId="docker.unit-test">
  <tasks>
    <exec command="mvn" args="clean test" />
  </tasks>
  <!-- Specify the id of the plugin that should manage elastic agents for this job -->
</job>

Cluster profile represents the cluster where a user wants to start an elastic agent. An elastic agent profile represents the configuration to be used to create an elastic-agent instance. Depending on the type of the plugin and the capabilities it provides, a profile configuration could contain the AMI ID, or docker image name, size of the machine, volumes to be mounted etc.

The plugin decides whether to assign an agent to a job, or to start another agent, when consulted by Go.

When a job is scheduled, if its job configuration contains a elasticProfileId, the job is not considered for normal assignment as described in the previous section.
The plugin corresponding for that job elasticProfileId is contacted (to create an agent), along with the corresponding profile configuration for which this assignment needs to be made.
At this point the plugin may, at its discretion, create an agent or not.
1. If it sees that there are no idle agents that satisfy the profile, it may choose to spin a new agent.
2. If it sees an existing idle agent that satisfies the profile, the plugin may choose to not spin a new agent.
3. If there is no more capacity available to create new agents, the plugin may choose to not spin a new agent. (This is same as point above).
In case the job is not assigned after a specified interval, the server waits some time, and then comes back to the plugin, asking it to choose, again.
When an elastic-agent pings the server for work, and if there is a job that requires an elastic-agent, then the corresponding plugin for that job is contacted (to check if the server should assign work to the agent), along with the profile configuration for which this assignment needs to be made.

Job Configuration Agent Configuration Should assign work?

Lax matching of agents to jobs
Job Configuration	Agent Configuration	Should assign work?
`<profile id="aws.small" pluginId="aws"> <property> <key>instance-type</key> <value>m1.small</value> </property> </profile> <job name="unit-tests" elasticProfileId="aws.small"> </job>`	`<profile id="aws.large" pluginId="aws"> <property> <key>instance-type</key> <value>m1.xlarge</value> </property> </profile> <!-- Agent for profile "aws.large" --> <agent elasticPluginId="aws"/>`	Yes Since the job requires a smaller agent, and a larger agent is available.
`<profile id="aws.small" pluginId="aws"> <property> <key>instance-type</key> <value>m1.small</value> </property> </profile> ... <job name="unit-tests" elasticProfileId="aws.small"> </job>`	`<profile id="aws.small-us-east" pluginId="aws"> <property> <key>aws-region</key> <value>us-east-1</value> </property> <property> <key>instance-type</key> <value>m1.small</value> </property> </profile> <!-- Agent for profile "aws.small-us-east" --> <agent elasticPluginId="aws"/>`	Yes Since the job does not care which region it runs in.

<profile id="aws.small" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
<job name="unit-tests"
     elasticProfileId="aws.small">
</job>

<profile id="aws.large" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.xlarge</value>
  </property>
</profile>

<!-- Agent for profile "aws.large" -->
<agent elasticPluginId="aws"/>

Yes

Since the job requires a smaller agent, and a larger agent is available.

<profile id="aws.small" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
...
<job name="unit-tests"
     elasticProfileId="aws.small">
</job>

<profile id="aws.small-us-east" pluginId="aws">
  <property>
    <key>aws-region</key>
    <value>us-east-1</value>
  </property>
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
<!-- Agent for profile "aws.small-us-east" -->
<agent elasticPluginId="aws"/>

Yes

Since the job does not care which region it runs in.

See the video linked below to help make this a little more clear.

Requests from the GoCD server

In order to implement an elastic agent extension point the following messages must be implemented by the plugin.

Get Plugin Icon
Capabilities
Create Agent
Should Assign Work
Server Ping
Job Completion
Migrate Config

These are general purpose messages that a plugin must implement to allow users to configure the plugin through the browser.

Get Cluster Profile View
Get Cluster Profile Metadata
Validate Cluster Profile

These are messages that a plugin must implement in order to allow users to configure elastic profiles through the browser.

Get Elastic Agent Profile View
Get Elastic Agent Profile Metadata
Validate Elastic Agent Profile

If a plugin supports supports status reports, apart from the above, the plugin must implement the following messages. The plugin should use the Capabilities to expose this feature.

Agent Status Report
Get Cluster Status Report
Plugin Status Report

Get Plugin Icon

This call is expected to return the icon for the plugin, so as to make it easy for users to identify the plugin.

Request name

cd.go.elastic-agent.get-icon

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example plugin response body:

{
  "content_type": "image/svg+xml",
  "data": "PHN2ZyB2ZXJzaW9u..."
}

The plugin is expected to return an image object.

Get Plugin Capabilities

This message is a request to the plugin to provide plugin capabilities. Based on these capabilities GoCD would enable or disable the plugin features for a user.

Request name

cd.go.elastic-agent.get-capabilities

Request body

Server sends request with Empty request body.

Response Body

An example response body:

{
  "supports_plugin_status_report": true,
  "supports_agent_status_report": true,
  "supports_cluster_status_report": true
}

The response body will contain the following JSON elements:

Key	Type	Description
`supports_plugin_status_report`	`boolean`	Whether plugin supports Plugin status report depends on this `boolean` value
`supports_agent_status_report`	`boolean`	Whether plugin supports Agent status report depends on this `boolean` value
`supports_cluster_status_report`	`boolean`	Whether plugin supports Cluster status report depends on this `boolean` value

The plugin is expected to return status 200 if it can understand the request.

Migrate config

This message is a request to the plugin perform migration of the existing config on load of the plugin. This allows a plugin to perform the migration on the existing config in order to support the newer version of the plugin.

Request name

cd.go.elastic-agent.migrate-config

Request body

The plugin will receive the following JSON body —

{
  "cluster_profiles": [
    {
      "id": "cluster_profile_id",
      "plugin_id": "plugin_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ],
  "elastic_agent_profiles": [
    {
      "id": "profile_id",
      "plugin_id": "plugin_id",
      "cluster_profile_id": "cluster_profile_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ],
  "plugin_settings": {
    "url": "https://some-url",
    "password": "my-password"
  }
}

Key	Type	Description
`cluster_profiles`	`Array`	List of cluster profiles configured for the plugin.
`elastic_agent_profiles`	`Array`	List of elastic agent profiles configured for the plugin.
`plugin_settings`	`Array`	Plugin settings of the plugin.

Response Body

An example response body:

{
  "cluster_profiles": [
    {
      "id": "cluster_profile_id",
      "plugin_id": "plugin_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    },
    {
      "id": "migrated_from_plugin_settings",
      "plugin_id": "plugin_id",
      "properties": {
        "url": "https://some-url",
        "password": "my-password"
      }
    }
  ],
  "elastic_agent_profiles": [
    {
      "id": "profile_id",
      "plugin_id": "plugin_id",
      "cluster_profile_id": "cluster_profile_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ]
}

The response body will contain the following JSON elements:

Key	Type	Description
`cluster_profiles`	`Array`	List of cluster profiles configured for the plugin.
`elastic_agent_profiles`	`Array`	List of elastic agent profiles configured for the plugin.

The plugin is expected to return status 200 if it can understand the request otherwise the plugin will be marked as invalid.

Create Agent

This message is a request to the plugin to create an agent for a job that has been scheduled.

Request name

cd.go.elastic-agent.create-agent

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
        <property>
          <!--
            The plugin currently only supports the `Image` property,
            which allows you to select the docker image that the build should run with
          -->
          <key>Image</key>
          <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines group="first">
    <pipeline name="build">
      ...
      <job name="test-job" elasticProfileId="docker.unit-test">
        ...
      </job>
      ...
    </pipeline>
  </pipelines>
  ...
</cruise>

The plugin will receive the following JSON body —

{
  "auto_register_key": "1e0e05fc-eb45-11e5-bc83-93882adfccf6",
  "elastic_agent_profile_properties": {
    "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
    "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "environment": "prod",
  "job_identifier": {
    "job_id": 100,
    "job_name": "test-job",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  }
}

The request body will contain the following JSON elements:

Key	Type	Description
`auto_register_key`	`String`	The key that an agent should use, if it should be auto-registered with the server. The plugin is expected to use the key to create an appropriate `autoregister.properties` file on the agent instance, before it starts the agent process. See the auto-register documentation for more information.
`environment`	`String`	The `environment` that this job belongs to. Agents are expected to auto-register using this environment so that they can be assigned to the correct job. See the environments section to know more about environments.
`elastic_agent_profile_properties`	`Object`	Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
`cluster_profile_properties`	`Object`	The field represents the cluster profile associated with elastic profile.
`job_identifier`	`Object`	Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Should Assign Work

When there are multiple agents available to run a job, the server will ask the plugin if jobs should be assigned to a particular agent. The request will contain information about the agent, the job configuration and the environment that the agent belongs to. This allows plugin to decide if proposed agent is suitable to schedule a job on it. For example, plugin can check if flavor or region of VM is suitable.

Request name

cd.go.elastic-agent.should-assign-work

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6"/>
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile id="ec2.small-us-east" clusterProfileId="docker-local">
        <property>
          <key>ami-id</key>
          <value>ami-6ac7408f</value>
        </property>
        <property>
          <key>region</key>
          <value>us-east-1</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines>
    <pipeline name='build'>
      <job name="run-upgrade" runOnAllAgents="true" timeout='30' elasticProfileId="ec2.small-us-east">
        <tasks>
          <ant target="upgrade"/>
        </tasks>
      </job>
    </pipeline>
  </pipelines>
  <environments>
    <environment name="staging">
      <pipelines>
        <pipeline name="build"/>
      </pipelines>
    </environment>
  </environments>
  <agents>
    <agent elasticAgentId='i-283432d4' elasticPluginId='com.example.go.testplugin'/>
  </agents>
</cruise>

The plugin will receive the following JSON body —

{
  "agent": {
    "agent_id": "i-283432d4",
    "agent_state": "Idle",
    "build_state": "Idle",
    "config_state": "Enabled"
  },
  "environment": "staging",
  "job_identifier": {
    "job_id": 100,
    "job_name": "run-upgrade",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  },
  "elastic_agent_profile_properties": {
      "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
      "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  }
}

The request body will contain the following JSON elements:

Key	Type	Description
`agent`	`Object`	An object describing the elastic agent.
`environment`	`String`	The `environment` that this job belongs to. Agents are expected to auto-register using this environment so that they can be assigned to the correct job. See the environments section to know more about environments.
`elastic_agent_profile_properties`	`Object`	Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
`cluster_profile_properties`	`Object`	The field represents the cluster profile associated with elastic profile.
`job_identifier`	`Object`	Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The server must return a JSON string response with a boolean — true or false to indicate whether the agent should be assigned work.

Server Ping

Each elastic agent plugin will receive a periodic signal at regular intervals for it to perform any cleanup operations. Plugins may use this message to disable and/or terminate agents at their discretion.

Request name

cd.go.elastic-agent.server-ping

Request body

Given the following config XML snippet —

<elastic>
  <clusterProfiles>
    <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
      <property>
        <key>DockerURI</key>
        <value>https://docker-uri/</value>
      </property>
    </clusterProfile>
    <clusterProfile id="ecs" pluginId="cd.go.contrib.elastic-agent.ecs">
      <property>
        <key>AWS_ACCESS_KEY</key>
        <value>AMSDKFSDOFSFSI</value>
      </property>
      <property>
        <key>AWS_SECRET_KEY</key>
        <value>yshfksdfasd,fmsldgjdflgjgdflgjdlfgjdfl</value>
      </property>
      <property>
        <key>CLUSTER_NAME</key>
        <value>Dev</value>
      </property>
    </clusterProfile>
  </clusterProfiles>
</elastic>

The plugin will receive the following JSON body —

{
  "all_cluster_profile_properties": [
    {
      "DockerURI": "https://docker-uri/"
    },
    {
      "AWS_ACCESS_KEY": "AMSDKFSDOFSFSI",
      "AWS_SECRET_KEY": "yshfksdfasd,fmsldgjdflgjgdflgjdlfgjdfl",
      "CLUSTER_NAME"  : "Dev"
    }
  ]
}

Key	Type	Description
`all_cluster_profile_properties`	`Array`	The field represents the list of cluster profiles for the plugin.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Get Cluster Profile View

This is a message that the plugin should implement, to allow users to configure cluster profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-cluster-profile-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Get Cluster Profile Metadata

This is a message that the plugin should implement, to allow users to configure cluster profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-cluster-profile-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body for a docker plugin

[
  {
    "key": "DockerURI",
    "metadata": {
      "required": true,
      "secure": false
    }
  },
  {
    "key": "MaxDockerContainersAllowed",
    "metadata": {
      "required": false,
      "secure": false
    }
  }
]

A JSON cluster profile metadata object.

Validate Cluster Profile

This call is expected to validate the user inputs that form a part of the cluster profile.

Request name

cd.go.elastic-agent.validate-cluster-profile

Request body

The request body will contain a JSON object with the keys and values that form part of the cluster profile.

An example validation request body for the docker elastic agent plugin

{
  "DockerURI": "https://docker-uri",
  "MaxDockerContainersAllowed": "foo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "MaxMemory",
   "message": "'foo' is not a valid value for `MaxDockerContainersAllowed`."
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the cluster profile is valid, the plugin should return an empty JSON array.

Validate Elastic Agent Profile

This call is expected to validate the user inputs that form a part of the elastic agent profile.

Request name

cd.go.elastic-agent.validate-elastic-agent-profile

Request body

The request body will contain a JSON object with the keys and values that form part of the elastic agent profile.

An example validation request body for the docker elastic agent plugin

{
  "Image": "alpine:latest",
  "MaxMemory": "foo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "MaxMemory",
   "message": "'foo' is not a valid value for `MaxMemory`."
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the profile is valid, the plugin should return an empty JSON array.

Get Elastic Agent Profile View

This is a message that the plugin should implement, to allow users to configure elastic agent profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-elastic-agent-profile-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Get Elastic Agent Profile Metadata

This is a message that the plugin should implement, to allow users to configure elastic agent profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-elastic-agent-profile-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body for a docker plugin

[
  {
    "key": "Image",
    "metadata": {
      "required": true,
      "secure": false
    }
  }
]

A JSON elastic agent profile metadata object.

Job Completion

The intent on this message is to notify the plugin on completion of the job. The plugin may choose to terminate the elastic agent or keep it running in case the same agent can be used for another job configuration.

Request name

cd.go.elastic-agent.job-completion

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
        <property>
          <!--
            The plugin currently only supports the `Image` property,
            which allows you to select the docker image that the build should run with
          -->
          <key>Image</key>
          <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines group="first">
    <pipeline name="build">
      ...
      <job name="test-job" elasticProfileId="docker.unit-test">
        ...
      </job>
      ...
    </pipeline>
  </pipelines>
  ...
  <agents>
    <agent hostname="1697e6b164f7" ipaddress="172.17.0.6" uuid="a45e3ca1-4419-4bd5-b18b-882f75ffd4c2" elasticAgentId="GoCD18efbeef995e40f688cd92dc22a4d332" 
      elasticPluginId="cd.example.elastic-agent" />
  </agents>
</cruise>

The plugin will receive the following JSON body —

{
  "elastic_agent_id": "GoCD18efbeef995e40f688cd92dc22a4d332",
  "elastic_agent_profile_properties": {
    "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
    "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "job_identifier": {
    "job_id": 100,
    "job_name": "test-job",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  }
}

The request body will contain the following JSON elements:

Key	Type	Description
`elastic_agent_id`	`String`	The Elastic agent ID of the agent on which the job has completed.
`elastic_agent_profile_properties`	`Object`	Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
`cluster_profile_properties`	`Object`	The field represents the cluster profile associated with elastic profile.
`job_identifier`	`Object`	Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Get agent status report

If plugin supports status report, this message must be implemented to report the status of a particular elastic agent brought up by the plugin to run a job. The purpose of this call is to provide specific information about the current state of the elastic agent.

Request name

cd.go.elastic-agent.agent-status-report

Request body

An example request body for a docker plugin

{
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "elastic_agent_id": "46d7aa499c9f44958e118ffb7f975c52",
  "job_identifier": {
    "pipeline_name": "test-pipeline",
    "pipeline_label": "Test Pipeline",
    "pipeline_counter": 1,
    "stage_name": "test-stage",
    "stage_counter": "1",
    "job_name": "test-job",
    "job_id": 100
  }
}

The request body will contain the following JSON elements:

Key	Type	Description
`job_identifier`	`Object`	The job identifier for which the agent is created.
`elastic_agent_id`	`String`	This key contains the elastic agent id, this is available only when agent is registered with GoCD server.
`cluster_profile_properties`	`Object`	The key represents the cluster profile for the given `elastic_agent`

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "agent-status-report-html-view"
}

The request body will contain the following JSON elements:

Key	Type	Description
`view`	`String`	Agent status report view html. GoCD server will render this view as a agent status report.

Get cluster status report

If plugin supports cluster status report, this message must be implemented to provide the overall status of the cluster.

Request name

cd.go.elastic-agent.cluster-status-report

Request body

An example request body for a docker plugin

{
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  }
}

Key	Type	Description
`cluster_profile_properties`	`Object`	This key cluster profile for which the status report request is made.

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "plugin-status-report-html-view"
}

The request body will contain the following JSON elements:

Key	Type	Description
`view`	`String`	Plugin should return html view containing information about elastic agents.

Get plugin status report

If plugin supports the plugin status report, this message must be implemented to provide the overall status of the environment.

Request name

cd.go.elastic-agent.plugin-status-report

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="Dev" pluginId="cd.go.contrib.elastic-agent.ecs">
        <property>
          <key>AWS_ACCESS_KEY</key>
          <value>developer-access-key</value>
        </property>
        <property>
          <key>AWS_SECRET_KEY</key>
          <value>developer-secret-key</value>
        </property>
        <property>
          <key>CLUSTER_NAME</key>
          <value>Dev</value>
        </property>
      </clusterProfile>
      <clusterProfile id="Prod" pluginId="cd.go.contrib.elastic-agent.ecs">
        <property>
          <key>AWS_ACCESS_KEY</key>
          <value>prod-access-key</value>
        </property>
        <property>
          <key>AWS_SECRET_KEY</key>
          <value>prod-secret-key</value>
        </property>
        <property>
          <key>CLUSTER_NAME</key>
          <value>Production</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
  </elastic>
</cruise>

The plugin will receive the following JSON body —

{
  "all_cluster_profiles_properties": [
    {
      "AWS_ACCESS_KEY": "developer-access-key",
      "AWS_SECRET_KEY": "developer-secret-key",
      "CLUSTER_NAME": "Dev"
    },
    {
      "AWS_ACCESS_KEY": "prod-access-key",
      "AWS_SECRET_KEY": "prod-secret-key",
      "CLUSTER_NAME": "Production"
    }
  ]
}

Key	Type	Description
`all_cluster_profiles_properties`	`Array`	The field represents the list of cluster profiles for the plugin.

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "plugin-status-report-html-view"
}

The request body will contain the following JSON elements:

Key	Type	Description
`view`	`String`	Plugin should return html view containing information about elastic agents.

Requests to the GoCD server

The plugin may make the following requests to the server using GoApplicationAccessor#submit(GoApiRequest)

List Agents
Disable Agents
Delete Agents
Get Plugin Settings
Get Server Info
Add Server Health Messages
Log Message to Job Console

List Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private List listAgents() {
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.list-agents",
      "1.0",
      pluginIdentifier()
    );

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }

    // parse the response, using a json parser of your choice
    List agents = new Gson().fromJson(response.responseBody(), ArrayList.class);
    return agents;
  }
}

This messages allows a plugin to query the server for a list of elastic agents that belong to a particular plugin.

Request name

go.processor.elastic-agents.list-agents

Request version

The request version must be set to 1.0.

Request body

Can be left blank, the server does not parse the request body.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will send a list of agents.

Disable Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private disableAgents(List agents) {
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.disable-agents",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    request.setRequestBody(new Gson().toJson(agents));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

Before a plugin terminates an agent instance, it must first disable it in order to prevent jobs from being assigned to it. Agents in any state can be disabled, this will prevent jobs from being assigned to it. Any jobs the agent is running will eventually complete.

Request name

go.processor.elastic-agents.disable-agents

Request version

The request version must be set to 1.0.

Request body

The body must contain a list of agents.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will not send a response body.

Delete Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private deleteAgents(List agents) {
    // first terminate the instance from AWS, or a cloud provider of your choice
    aws.terminateInstances(agents);
    // ensure that they are really terminated,
    // to prevent stray agents from re-registering
    aws.waitForTermination(agents);

    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.disable-agents",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    request.setRequestBody(new Gson().toJson(agents));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

Before the delete-agent message is sent to the server, the plugin MUST ensure that the agent is terminated.

Important: Before an agent is terminated, it is recommended to ensure that it is first disabled and that the agent_state is Idle. This will ensure that any jobs that an agent is running are completed.

Not doing so will cause jobs to be rescheduled on another instance after a timeout interval.

Request name

go.processor.elastic-agents.delete-agents

Request version

The request version must be set to 1.0.

Request body

The body must contain a list of agents that should be removed from the config XML.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will not send a response body.

Get Plugin Settings

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private PluginSettings getSettings() {
    Gson gson = new Gson();
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.plugin-settings.get",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    Map<String, String> map = new HashMap<>();
    map.put("plugin-id", "com.example.rocket.launcher");
    request.setRequestBody(gson.toJson(map));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }

    // parse the response, using a json parser of your choice
    return gson.fromJson(response.responseBody(), PluginSettings.class);
  }
}

This messages allows a plugin to query the server to get the user configured settings for this plugin.

Request name

go.processor.plugin-settings.get

Request version

The request version must be set to 5.0.

Request body

An example request body:

{
  "plugin-id": "sample-plugin-id"
}

Must be a JSON object with a key plugin-id with the value being the ID of your plugin.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

An example response body:

{
  "server_url": "https://build.go.cd",
  "username": "view",
  "password": "password"
}

The server will send a map of settings.

Get Server Info

This messages allows a plugin to query the server to get some metadata about the server.

Available since v17.9.0.

Request name

go.processor.server-info.get

Request version

The request version must be set to 1.0.

Request body

The plugin should not provide a request body.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The plugin will provide a server info object.

An example response body:

{
  "server_id": "df0cb9be-2696-4689-8d46-1ef3c4e4447c",
  "site_url": "http://example.com:8153/go",
  "secure_site_url": "https://example.com:8154/go"
}

Add Server Health Messages

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"));
  }

  private void addErrorsAndWarnings() {
    Gson gson = new Gson();
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.server-health.add-messages",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    List<Map<String, String>> messages = new ArrayList<>();

    Map<String, String> message1 = new HashMap<>();
    message1.put("type", "warning");
    message1.put("message", "A warning message from the plugin.");

    Map<String, String> message2 = new HashMap<>();
    message2.put("type", "error");
    message2.put("message", "An error message from the plugin.");

    messages.add(message1);
    messages.add(message2);

    request.setRequestBody(gson.toJson(messages));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

This message allows a plugin to add error and warning messages to be shown in GoCD. Any previous messages sent by the plugin will be cleared and replaced with the newly specified messages (or cleared if the body is an empty list).

Available since v18.3.0.

Request name

go.processor.server-health.add-messages

Request version

The request version must be set to 1.0.

Request body

An example request body:

[
  {
    "type": "warning",
    "message": "A warning message from the plugin."
  },
  {
    "type": "error",
    "message": "An error message from the plugin."
  }
]

Must be a JSON array made up of JSON objects as described below:

Key	Type	Description
`type`	`String`	Should be either `warning` or `error`, corresponding to the type of message to be shown.
`message`	`String`	A message to be shown in the “Errors and Warnings” box.

Response code

The server is expected to return status 200 if it could process the request. It is expected to return status 500 if it failed to process the request.

Response Body

An example response body for a failure:

{
  "message": "An error occurred ..."
}

The server will respond with a single JSON object with an error message with the key message, if it is unable to process the request. If successful, the response body will be empty.

Log Messages to Job Console

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"));
  }

  private void appendToConsoleLog(String text) throws ServerRequestFailedException {
    Map<String, String> requestMap = new HashMap<>();
    requestMap.put("pipeline_name", "my_pipeline_1");
    requestMap.put("pipeline_counter", "123");
    requestMap.put("stage_name", "stage1");
    requestMap.put("stage_counter", "1");
    requestMap.put("job_name", "job1");
    requestMap.put("text", text);

    DefaultGoApiRequest request = new DefaultGoApiRequest("go.processor.console-log.append", "2.0", pluginIdentifier());
    request.setRequestBody(new GsonBuilder().create().toJson(requestMap));

    GoApiResponse response = accessor.submit(request);

    if (response.responseCode() != 200) {
      LOG.error("Failed to append console log for " + jobIdentifier.represent() + " with text: " + text);
    }
  }
}

This message allows a plugin to add messages to be shown in the GoCD job console.

Available since v19.9.0.

Request name

go.processor.console-log.append

Request version

The request version must be set to 2.0.

Request body

An example request body:

{
  "pipeline_name": "my_pipeline_1",
  "pipeline_counter": "123",
  "stage_name": "stage1",
  "stage_counter": "1",
  "job_name": "job1",
  "text": "Message to show in console log."
}

Must be a JSON object made up of JSON elements as described below:

Key	Type	Description
`pipeline_name`	`String`	Name of the pipeline in which the job is.
`pipeline_counter`	`String`	The pipeline counter (run counter of the pipeline).
`stage_name`	`String`	Name of the stage the job is in.
`stage_counter`	`String`	The run counter of the stage.
`job_name`	`String`	The name of the job.
`text`	`String`	The message to be shows in the console log of the job.

Response code

The server is expected to return status 200 if it could process the request. It is expected to return status 500 if it failed to process the request.

Response Body

An example response body for a failure:

{
  "message": "An error occurred ..."
}

The server will respond with a single JSON object with an error message with the key message, if it is unable to process the request. If successful, the response body will be empty.

Request/Response JSON Objects

The Elastic Agent Object

Here’s an example of the elastic agent object:

{
  "agent_id": "i-283432d4",
  "agent_state": "Idle",
  "build_state": "Idle",
  "config_state": "Enabled"
}

Attribute	Type	Description
`agent_id`	String	The elastic agent ID. This is the value of the `elasticAgentId` attribute of the `<agent/>` element in the config XML.
`agent_state`	String	The state that an agent is in. Can be one of `Idle`, `Building`, `LostContact`, `Missing`, `Unknown`.
`build_state`	String	The state the build is in. Can be one of `Idle`, `Building`, `Cancelled`, `Unknown`.
`config_state`	String	The state of the agent in the config file. This is the value of the `isDisabled` attribute of the `<agent/>` element in the config XML. Can be one of `Pending`, `Enabled`, `Disabled`.

The Settings View Object

Here’s an example of the settings view object:

{
  "template": "<div class=\"form_item_block\">...</div>"
}

Attribute	Type	Description
`template`	String	A string containing an HTML AngularJS based view.

This template is an AngularJS based template.

GoCD uses Angular JS as its template engine for the plugin UI. This allows plugin authors to use a limited set of AngularJS features to specify how the UI of their plugins looks.

Getting started with AngularJS based templates

Given a configuration:

<configuration>
  <property>
    <key>username</key>
    <value>alice</username>
  </property>
</configuration>

This gets converted into the following JSON representation in the browser:

{
  "username": "alice"
}

The AngularJS template is expected to bind to the JSON object shown above:

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
</div>

When an Angular template is used in a Go plugin, to define the configuration UI, the configuration key which is stored in the configuration XML is used everywhere and is expected to be consistent. Since Angular works off of JSON, GoCD will make sure that the key in the JSON provided to the Angular template is the same as the key in the configuration XML.

Suppose the key of the configuration property stored in the XML is “username”, with value, “alice”, then Go will make sure that the value is available to the template as “username” when used in an Angular-specific HTML attribute like “ng-model”.

Plugin Angular Architecture

So, the name “foobar” needs to be the same across the configuration XML, the Angular template as well as in any code that the plugin has.

Showing validation errors in the UI

We use some simple string replacement
to substitute GOINPUTNAME with a unique identifier
for your plugin in order to render
any server side errors

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
  <span class="form_error" ng-show="GOINPUTNAME[username].$error.server">
    {{ GOINPUTNAME[username].$error.server}}
  </span>
</div>

In case of validation errors returned by go.plugin-settings.validate-configuration, the error messages needs to be populated on the UI, use the snippet here to show the validation errors.

The Plugin Settings Configuration Object

Here’s an example of the plugin settings configuration object:

{
  "server_url": {
    "display-name": "Server URL",
    "display-order": "0"
  },
  "username": {
    "required": false,
    "display-name": "Username",
    "display-order": "1"
  },
  "password": {
    "secure": true,
    "required": false,
    "display-name": "Password",
    "display-order": "2"
  }
}

Attribute	Type	Description
`display-name`	String	The name of the property.
`default-value`	String	The default value of the property.
`display-order`	String	A string containing a numerical value.
`required`	Boolean	If the field is mandatory.
`secure`	Boolean	If the data in the field should be stored encrypted.

The Validation Error Object

Here’s an example of the validation error object:

[
  {
    "key": "email_address",
    "message": "Email address is invalid"
  },
  {
    "key": "password",
    "message": "Password must be provided"
  }
]

Attribute	Type	Description
`key`	String	The name of configuration key that has an error.
`message`	String	The error message associated with that key.

The Image Object

Here’s an example of the image object:

{
  "content_type": "image/svg+xml",
  "data": "...."
}

Attribute	Type	Description
`content_type`	String	A valid content type for the image. Please make sure the content type is supported by most browsers.
`data`	String	A base-64 encoded (single-line non-chunking) byte array of the byte-sequence that composes the image.

The Profile Metadata Object

Here’s an example of the profile metadata object:

{
  "content_type": "image/svg+xml",
  "data": "...."
}

Attribute	Type	Description
`key`	String	The name of the configuration property supported by an elastic profile.
`metadata`	Object	The metadata associated with the key used in the elastic profile. Valid keys are `required` and `secure`.

The server info object

Here’s an example of the server info object:

{
  "server_id": "df0cb9be-2696-4689-8d46-1ef3c4e4447c",
  "site_url": "http://example.com:8153/go",
  "secure_site_url": "https://example.com:8154/go"
}

Attribute	Type	Description
`server_id`	String	This contains a unique identifier for this server.
`site_url`	String	This contains the site url configured for this server.
`secure_site_url`	String	This contains the secure site url configured for this server.

Glossary

Scheduled: The state that a job is in, before it is assigned (see “Assigned”) to an agent. A job can stay in a scheduled state for a long time, if all suitable (see “Suitable”) agents are busy running other jobs.
Assigned: The state that a job is in, when a suitable (see “Suitable”) agent has been decided for it. The job state changes to “Preparing” once the agent actually picks up the job, and starts checking out materials, etc. in preparation for the job.
Suitable: An agent is said to be “suitable” for a job, if it has all the resources that the job needs, and is in the environment that the pipeline of this job is in.