Artifact Plugins

Extension Information

Background

By default, GoCD internally manages and stores artifacts. The Artifact extension allows GoCD to make use of an external artifact store for storing artifacts while retaining traceability between pipelines. For example, by using the Docker artifact registry plugin, a user can publish an image to a docker registry, and subsequently fetch it in a downstream pipeline. Since GoCD is “aware” of the artifact store with this extension, it can still maintain the Value Stream between the upstream and downstream pipelines even if the artifact is stored outside of GoCD.

Developers can start building their own artifact plugins by forking the docker plugin as an example reference implementation.

How will it help you?

A feature like this will allow for artifacts to be uploaded and stored outside of GoCD. Previously, the artifacts were only managed on the GoCD server.

Requests from the GoCD server

In order to implement an artifact extension point the following messages must be implemented by the plugin.

• Publish Artifact
• Fetch Artifact
• Capabilities

These are general purpose messages that a plugin must implement to allow users to configure the plugin through the browser.

• Get Plugin Icon

These are messages that a plugin must implement in order to allow users to configure artifact stores through the browser.

• Get Store Config Metadata
• Get Store Config View
• Validate Store Config

These are messages that a plugin must implement in order to allow users to configure publishing an artifact through the browser.

• Get Publish Artifact Metadata
• Get Publish Artifact View
• Validate Publish Artifact Configuration

These are messages that a plugin must implement in order to allow users to configure fetching an artifact through the browser.

• Get Fetch Artifact Metadata
• Get Fetch Artifact View
• Validate Fetch Artifact Configuration

Publish Artifact

This message is a request to the plugin to publish an artifact to the specified artifact store.

Request name

cd.go.artifact.publish-artifact

Request body

Given the following config XML snippet —

<artifactStores>
      <artifactStore id="dockerhub" pluginId="cd.go.artifact.docker.registry">
        <property>
          <key>RegistryURL</key>
          <value>https://index.docker.io/v1/</value>
        </property>
        <property>
          <key>Username</key>
          <value>boohoo</value>
        </property>
        <property>
          <key>Password</key>
          <value>password</value>
        </property>
      </artifactStore>
    </artifactStores>
<pipelines group="first">
  <pipeline name="build">
    ...
    <job name="build-job">
      ...
      <artifacts>
        <artifact id="app-image" storeId="dockerhub">
          <configuration>
            <property>
              <key>Image</key>
              <value>gocd/gocd-demo</value>
            </property>
            <property>
              <key>Tag</key>
              <value>v${GO_PIPELINE_COUNTER}</value>
            </property>
          </configuration>
        </artifact>
      </artifacts>
    </job>
    ...
  </pipeline>
</pipelines>

The plugin will receive the following JSON body —

{
  "environment_variables":{
    "GO_PIPELINE_NAME":"build",
    "GO_TRIGGER_USER":"admin",
    "FOO": "bar"
  },
  "artifact_plan":{
    "configuration":{
      "BuildFile":"",
      "Image":"gocd/gocd-demo",
      "Tag":"v${GO_PIPELINE_COUNTER}"
    },
    "id":"app-image",
    "storeId":"dockerhub"
  },
  "artifact_store":{
    "configuration":{
      "RegistryURL":"https://index.docker.io/v1/",
      "Username":"boohoo",
      "Password":"password"
    },
    "id":"dockerhub"
  },
  "agent_working_directory":"/Users/varshavs/gocd/agent/pipelines/build"}

The request body will contain the following JSON elements:

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin is expected to return a json as shown. This json is written into a file called <plugin-id>.json on the agent and uploaded as a Build Artifact to the GoCD server to a directory called pluggable-artifact-metadata. This directory is never removed as part of cleaning GoCD artifacts.

Plugin response body

{
  "metadata":{
    "image":"gocd/gocd-demo:v23",
    "digest":"sha256:f7840887b6f09f531935329a4ad1f6176866675873a8b3eed6a5894573da8247"
    }
}

The content of plugin-id.json file

{
  "app-image":{
  "image":"gocd/gocd-demo:v23",
  "digest":"sha256:f7840887b6f09f531935329a4ad1f6176866675873a8b3eed6a5894573da8247"
  }
}

Note: app-image is the artifact id specified in the job configuration

Fetch Artifact

This message is a request to the plugin to fetch an artifact from the specified artifact store.

Request name

cd.go.artifact.fetch-artifact

Request body

Given the following config XML snippet —

<artifactStores>
      <artifactStore id="dockerhub" pluginId="cd.go.artifact.docker.registry">
        <property>
          <key>RegistryURL</key>
          <value>https://index.docker.io/v1/</value>
        </property>
        <property>
          <key>Username</key>
          <value>boohoo</value>
        </property>
        <property>
          <key>Password</key>
          <value>password</value>
        </property>
      </artifactStore>
    </artifactStores>
<pipelines group="first">
  <pipeline name="build">
    ...
    <job name="build-job">
      ...
      <artifacts>
        <artifact type="external" id="app-image" storeId="dockerhub">
          <configuration>
            <property>
              <key>Image</key>
              <value>gocd/gocd-demo</value>
            </property>
            <property>
              <key>Tag</key>
              <value>v${GO_PIPELINE_COUNTER}</value>
            </property>
          </configuration>
        </artifact>
      </artifacts>
    </job>
    ...

    <stage name="downstream_stage">
      ...
      <job name="downstream_job">
        <tasks>
            <fetchartifact artifactOrigin="external" stage="upstream_stage" job="build-job" artifactId="app-image">
              <configuration>
                <property>
                  <key>image_name</key>
                  <value>release-candidate</value>
                </property>
              </configuration>
            </fetchartifact>
        </tasks>
      </job>
      ...
    </stage>
  </pipeline>
</pipelines>

The plugin will receive the following JSON body —

{
  "fetch_artifact_configuration":{
    "image_name": "release-candidate"
  },
  "artifact_metadata":{
    "image":"gocd/gocd-demo:v23",
    "digest":"sha256:f7840887b6f09f531935329a4ad1f6176866675873a8b3eed6a5894573da8247"
  },
  "store_configuration":{
    "RegistryURL":"https://index.docker.io/v1/",
    "Username":"boohoo",
    "Password":"password"
  },
  "agent_working_directory":"/Users/varshavs/gocd/agent/pipelines/build"
}

The request body will contain the following JSON elements:

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin can respond with a list of environment variables which will be set on the job, for all further tasks in that job to use. It can also return an empty list ([]). Existing environment variables will be overridden, if they are returned as a part of this response.

Example response body:

[
  {
    "name": "IMAGE_ID",
    "value": "image1/v23",
    "secure": false
  },
  {
    "name": "MY_SECRET_ENV_VAR",
    "value": "some-password",
    "secure": true
  }
]

The response body will contain the following JSON elements:

Get Plugin Capabilities

This message is a request to the plugin to provide plugin capabilities. Currently the plugin does not support any capabilities, so plugins should return an empty JSON as a response. In the future, based on these capabilities GoCD would enable or disable the plugin features for a user.

Request name

cd.go.artifact.get-capabilities

Request body

Server sends request an Empty request body.

Response Body

The plugin is expected to return status 200 if it can understand the request, with an empty response body

Get Plugin Icon

This call is expected to return the icon for the plugin, so as to make it easy for users to identify the plugin.

Request name

cd.go.artifact.get-icon

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example plugin response body:

{
  "content_type": "image/svg+xml",
  "data": "PHN2ZyB2ZXJzaW9u..."
}

The plugin is expected to return an image object.

Get Store Config Metadata

This is a message that the plugin should implement, to allow users to configure global artifact stores from the Artifact Stores View in GoCD.

Request name

cd.go.artifact.store.get-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body for a docker plugin

[
  {
    "key": "RegistryURL",
    "metadata": {
      "required": true,
      "secure": false
    }
  },
  {
    "key": "Password",
    "metadata": {
      "required": true,
      "secure": true
    }
  }
]

A JSON metadata object.

Get Store Config View

This is a message that the plugin should implement, to allow users to configure artifact stores from the Artifact Stores View in GoCD.

Request name

cd.go.artifact.store.get-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Validate Store Config

This call is expected to validate the user inputs that form a part of the artifact store.

Request name

cd.go.artifact.store.validate

Request body

The request body will contain a JSON object with the keys and values that form part of the profile.

An example validation request body

{
  "RegistryURL": "https://index.docker.io/v1/",
  "Username": "boohoo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "RegistryURL",
   "message": "'RegistryURL is invalid'"
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the profile is valid, the plugin should return an empty JSON array.

Get Publish Artifact Metadata

This is a message that the plugin should implement, to allow users to configure external artifacts from the Job Config View in GoCD.

Request name

cd.go.artifact.publish.get-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body

[
  {
    "key": "Image",
    "metadata": {
      "required": true,
      "secure": false
    }
  },
  {
    "key": "Tag",
    "metadata": {
      "required": true,
      "secure": false
    }
  }
]

A JSON metadata object.

Get Publish Artifact View

This is a message that the plugin should implement, to allow users to configure external artifacts from the Job Config View in GoCD.

Request name

cd.go.artifact.publish.get-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Validate Publish Artifact Config

This call is expected to validate the user inputs that form a part of the publish external artifact config.

Request name

cd.go.artifact.publish.validate

Request body

The request body will contain a JSON object with the keys and values that form part of the profile.

An example validation request body

{
  "Image": "gocd/gocd-demo",
  "Tag": "boohoo",
  "BuildFile": "foo.json"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "Image",
   "message": "Either `BuildFile` or `Image` must be specified"
  },
  {
   "key": "BuildFile",
   "message": "Either `BuildFile` or `Image` must be specified"
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the profile is valid, the plugin should return an empty JSON array.

Get Fetch Artifact Metadata

This is a message that the plugin should implement, to allow users to configure fetch artifact task from the Task View in GoCD.

Request name

cd.go.artifact.fetch.get-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body:

[
  {
    "key": "DestOnAgent",
    "metadata": {
      "required": true,
      "secure": false
    }
  }
]

A JSON metadata object.

Get Fetch Artifact View

This is a message that the plugin should implement, to allow users to configure fetch artifact task from the Task View in GoCD.

Request name

cd.go.artifact.fetch.get-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Validate Fetch Artifact Config

This call is expected to validate the user inputs that form a part of the fetch external artifact config.

Request name

cd.go.artifact.publish.validate

Request body

The request body will contain a JSON object with the keys and values that form part of the profile.

An example validation request body

{
  "Dest": "foo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "Dest",
   "message": "Invalid"
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the profile is valid, the plugin should return an empty JSON array.

Requests to the GoCD server

The plugin may make the following requests to the server using GoApplicationAccessor#submit(GoApiRequest)

• Send Console Log

Send Console Log

The plugin can send console logs to the server while fetching or publishing artifacts.

Request name

go.processor.artifact.console-log

Request version

The request version must be set to 1.0.

Request body

The request body must contain the logLevel and a message string. The logLevel must be INFO or ERROR.

Example Request Body

{
  "logLevel": "INFO",
  "message" : "This is an info message."
}

Response code

The server is expected to return status 200 if it could process the request.

Response body

The server will not send a response body.

Console Logger Example

import com.google.gson.Gson;
import com.thoughtworks.go.plugin.api.GoApplicationAccessor;
import com.thoughtworks.go.plugin.api.GoPluginIdentifier;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.DefaultGoApiRequest;
import com.thoughtworks.go.plugin.api.response.DefaultGoApiResponse;
import com.thoughtworks.go.plugin.api.response.GoApiResponse;

import java.util.Collections;

public class ConsoleLogger {
    private static final String SEND_CONSOLE_LOG = "go.processor.artifact.console-log";
    private static final String API_VERSION = "1.0";
    private GoPluginIdentifier PLUGIN_IDENTIFIER = new GoPluginIdentifier("artifact", Collections.singletonList(API_VERSION));
    private static final Logger LOG = Logger.getLoggerFor(DockerRegistryArtifactPlugin.class);
    private static ConsoleLogger consoleLogger;
    private final GoApplicationAccessor accessor;

    private ConsoleLogger(GoApplicationAccessor accessor) {
        this.accessor = accessor;
    }

    public void info(String message) {
        sendLog(new ConsoleLogMessage(ConsoleLogMessage.LogLevel.INFO, message));
    }

    public void error(String message) {
        sendLog(new ConsoleLogMessage(ConsoleLogMessage.LogLevel.ERROR, message));
    }

    private void sendLog(ConsoleLogMessage consoleLogMessage) {
        DefaultGoApiRequest request = new DefaultGoApiRequest(SEND_CONSOLE_LOG, API_VERSION, PLUGIN_IDENTIFIER);
        request.setRequestBody(consoleLogMessage.toJSON());

        GoApiResponse response = accessor.submit(request);
        if (response.responseCode() != DefaultGoApiResponse.SUCCESS_RESPONSE_CODE) {
            LOG.error(String.format("Failed to submit console log: %s", response.responseBody()));
        }
    }

    public static ConsoleLogger getLogger(GoApplicationAccessor accessor) {
        if (consoleLogger == null) {
            synchronized (ConsoleLogger.class) {
                if (consoleLogger == null) {
                    consoleLogger = new ConsoleLogger(accessor);
                }
            }
        }

        return consoleLogger;
    }

    static class ConsoleLogMessage {
        private LogLevel logLevel;
        private String message;

        public ConsoleLogMessage(LogLevel logLevel, String message) {
            this.message = message;
            this.logLevel = logLevel;
        }

        public String toJSON() {
            return new Gson().toJson(this);
        }

        enum LogLevel {
            INFO, ERROR
        }
    }
}

Request/Response JSON Objects

The Settings View Object

Here’s an example of the settings view object:

{
  "template": "<div class=\"form_item_block\">...</div>"
}

This template is an AngularJS based template.

GoCD uses Angular JS as its template engine for the plugin UI. This allows plugin authors to use a limited set of AngularJS features to specify how the UI of their plugins looks.

Getting started with AngularJS based templates

Given a configuration:

<configuration>
  <property>
    <key>username</key>
    <value>alice</username>
  </property>
</configuration>

This gets converted into the following JSON representation in the browser:

{
  "username": "alice"
}

The AngularJS template is expected to bind to the JSON object shown above:

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
</div>

When an Angular template is used in a Go plugin, to define the configuration UI, the configuration key which is stored in the configuration XML is used everywhere and is expected to be consistent. Since Angular works off of JSON, GoCD will make sure that the key in the JSON provided to the Angular template is the same as the key in the configuration XML.

Suppose the key of the configuration property stored in the XML is “username”, with value, “alice”, then Go will make sure that the value is available to the template as “username” when used in an Angular-specific HTML attribute like “ng-model”.

So, the name “foobar” needs to be the same across the configuration XML, the Angular template as well as in any code that the plugin has.

Showing validation errors in the UI

We use some simple string replacement
to substitute GOINPUTNAME with a unique identifier
for your plugin in order to render
any server side errors

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
  <span class="form_error" ng-show="GOINPUTNAME[username].$error.server">
    {{ GOINPUTNAME[username].$error.server}}
  </span>
</div>

In case of validation errors returned by go.plugin-settings.validate-configuration, the error messages needs to be populated on the UI, use the snippet here to show the validation errors.

The Plugin Settings Configuration Object

Here’s an example of the plugin settings configuration object:

{
  "server_url": {
    "display-name": "Server URL",
    "display-order": "0"
  },
  "username": {
    "required": false,
    "display-name": "Username",
    "display-order": "1"
  },
  "password": {
    "secure": true,
    "required": false,
    "display-name": "Password",
    "display-order": "2"
  }
}

The Validation Error Object

Here’s an example of the validation error object:

[
  {
    "key": "email_address",
    "message": "Email address is invalid"
  },
  {
    "key": "password",
    "message": "Password must be provided"
  }
]

The Image Object

Here’s an example of the image object:

{
  "content_type": "image/svg+xml",
  "data": "...."
}

The Metadata Object

Here’s an example of the metadata object:

{
  "key": {
    "required": "true",
    "secure": "true"
  }
}

The server info object

Here’s an example of the server info object:

{
  "server_id": "df0cb9be-2696-4689-8d46-1ef3c4e4447c",
  "site_url": "http://example.com:8153/go",
  "secure_site_url": "https://example.com:8154/go"
}