Config Repositories
Extension Information
Availability | GoCD version 20.8.0 onwards |
Extension Name | configrepo |
Extension API Version | 3.0 |
Introduction
Config repository extension endpoint allows a plugin to externalize pipeline and environment configuration. This allows the configuration to be maintained and versioned outside of GoCD.
Requests from the GoCD server
In order to implement config repository extension point the following message must be implemented by the plugin.
These are general purpose messages that a plugin may implement to allow users to configure the plugin through the browser.
Parse directory
This message is a request to the plugin to parse a directory containing files which represent pipeline configuration. GoCD will periodically check the configured material repository (git, svn, etc) and if it finds a new commit, will send this request to the plugin, so that it can parse the configuration and tell it which pipelines have been defined in the external configuration.
Request name
parse-directory
Request body
Given the following config XML snippet —
<config-repos>
<config-repo pluginId="your.plugin.id" id="some-unique-id">
<git url="https://some.repo" />
<configuration>
<property>
<key>some.key</key>
<value>some.value</value>
</property>
</configuration>
</config-repo>
The plugin will receive the following JSON body —
{
"directory": "/absolute/path/to/directory/with/configs",
"configurations": [
{
"key": "some.key",
"value": "some.value"
}
]
}
The request body will contain a JSON object with an attribute directory
, which points to a local checkout of the specified
material repository. It will also contain an attribute configuration
which contains the specified configuration. Note:
This is not the configuration in the plugin settings page.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response body
An example response body of a problematic config:
{
"target_version": 2,
"environments": [],
"pipelines": [],
"errors": [
{
"location": "1.pipeline.json",
"message": "There is an error here ..."
}
]
}
A full response, with environments and pipelines:
{
"target_version": 2,
"environments": [
{
"name": "environmentName",
"environment_variables": [
{
"name": "env1",
"value": "val1"
}
],
"agents": [
"agent-uuid-1",
"agent-uuid-2"
],
"pipelines": [
"pipeline1"
]
}
],
"pipelines": [
{
"name": "pipeline1",
"group": "group1",
"display_order_weight": 10,
"label_template": "labelTemplate1",
"lock_behavior": "lockOnFailure",
"tracking_tool": {
"link": "link",
"regex": "regex",
"location": "optional: key to group errors by"
},
"mingle": {
"base_url": "baseUrl",
"project_identifier": "projectId",
"location": "optional: key to group errors by"
},
"timer": {
"spec": "timerSpec",
"only_on_changes": true,
"location": "optional: key to group errors by"
},
"environment_variables": [
{
"name": "key1",
"value": "value1",
"location": "optional: key to group errors by"
}
],
"parameters": [
{
"name": "param1",
"value": "value1",
"location": "optional: key to group errors by"
}
],
"materials": [
{
"type": "git",
"url": "gitrepo",
"branch": "feature12",
"shallow_clone": true,
"filter": {
"ignore": [
"externals",
"tools"
],
"whitelist": []
},
"destination": "dir1",
"auto_update": false,
"name": "gitMaterial1",
"location": "optional: key to group errors by"
},
{
"type": "hg",
"url": "repos/myhg",
"filter": {
"ignore": [
"externals",
"tools"
],
"whitelist": []
},
"destination": "dir1",
"auto_update": false,
"name": "hgMaterial1",
"location": "optional: key to group errors by"
},
{
"type": "p4",
"port": "10.18.3.102:1666",
"username": "user1",
"password": "pass1",
"use_tickets": false,
"view": "//depot/dev/src... //anything/src/...",
"filter": {
"ignore": [
"lib",
"tools"
],
"whitelist": []
},
"destination": "dir1",
"auto_update": false,
"name": "p4materialName",
"location": "optional: key to group errors by"
},
{
"type": "svn",
"url": "http://svn",
"username": "user1",
"password": "pass1",
"check_externals": true,
"filter": {
"ignore": [
"tools",
"lib"
],
"whitelist": []
},
"destination": "destDir1",
"auto_update": false,
"name": "svnMaterial1",
"location": "optional location for errors"
},
{
"type": "tfs",
"url": "url3",
"username": "user4",
"domain": "example.com",
"password": "pass",
"project": "projectDir",
"filter": {
"ignore": [
"tools",
"externals"
],
"whitelist": []
},
"destination": "dir1",
"auto_update": false,
"name": "tfsMaterialName",
"location": "optional: key to group errors by"
},
{
"type": "package",
"package_id": "package-id-in-bigger-config-xml",
"name": "packageMaterialName",
"location": "optional: key to group errors by"
},
{
"type": "plugin",
"scm_id": "someScmGitRepositoryId",
"destination": "destinationDir",
"filter": {
"ignore": [
"dir1",
"dir2"
],
"whitelist": []
},
"name": "myPluggableGit",
"location": "optional: key to group errors by"
},
{
"type": "plugin",
"plugin_configuration": {
"id": "some-plugin-id",
"version": "1",
"location": "optional: key to group errors by"
},
"configuration": [
{
"key": "url",
"value": "git@github.com:gocd/plugin-api.go.cd.git",
"encrypted_value": "encryptedValue1",
"location": "optional: key to group errors by"
}
],
"destination": "scmDestinationDir",
"filter": {
"ignore": [
"dir1",
"dir2"
],
"whitelist": []
},
"name": "myNewPluggableGitDefinition",
"location": "optional: key to group errors by"
},
{
"type": "dependency",
"pipeline": "pipeline2",
"stage": "build",
"name": "pipe2",
"location": "optional: key to group errors by"
}
],
"stages": [
{
"name": "stage1",
"fetch_materials": true,
"never_cleanup_artifacts": true,
"clean_working_directory": true,
"approval": {
"type": "manual",
"users": [
"user1"
],
"roles": [
"role1"
],
"location": "optional: key to group errors by"
},
"environment_variables": [
{
"name": "key1",
"value": "value1",
"location": "optional: key to group errors by"
}
],
"jobs": [
{
"name": "job1",
"environment_variables": [
{
"name": "key1",
"value": "value1",
"location": "optional: key to group errors by"
}
],
"tabs": [
{
"name": "tab1",
"path": "path/to/file",
"location": "optional: key to group errors by"
}
],
"resources": [
"resource1"
],
"artifacts": [
{
"source": "src1",
"destination": "dest1",
"type": "build",
"location": "optional: key to group errors by"
},
{
"source": "src2",
"destination": "dest2",
"type": "test",
"location": "optional: key to group errors by"
}
],
"properties": [
{
"name": "name1",
"source": "target/file.xml",
"xpath": "string(//path/to/element)",
"location": "optional: key to group errors by"
}
],
"elastic_profile_id": "elasticProfile1",
"run_instance_count": "all",
"timeout": 20,
"tasks": [
{
"command": "command1",
"working_directory": "workDir1",
"timeout": 120,
"arguments": [
"arg1",
"arg2"
],
"run_if": "passed",
"on_cancel": {
"plugin_configuration": {
"id": "pluginId",
"version": "version1",
"location": "optional: key to group errors by"
},
"configuration": [
{
"key": "key1",
"value": "value1",
"encrypted_value": "encryptedValue1",
"location": "optional: key to group errors by"
}
],
"run_if": "passed",
"location": "optional: key to group errors by"
},
"location": "optional: key to group errors by"
}
],
"location": "optional: key to group errors by"
}
],
"location": "optional: key to group errors by"
}
],
"template": "template1",
"location": "optional: key to group errors by"
}
],
"errors": []
}
The plugin is expected to return a JSON object (corresponding to ParseDirectoryResponseMessage with these top level attributes:
1. The target_version
element
The target_version
element should be set to 2 (static) for all new plugins. The change in the response based on
changes in the value of target_version
are described below:
Version change from … | Change in schema |
---|---|
1 to 2 | The property enable_pipeline_locking was changed to lock_behavior . The old values of true and false in enable_pipeline_locking were changed to lockOnFailure and none respectively in lock_behavior . A new value of unlockWhenFinished was introduced. |
2. The environments
element
The environments
element is a list of environment objects (corresponding to
CREnvironment)
, each of which has a name
, environment_variables
(for the whole environment), agents
in this environment and the
pipelines
in this environment.
See example shown.
3. The errors
element
If this list has any errors defined (corresponding to CRError), the configuration will be considered invalid and the errors will be shown to the user in an appropriate manner.
4. The pipelines
element
A fairly comprehensive response with a pipelines
element is shown in the example. Not all of the parts are
mandatory. Some noteworthy properties of this element are:
- This element corresponds to the CRPipeline object in GoCD.
- name
is mandatory (and should be unique across the GoCD server).
- group
is mandatory. It is the pipeline group that this pipeline should be added to.
- There should be at least one material and material names should be unique across the pipeline.
- There should either be stages
defined or a template
to fetch stages from. Not both.
- Environment variable names and parameter names should be unique across the pipeline.
- The different kinds of materials are shown in the example. They correspond to these objects in the GoCD contract part of the codebase: CRGitMaterial (git), CRSvnMaterial (SVN), CRP4Material (P4), CRHgMaterial (Hg), CRPackageMaterial (Package), CRTfsMaterial (TFS), CRDependencyMaterial (Dependency) and CRPluggableScmMaterial (SCM plugin).
- Jobs cannot have both resources
and elastic_profile_id
defined. They can have one of them or neither define.
- Most top-level attributes of a pipeline can have a location
defined. If defined, that value will be used to
show more meaningful error messages, if the GoCD server detects errors during its parsing and validation.
- The display_order_weight
property will be used to decide the order in which the pipelines will be shown on the GoCD dashboard. If not provided, it defaults to -1
.
- See this note about the special material type: configrepo
.
The special material type: configrepo
Given this config repo XML:
<config-repos>
<config-repo pluginId="plugin.id" id="repo1">
<git url="https://your/config/repo" branch="something" />
</config-repo>
</config-repos>
Instead of using a material like this:
{
"materials": [
{
"url": "https://your/config/repo",
"branch": "something",
"type": "git",
"name": "mygit"
}
]
}
… you can use this:
{
"materials": [
{
"type": "configrepo",
"name": "mygit"
}
]
}
If the type of a material is configrepo
, then the corresponding config repository’s information will be used as that
material. This allows the pipeline definition to not have to repeat the config repository’s material information. This
is most useful when the config repo definition and the rest of the source code are stored together.
The GoCD server clones the material configuration of the current repository (as is in XML) and replaces name,
destination and filters (whitelist/blacklist). Then, it uses the modified clone in place of configrepo
material.
Parse content
This message is a request to the plugin to parse the provided content containing one or more pipeline configurations. GoCD will send this request to the plugin in order to validate the pipeline configurations as part of a preflight check. It is very similar to the parse-directory
request except that it is not dependent on an existing config repo or a specific directory of files.
Request name
parse-content
Request body
Given this yaml content -
pipelines:
mypipeline:
group: simple
materials:
mygit:
git: http://my.example.org/mygit.git
stages:
- mystage:
jobs:
myjob:
tasks:
- exec:
command: make
The plugin will receive the following JSON body —
{
"contents": {
"build.gocd.yml": "contents of build.gocd.yml",
"deploy.gocd.yml": "contents of deploy.gocd.yml"
}
}
The request body will contain a JSON object with an attribute contents
, which contains a map of filenames and the content of each file.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response body
The plugin is expected to return a JSON object (corresponding to
ParseDirectoryResponseMessage). Please reference to parse-drectory
for details and examples of the response body.
Pipeline export
This message is a request to the plugin to convert a pipeline configuration into the plugin’s native format. It is
essentially the reverse of the parse-directory
request.
Request name
pipeline-export
Request body
Given the following config XML snippet (assume this pipeline is in a group named
my-group
) —
<pipeline name="my-pipeline">
<materials>
<git url="https://my.repo/path/to.git" />
</materials>
<stage name="first-stage">
<jobs>
<job name="first-job">
<tasks>
<exec command="/bin/echo">
<arg>hello</arg>
<runif status="passed" />
</exec>
</tasks>
</job>
</jobs>
</stage>
</pipeline>
The request body will contain a JSON object with an attribute pipeline
, which contains a JSON representation of a full pipeline configuration. For a more detailed example showing, see the parse-directory
request body.
The plugin will receive the following JSON body —
{
"pipeline": {
"group": "my-group",
"name": "my-pipeline",
"label_template": "${COUNT}",
"lock_behavior": "none",
"mingle": {
"mql_grouping_conditions": "\"\""
},
"environment_variables": [],
"parameters": [],
"materials": [
{
"url": "https://my.repo/path/to.git",
"branch": "master",
"shallow_clone": false,
"filter": {
"ignore": [],
"whitelist": []
},
"auto_update": true,
"type": "git"
}
],
"stages": [
{
"name": "first-stage",
"fetch_materials": true,
"never_cleanup_artifacts": false,
"clean_working_directory": false,
"approval": {
"type": "success",
"users": [],
"roles": []
},
"environment_variables": [],
"jobs": [
{
"name": "first-job",
"environment_variables": [],
"tabs": [],
"resources": [],
"artifacts": [],
"properties": [],
"run_instance_count": "0",
"tasks": [
{
"command": "/bin/echo",
"timeout": -1.0,
"arguments": [
"hello"
],
"run_if": "passed",
"type": "exec"
}
]
}
]
}
]
}
}
Response example
Here is an example response from the GoCD YAML Configuration Plugin:
HTTP/1.1 200 OK
Content-Type: application/x-yaml; charset=utf-8
X-Export-Filename: my-pipeline.gocd.yaml
{
"pipeline": "\"format_version: 3\\npipelines:\\n my-pipeline:\\n group: my-group\\n label_template: ${COUNT}\\n lock_behavior: none\\n materials:\\n git:\\n git: https://my.repo/path/to.git\\n shallow_clone: false\\n auto_update: true\\n branch: master\\n stages:\\n - first-stage:\\n fetch_materials: true\\n keep_artifacts: false\\n clean_workspace: false\\n approval:\\n type: success\\n jobs:\\n first-job:\\n tasks:\\n - exec:\\n arguments:\\n - hello\\n command: /bin/echo\\n run_if: passed\\n\""
}
Response code
The plugin is expected to return status 200
if it can understand the request.
Response body
The plugin is expected to return a JSON object (corresponding to
PipelineExportResponseMessage
. The response body contains a single attribute:
pipeline
This pipeline
element is a string representation of the exported pipeline configuration in the native format supported by the plugin. The value of this element must deserialize to a string
.
Response headers
The plugin response must also set the following headers:
Content-Type
Plugin responses are expected to provide a MIME type to indicate how to treat the content of the pipeline
attribute. Here are some examples of common MIME types:
text/plain; charset=utf-8
application/json; charset=utf-8
application/x-yaml; charset=utf-8
X-Export-Filename
Plugin responses must also suggest a filename for the exported content. This is used by GoCD to construct a Content-Disposition
header for REST API responses.
Get Config Files
This message is a request to the plugin to parse the given directory and return all the configuration files found.
Request name
config-files
Request body
Given the following config XML snippet —
<config-repos>
<config-repo pluginId="your.plugin.id" id="some-unique-id">
<git url="https://some.repo" />
<configuration>
<property>
<key>some.key</key>
<value>some.value</value>
</property>
</configuration>
</config-repo>
The plugin will receive the following JSON body —
{
"directory": "/absolute/path/to/directory/with/configs",
"configurations": [
{
"key": "some.key",
"value": "some.value"
}
]
}
The request body will contain a JSON object with an attribute directory
, which points to a directory of the specified
material repository. It will also contain an attribute configuration
which contains the specified configuration. Note:
This is not the configuration in the plugin settings page.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response body
An example response body:
{
"target_version": 10,
"files": [
"build.gocd.yml",
"deploy.gocd.yml"
]
}
Get Settings View
This is an optional message that the plugin may implement, should users want to configure the plugin from the GoCD admin page.
Request name
go.plugin-settings.get-view
Request body
The server will not provide a request body.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response Body
A JSON settings view object
An example response body:
{
"template": "<div>some html</div>"
}
Get Plugin Configuration
This is an optional message that the plugin may implement, should users want to configure the plugin from the GoCD admin page. This message allows the server to query a plugin about what properties are supported by this plugin.
Request name
go.plugin-settings.get-configuration
Request body
The server will not provide a request body.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response Body
A JSON plugin settings configuration object.
Validate Plugin Configuration
If a plugin requires any configuration, this message must be implemented in order to validate the configuration.
Request name
go.plugin-settings.validate-configuration
Request body
An example validation request body
{
"plugin-settings": {
"server_url": {
"value": "http://localhost.com"
},
"username": {
"value": "user"
},
"password": {
"value": "password"
}
}
}
The request body will contain a JSON with an attribute plugin-settings
, which contains an object with the configuration keys and values that the plugin is expected to validate.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response Body
The plugin should respond with JSON array response for each configuration key that has a validation error
[
{
"key": "server_url",
"message": "Server URL cannot be localhost"
}
]
If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the configuration is valid, the plugin should return an empty JSON array.
Get Plugin Icon
This call is expected to return the icon for the plugin, so as to make it easy for users to identify the plugin.
Request name
get-icon
Request body
The server will not provide a request body.
Response code
The plugin is expected to return status 200
if it can understand the request.
Response Body
An example plugin response body:
{
"content_type": "image/svg+xml",
"data": "PHN2ZyB2ZXJzaW9u..."
}
The plugin is expected to return an image object.
Get Plugin Capabilities
This message is a request to the plugin to provide plugin capabilities. Based on these capabilities GoCD would enable or disable the plugin features for a user.
Request name
get-capabilities
Request body
Server sends request with Empty
request body.
Response Body
An example response body:
{
"supports_pipeline_export": true,
"supports_parse_content": true,
"supports_list_config_files": false,
"supports_user_defined_properties": true
}
The response body will contain the following JSON attributes:
Key | Type | Description |
---|---|---|
supports_pipeline_export |
boolean |
Whether this plugin supports the pipeline-export request. |
supports_parse_content |
boolean |
Whether this plugin supports the parse-content request. |
supports_list_config_files |
boolean |
Whether this plugin supports the config-files request. |
supports_user_defined_properties |
boolean |
Whether this plugin supports defining of configuration properties by the user. |
The plugin is expected to return status 200
if it can understand the request.
Requests to the GoCD server
The plugin may make the following requests to the server using GoApplicationAccessor#submit(GoApiRequest)
Get Plugin Settings
import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;
@Extension
public class SimpleConfigRepositoryPlugin implements GoPlugin {
private GoApplicationAccessor accessor;
public static final Logger LOG = Logger.getLoggerFor(SimpleConfigRepositoryPlugin.class);
public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
this.accessor = accessor;
}
public GoPluginIdentifier pluginIdentifier() {
return new GoPluginIdentifier("configrepo", Arrays.asList("1.0"))
}
private PluginSettings getSettings() {
Gson gson = new Gson();
// create a request
DefaultGoApiRequest request = new DefaultGoApiRequest(
"go.processor.plugin-settings.get",
"1.0",
pluginIdentifier()
);
// set the request body
Map<String, String> map = new HashMap<>();
map.put("plugin-id", "com.example.rocket.launcher");
request.setRequestBody(gson.toJson(map));
// submit the request
GoApiResponse response = accessor.submit(request);
// check status
if (response.responseCode() != 200) {
LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
}
// parse the response, using a json parser of your choice
return gson.fromJson(response.responseBody(), PluginSettings.class);
}
}
This messages allows a plugin to query the server to get the user configured settings for this plugin.
Request name
go.processor.plugin-settings.get
Request version
The request version must be set to 1.0
.
Request body
An example request body:
{
"plugin-id": "sample-plugin-id"
}
Must be a JSON object with a key plugin-id
with the value being the ID of your plugin.
Response code
The server is expected to return status 200
if it could process the request.
Response Body
An example response body:
{
"server_url": "https://build.go.cd",
"username": "view",
"password": "password"
}
The server will send a map of settings.
Request/Response JSON Objects
The Settings View Object
Here’s an example of the settings view object:
{
"template": "<div class=\"form_item_block\">...</div>"
}
Attribute | Type | Description |
---|---|---|
template |
String | A string containing an HTML AngularJS based view. |
This template is an AngularJS based template.
GoCD uses Angular JS as its template engine for the plugin UI. This allows plugin authors to use a limited set of AngularJS features to specify how the UI of their plugins looks.
Getting started with AngularJS based templates
Given a configuration:
<configuration>
<property>
<key>username</key>
<value>alice</username>
</property>
</configuration>
This gets converted into the following JSON representation in the browser:
{
"username": "alice"
}
The AngularJS template is expected to bind to the JSON object shown above:
<div class="form_item_block">
<label>Username:<span class='asterix'>*</span></label>
<input ng-model="username" />
</div>
When an Angular template is used in a Go plugin, to define the configuration UI, the configuration key which is stored in the configuration XML is used everywhere and is expected to be consistent. Since Angular works off of JSON, GoCD will make sure that the key in the JSON provided to the Angular template is the same as the key in the configuration XML.
Suppose the key of the configuration property stored in the XML is “username”, with value, “alice”, then Go will make sure that the value is available to the template as “username” when used in an Angular-specific HTML attribute like “ng-model”.
So, the name “foobar” needs to be the same across the configuration XML, the Angular template as well as in any code that the plugin has.
Showing validation errors in the UI
We use some simple string replacement
to substituteGOINPUTNAME
with a unique identifier
for your plugin in order to render
any server side errors
<div class="form_item_block">
<label>Username:<span class='asterix'>*</span></label>
<input ng-model="username" />
<span class="form_error" ng-show="GOINPUTNAME[username].$error.server">
{{ GOINPUTNAME[username].$error.server}}
</span>
</div>
In case of validation errors returned by go.plugin-settings.validate-configuration
, the error messages needs to be populated on the UI, use the snippet here to show the validation errors.
The Plugin Settings Configuration Object
Here’s an example of the plugin settings configuration object:
{
"server_url": {
"display-name": "Server URL",
"display-order": "0"
},
"username": {
"required": false,
"display-name": "Username",
"display-order": "1"
},
"password": {
"secure": true,
"required": false,
"display-name": "Password",
"display-order": "2"
}
}
Attribute | Type | Description |
---|---|---|
display-name |
String | The name of the property. |
default-value |
String | The default value of the property. |
display-order |
String | A string containing a numerical value. |
required |
Boolean | If the field is mandatory. |
secure |
Boolean | If the data in the field should be stored encrypted. |
The Validation Error Object
Here’s an example of the validation error object:
[
{
"key": "email_address",
"message": "Email address is invalid"
},
{
"key": "password",
"message": "Password must be provided"
}
]
Attribute | Type | Description |
---|---|---|
key |
String | The name of configuration key that has an error. |
message |
String | The error message associated with that key. |
The Image Object
Here’s an example of the image object:
{
"content_type": "image/svg+xml",
"data": "...."
}
Attribute | Type | Description |
---|---|---|
content_type |
String | A valid content type for the image. Please make sure the content type is supported by most browsers. |
data |
String | A base-64 encoded (single-line non-chunking) byte array of the byte-sequence that composes the image. |