Skip to content

GitLab CI Integration Configuration

1. Background

GitLab is a DevOps platform that automates the software development lifecycle through its integrated CI/CD functionality.

This article explains how to use DataFlux Func to collect CI Pipeline execution data and report it to the Guance/TrueWatch platform. Through the CI visualization features of Guance, you can analyze bottlenecks and optimize deployment processes.

2. Prerequisites

  • Register/Login to Guance/TrueWatch
  • Enable Func Managed Edition or self-install DataFlux Func GSE edition

3. Configuration Process

  1. Install GitLab CI integration from the official script market.
  2. Create a "Synchronization API" (Old version: Authorization Link).
  3. Configure webhook on the GitLab platform.

3.1 Install GitLab CI Integration

Learn Basic Operations of Script Market

Install GitLab CI integration via the official script market (Script ID: guance_gitlab_ci).

Learn Script Development / Basic Concepts

Learn Synchronization API

3.2.1 Create Function

Users can create their own script sets and scripts, copy the following code into the script, and click publish.

Python
1
2
3
4
5
6
import guance_gitlab_ci__main as main

@DFF.API('Receive GitLab CI Event', timeout=10)
def runner(**kwargs):
    response = main.run(kwargs)
    return response

3.2.2 Specify Reporting Data Source

Please ensure that connectors associated with the Guance/TrueWatch workspace are created in advance in your Func (ignore this step if you have enabled the Func Managed Edition).

Learn Connectors

Users can specify the connector ID to report data to the associated Guance workspace. Currently supported connectors include: Guance, DataKit, DataWay, for example:

Python
1
2
3
4
5
6
7
8
import guance_gitlab_ci__main as main

@DFF.API('Receive GitLab CI Event', timeout=10)
def runner(**kwargs):
    # Report data to the workspace associated with the Guance connector ID: xxx
    response = main.run(kwargs, guance_id='xxx')
    # response = main.run(kwargs, datakit_id=None, dataway_id=None, guance_id=None, workspace_token=None)
    return response

Note: Only one connector will take effect, do not specify multiple ones.

If multiple ones are specified, only one will take effect, with priority from high to low as follows:

  1. Guance connector, parameter: guance_id
  2. DataKit connector, parameter: datakit_id
  3. DataWay connector, parameter: dataway_id

Specifying connector parameters is not mandatory. If not specified, the program will automatically select a connector, with priority from high to low as follows:

  1. Guance connector with ID guance
  2. DataKit connector with ID datakit
  3. DataWay connector with ID dataway

3.2.3 Create "Synchronization API"

Create a "Synchronization API" by selecting Execute the function created in the previous step, then click save.

Click the Example of the "Synchronization API".

Copy the simplified form of the GET URL for later use.

3.3 Configure Webhook on the GitLab Platform

Navigate to the target repository Settings > Webhooks and add a new webhook:

  • URL: Paste the copied "Synchronization API" URL from the previous step.
  • Trigger: Select Job events and Pipeline events.

Additional Configuration Options

Configure Custom Tags

Note: If custom tags conflict with key fields in Pipeline or Job pushes, they will be ignored. Refer to the 'Data Description' section below for key fields.

Configure custom tags using config.extra_tags, as follows:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import guance_gitlab_ci__main as main

config = {
    'extra_tags': {
        'key1': 'value1',
        'key2': 'value2'
    }
}

@DFF.API('Receive GitLab CI Event', timeout=10)
def runner(**kwargs):
    response = main.run(kwargs, config=config)
    return response

Data Description

GitLab visualization data is stored in log format and can be viewed in the 'CI Visualization' section of Guance/TrueWatch, or you can view logs by specifying the data source (source).

The following fields do not include user-defined fields.

GitLab Pipeline

Data Source: gitlab_pipeline

Name Description Type Unit
author_email Author email string
ci_status CI type string
commit_sha The commit SHA of the most recent commit of the code that triggered the Pipeline string
object_kind Event type, in this case Pipeline string
operation_name Operation name string
pipeline_name Pipeline name string
pipeline_source Sources of Pipeline triggers string
pipeline_url Pipeline URL string
ref Branches involved string
repository_url Repository URL string
resource Project name
commit_message The message attached to the most recent commit of the code that triggered the Pipeline. string
created_at Millisecond timestamp of Pipeline creation int msec
duration Pipeline duration (microseconds) int μs
finished_at Millisecond timestamp of the end of the Pipeline int msec
message The message attached to the most recent commit of the code that triggered the Pipeline. Same as commit_message string
pipeline_id Pipeline id string
gitlab_version GitLab Version string

GitLab Job

Data Source: gitlab_job

Name Description Type Unit
build_commit_sha The commit SHA corresponding to build string
build_failure_reason Build failure reason string
build_name Build name string
build_repo_name Repository name corresponding to build string
build_stage Build stage string
build_status Build status string
object_kind Event type, in this case Job string
project_name Project name string
sha The commit SHA corresponding to build string
user_email User email string
build_commit_message The message attached to the most recent commit of the code that triggered the build string -
build_duration Build duration (microseconds) int μs
build_finished_at Millisecond timestamp of the end of build int msec
build_id build id string -
build_started_at Millisecond timestamp of the start of build int msec
message The message attached to the most recent commit of the code that triggered the build. Same as build_commit_message string -
pipeline_id Pipeline id for build string -
project_id Project id for build string -
runner_id Runner id for build string -
gitlab_version GitLab Version string

Troubleshooting

After running GitLab CI/CD for some time, no relevant data appears in the Guance workspace.

Troubleshooting ideas:

  1. First confirm whether the connector configuration is correct.
  2. Check the related logs of the "Synchronization API" for errors. If there are no logs, it means it hasn't been successfully triggered. In the GitLab platform, find the target webhook and click edit. On the edit page, check if there are records under Recent Deliveries. If there are records and no abnormalities, verify that the URL matches the "Synchronization API" URL.

If logs exist, troubleshoot based on the log information. Contact the script owner if necessary.