Skip to content

Configuration Manual for the "GoogleCloud-Cloud Monitoring" Collector

Before reading this, please first review:

Before using this collector, you must install the 'Integration Core Package' and its corresponding third-party dependency packages.

This collector supports multi-threading by default (five threads are enabled by default). If you need to change the thread pool size, set the environment variable COLLECTOR_THREAD_POOL_SIZE.

1. Configuration Structure

The configuration structure of this collector is as follows:

Field Type Required Description
targets list Required Cloud monitoring collection object configuration list.
Logical relationship between multiple configurations with the same namespace is "AND".
targets[#].namespace str Required The cloud monitoring namespace to be collected. For example: 'compute.googleapis.com'.
targets[#].metrics list Required List of cloud monitoring metric names to be collected.
targets[#].metrics[#] str Required Metric name pattern, supporting "NOT" and wildcard matching.
In normal cases, logical relationship between multiple ones is "OR".
When containing "NOT" marker, logical relationship between multiple ones is "AND".
See details below.

1.1 Metric Types

Before understanding the collector configuration structure, we first look at GCP's metric types (Metric Type). A complete metric type consists of {namespace}/{metric}, for example:

Compute Engine CPU usage rate: compute.googleapis.com/instance/cpu/utilization

  • namespace: compute.googleapis.com

namespace

  • metric: instance/cpu/utilization

metric

Source of the above image: Google Cloud Metrics

1.2 Resource Types

Cloud monitoring metrics can be described as Google Cloud x service x resources x metrics, where x resources are resource types (resource_type).

resource-type

The collector configuration does not require specifying a resource type. Note that most measurement sets (measurement) reported to Guance are named in the format of gcp_{resource_type} (some resource types may also be renamed).

The resource type of a metric is not mandatory because some metrics are service-level and do not specifically refer to the resource level. In this case, the measurement reported to Guance will be gcp_{service}, where service is the prefix part of the namespace: {service}.googleapis.com.

2. Configuration Examples

Specifying Specific Metrics

Collect two metrics named instance/cpu/utilization and guest/disk/io_time from Compute Engine.

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'instance/cpu/utilization',
              'guest/disk/bytes_used'
            ]
        }
    ]
}

Wildcard Matching Metrics

Metric names can use * as a wildcard for matching.

In this example, the following metrics will be collected:

  • Metrics named instance/cpu/utilization
  • Metrics whose names start with instance
  • Metrics whose names end with cpu/utilization
  • Metrics whose names contain cpu
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'instance/cpu/utilization',
              'instance*',
              '*cpu/utilization',
              '*cpu*'
            ]
        }
    ]
}

Excluding Certain Metrics

Adding the "NOT" marker at the beginning indicates that the subsequent metrics should be excluded.

In this example, the following metrics will not be collected:

  • Metrics named instance/cpu/utilization
  • Metrics whose names start with instance
  • Metrics whose names end with cpu/utilization
  • Metrics whose names contain cpu
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : [
              'NOT',
              'instance/cpu/utilization',
              'instance*',
              '*cpu/utilization',
              '*cpu*'
            ]
        }
    ]
}

Multiple Filtering for Desired Metrics

The same namespace can be specified multiple times, filtering metrics sequentially from top to bottom based on their names.

In this example, it is equivalent to performing the following filtering steps on metric names:

  1. Select all metrics whose names contain cpu.
  2. From the results of the previous step, exclude metrics whose names are cpu/utilization.
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['*cpu*'],
        },
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['NOT', '*cpu/utilization'],
        }
    ]
}

Configuring Filters (Optional)

This collector script supports user-defined filters, allowing users to filter out the data they want to report through the tags of the reported data. Filter functions return True or False.

  • True: Target resources need to be collected.
  • False: Target resources do not need to be collected.
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# Example: Enable filters, filter by InstanceId and RegionId properties, configuration format as follows:

def filter_instance(instance, namespace='compute.googleapis.com'):
    '''
    Collect metrics for instance_id i-xxxxxa, i-xxxxxb
    '''
    # print(instance) # Consider printing instance for the first time to check the data structure to be filtered
    instance_id = instance['tags'].get('instance_id')
    if instance_id in ['i-xxxxxa', 'i-xxxxxb']:
        return True
    return False

from guance_integration__runner import Runner
import guance_aliyun_monitor__main as main

@DFF.API('GoogleCloud-Monitor Collection', timeout=300, fixed_crontab="* * * * *")
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

Filters apply to already collected data, so they will not reduce the number of calls to the Cloud Monitoring API.

3. Data Reporting Format

After data synchronization, you can view the data under the 'Metrics' section in TrueWatch.

For example, consider the following collector configuration:

Python
1
2
3
4
5
6
7
8
collector_configs = {
    'targets': [
        {
            'namespace': 'compute.googleapis.com',
            'metrics'  : ['instance/cpu/utilization'],
        }
    ]
}

An example of the reported data is as follows:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "measurement": "gcp_gce_instance",
  "tags": {
    "instance_id": "i-xxxxx",
    "userId"    : "xxxxx"
  },
  "fields": {
    "instance_cpu_utilization": 1.23
  }
}
  • measurement: Field definition refers to the '1.2 Resource Types' chapter.
  • tags: Dimensions corresponding to the metric, along with additional tags.
  • fields: The metrics configured in metrics (all converted to lowercase letters with underscores before reporting).

Since the metrics are obtained in real-time, only the latest value is reported, thus omitting timestamps (default timestamp is the reporting time).

All metric values are reported as float type.

4. Explanation of Cloud Monitoring API Call Counts

Google Cloud Cloud Monitoring has free quota limits for certain API calls. Before October 1, 2025: Each billing account gets a free quota of 1 million queries per month, with excess charged at $0.01 per thousand queries. The /v3/project/{project_id}/timeSeries used by this collector is also within this limit:

Find the actual call count by viewing task execution logs

The collector tracks the number of API calls made during each task execution, which can be viewed in the logs, for example:

Bash
1
2
3
4
[2023-04-21 15:32:13.194] [+0ms] Completed collection for the 【1】st account, total execution time 【274 milliseconds】, called APIs 【2 times】
[2023-04-21 15:32:13.194] [+0ms] Detailed calls as follows:
[2023-04-21 15:32:13.194] [+0ms] -> monitoring.googleapis.com/v3/projects/{project_id}/metricDescriptors: 1 time
[2023-04-21 15:32:13.194] [+0ms] -> monitoring.googleapis.com/v3/project/{project_id}/timeSeries: 1 time

After October 2, 2025, billing will no longer be based on API call counts but instead offer the first 1 million time series free for each settlement account, charging $0.50 for every 1 million time series thereafter. For more details, see: Google Cloud Observability Pricing

Given the free quota for cloud monitoring calls, it is recommended that users configure monitoring items as needed to avoid additional costs due to wildcards.

X. Appendix

Refer to official documentation: