Skip to content

Collector "Huawei Cloud-Kafka" Configuration Manual

Before reading this article, please read the following first:

Before using this collector, you must install the 'Integration Core Package' and its corresponding third-party dependency packages

1. Configuration Structure

The configuration structure of this collector is as follows:

Configuration One (Recommended)

Field Type Required Description
regions list Required List of regions to collect data from
regions[#] str Required Region ID. For example: 'cn-north-4'
See appendix for full list

The collector will automatically fetch all IAM projects under the region, then fetch resources based on IAM projects

Configuration Two (Legacy Compatibility)

Field Type Required Description
region_projects dict Required List of "Region - Project ID" pairs for required data collection
region_projects[#] str:list Required In key-value pairs:
Key represents the region (e.g., 'cn-north-4')
Value represents the list of project IDs to collect data from within that region
See appendix for full list

2. Configuration Example

Python
1
2
3
configs = {
    'regions': ['cn-north-4']
}

Configuration Filter (Optional)

This collector script supports user-defined filters, allowing users to filter target resources by object attributes. The filter function returns True｜False.

  • True: The target resource needs to be collected.

  • False: The target resource does not need to be collected.

Supported object attributes for filtering:

Attribute Description
engine Instance engine
status Instance status
instance_id Instance ID
port Port
type Instance type: cluster, cluster
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
def filter_instance(instance):
    '''
    Collect instances with instance id xxxx
    '''
    instance_id = instance.get('instance_id')
    if instance_id in ['xxx']:
        return True
    return False


###### Do not modify the following contents ######
from guance_integration__runner import Runner
import guance_huaweicloud_kafka__main as kafka_main
import guance_huaweicloud_ces__main as ces_main


@DFF.API('HuaweiCloud Kafka Collection', timeout=3600, fixed_crontab='* * * * *')
def run():
    collectors = [
        kafka_main.DataCollector(account, collector_configs, filter_instance=filter_instance),
    ]
    Runner(collectors).run()

3. Data Reporting Format

After data synchronizes successfully, you can view the data in the {{( brand_name )}} "Infrastructure - Resource Catalog".

Example of reported data:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "measurement": "huaweicloud_kafka",
  "tags": {
    "RegionId"             : "cn-north-4",
    "project_id"           : "f5f4c067d68xxxx86e173b18367bf",
    "enterprise_project_id": "0",
    "name"                 : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "instance_id"          : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "instance_name"        : "xxx",
    "status"               : "RUNNING",

  },
  "fields": {
    "charging_mode"        : "1",
    "connect_address"      : "192.168.0.161,192.168.0.126,192.168.0.31",
    "description"          : "",
    "engine"               : "kafka",
    "engine_version"       : "2.7",
    "kafka_manager_user"   : "",
    "name"                 : "beb33e02-xxxx-xxxx-xxxx-628a3994fd1f",
    "port"                 : "9092",
    "resource_spec_code"   : "",
    "service_type"         : "advanced",
    "specification"        : "kafka.2u4g.cluster.small * 3 broker",
    "storage_type"         : "hec",
    "user_id"              : "e4b27d49128e4bd0893b28d032a2e7c0",
    "user_name"            : "xxxx",
    "created_at"           : "1693203968959",
    "maintain_begin"       : "02:00:00",
    "maintain_end"         : "06:00:00",
    "storage_space"        : 186,
    "total_storage_space"  : 300,
    "message"              : "{Instance JSON data}"
  }
}

Descriptions of some fields are as follows:

Field Type Description
specification String Instance specification.
charging_mode String Billing mode, 1 indicates pay-as-you-go billing, 0 indicates annual/monthly subscription billing.
created_at String Completion creation time. Timestamp format, representing the total milliseconds deviation from Greenwich Mean Time January 1, 1970, 00:00:00 UTC.
resource_spec_code String Resource specification
maintain_begin String Start time of maintenance window, format HH:mm:ss
maintain_end String End time of maintenance window, format HH:mm:ss

Fields in tags and fields may change with subsequent updates

The value of tags.name is the instance ID, used as a unique identifier

fields.message is a string serialized in JSON format

X. Appendix

Refer to the official Huawei Cloud documentation: