Skip to content

Configuration Manual for the "Volcengine-ES-CLOUD" Collector

Before reading this article, please read the following first:

Before using this collector, you must install the 'Integration Core Package' and its associated third-party dependency packages

1. Configuration Structure

The configuration structure of this collector is as follows:

Field Type Required Description
regions list Required List of regions to collect data from
regions[#] str Required Region ID. For example: 'cn-shanghai'
Refer to the appendix for the full list

2. Configuration Example

Specify Regions

Collect data from the Beijing region.

Python
1
2
3
collector_configs = {
    'regions': [ 'cn-beijing' ]
}

Configure Filters (Optional)

This collector script supports user-defined filters, allowing users to filter target resources based on object attributes. The filter function returns True or False.

  • True: The target resource needs to be collected.
  • False: The target resource does not need to be collected.

Supported object attributes for filtering:

Attribute Description
InstanceId Instance ID
InstanceName Instance name
Status Status
UserId User ID
ClusterId ID of the cluster the instance belongs to
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# Example: Enable filters, filter based on InstanceId and name properties, configuration format as follows:
def filter_instance(instance):
    '''
    Collect instances where instance_id is xxxxx or instance_name is xxxxx
    '''
    # return True
    instance_id = instance['InstanceId']
    instance_name = instance['name']
    if instance_id in ['xxxxx'] or instance_name in ['xxxxx']:
        return True
    return False


###### Do not modify the following contents #####
from guance_integration__runner import Runner
import guance_volcengine_es_cloud__main as main


@DFF.API('Volcengine-ES-Cloud Collection', timeout=3600, fixed_crontab='*/15 * * * *')
def run():
    Runner(main.DataCollector(account, collector_configs, filters=[filter_instance])).run()

3. Data Reporting Format

After data synchronization, you can view the data in the "Infrastructure - Resource Catalog" section of {{( brand_name }}}.

Example of reported data:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
{
  "measurement": "volcengine_es_cloud",
  "tags": {
    "CerebroEnabled": "False",
    "CerebroPrivateDomain": "",
    "CerebroPublicDomain": "",
    "ClusterId": "xxx",
    "ESEipId": "",
    "ESEip": "xxx",
    "EnableESPrivateDomainPublic": "False",
    "EnableESPrivateNetwork": "True",
    "EnableESPublicNetwork": "False",
    "EnableKibanaPublicNetwork": "",
    "EnableKibanaPrivateNetwork": "",
    "EnableKibanaPrivateDomainPublic": "False",
    "InstanceId": "o-xxx",
    "KibanaEip": "",
    "KibanaEipId": "",
    "KibanaPrivateDomain": "https://visual-xxxx:5601",
    "KibanaPublicDomain": "",
    "RegionId": "cn-beijing",
    "Status": "Running",
    "UserId": "2100xxx",
    "name": "o-xxxx"
  },
  "fields": {
    "TotalNodes": "2",
    "ChargeEnabled": "True",
    "DeletionProtection": "False",
    "CreateTime": "2024-04-03T15:17:41+08:00",
    "ESPrivateEndpoint": "https://xxxx",
    "ESPrivateIpWhitelist": "",
    "ESPublicEndpoint": "",
    "ESPublicIpWhitelist": "",
    "ExpireDate": "",
    "KibanaConfig": "{Kibana parameter configuration}",
    "KibanaPrivateIpWhitelist": "",
    "KibanaPublicIpWhitelist": "",
    "MaintenanceDay": "[Instance's maintenance date]",
    "ResourceTags": "[Instance's tag information]",
    "SubInstances": "[Enterprise-level SQL analysis instance configuration information]",
    "TransferInfo": "{Data migration task information}",
    "InstanceConfiguration": "{Instance detailed configuration information.}",
    "message": "{Instance JSON data}"
  }
}

Fields in tags and fields may change with subsequent updates

The value of tags.name is the instance ID, used as a unique identifier. fields.message, fields.KibanaConfig, fields.MaintenanceDay, fields.ResourceTags, and fields.InstanceConfiguration are all JSON serialized strings.

X. Appendix

Please refer to the official Volcengine documentation: