Skip to content

Configuration Manual for the "Alibaba Cloud - Elasticsearch Logs" Collector

Before reading this article, please first read:

Before using this collector, you must install the 'Integration Core package' and its corresponding third-party dependency packages

This script's code execution depends on elasticsearch instance object collection. If custom object collection for elasticsearch is not configured, the log script will not be able to collect log data

1. Configuration Structure

The configuration structure of this collector is as follows:

Field Type Required Description
regions list Required List of regions to collect data from
regions[#] str Required Region ID. Example: 'cn-hangzhou'
Refer to appendix for full table
log_types list Required List of log types to collect
log_types[#] str Required Log type. Example: 'JVMLOG'
Details in the table below

Assigning values to log_types (Log Types):

Value Description
INSTANCELOG Main logs
SEARCHSLOW Searching slow logs
INDEXINGSLOW Indexing slow logs
JVMLOG GC logs
ES_SEARCH_ACCESS_LOG ES access logs
AUDIT Audit logs

2. Configuration Example

Specify Region

Collect data from Hangzhou region

Python
1
2
3
4
collector_configs = {
    'regions': [ 'cn-hangzhou' ],
    'log_types': ['JVMLOG']
}

3. Data Reporting Format

After data synchronization, you can view the data under 'Infrastructure - Resource Catalog' in Guance or TrueWatch.

Example of reported data:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
  "measurement": "aliyun_elasticsearch_log",
  "tags": {
    "RegionId"       : "cn-hangzhou",
    "esVersion"      : "7.10.0_with_X-Pack",
    "host"           : "10.14.xxx.xxx",
    "instanceId"     : "es-cn-xxxx",
    "name"           : "es-cn-xxxx",
    "paymentType"    : "prepaid",
    "resourceGroupId": "rg-aekzkcwe4dxxxx",
    "serviceVpc"     : "True",
    "status"         : "active"
  },
  "fields": {
    "timestamp"        : 1684304299000,
    "contentCollection": "[ {JSON data with detailed log information}, ...]",
    "message"          : "{Instance JSON data}"
  }
}

Fields in tags and fields may change with subsequent updates

The value of tags.name is the instance ID, used as a unique identifier, and fields.message is a string serialized in JSON format

X. Appendix

Please refer to the official Alibaba Cloud documentation: