Configuration Manual for the "Alibaba Cloud - Elasticsearch Logs" Collector

Before reading this article, please read the following first:

TrueWatch Integration

Before using this collector, you must install the 'Integration Core Package' and its associated third-party dependency packages

This script's code execution depends on elasticsearch instance object collection. If custom elasticsearch object collection is not configured, the log script will be unable to collect log data

1. Configuration Structure

The configuration structure of this collector is as follows:

Field	Type	Required	Description
`regions`	list	Required	List of regions to collect data from
`regions[#]`	str	Required	Region ID. Example: `'cn-hangzhou'` See appendix for full list
`log_types`	list	Required	List of log types to collect
`log_types[#]`	str	Required	Log type. Example: `'JVMLOG'` See table below for details

Meanings of values assigned to log_types (Log Types):

Value	Description
`INSTANCELOG`	Main logs
`SEARCHSLOW`	Searching slow logs
`INDEXINGSLOW`	Indexing slow logs
`JVMLOG`	GC logs
`ES_SEARCH_ACCESS_LOG`	ES access logs
`AUDIT`	Audit logs

2. Configuration Example

Specifying a Region

Collect data from Hangzhou region

Python
collector_configs = {
    'regions': [ 'cn-hangzhou' ],
    'log_types': ['JVMLOG']
}

3. Data Reporting Format

After data synchronization, you can view the data in the {{( brand_name )}} 'Infrastructure - Resource Catalog'.

An example of the reported data is as follows:

JSON
{
  "measurement": "aliyun_elasticsearch_log",
  "tags": {
    "RegionId"       : "cn-hangzhou",
    "esVersion"      : "7.10.0_with_X-Pack",
    "host"           : "10.14.xxx.xxx",
    "instanceId"     : "es-cn-xxxx",
    "name"           : "es-cn-xxxx",
    "paymentType"    : "prepaid",
    "resourceGroupId": "rg-aekzkcwe4dxxxx",
    "serviceVpc"     : "True",
    "status"         : "active"
  },
  "fields": {
    "timestamp"        : 1684304299000,
    "contentCollection": "[ {JSON data with detailed log information}, ...]",
    "message"          : "{Instance JSON data}"
  }
}

Fields in tags and fields may change with subsequent updates

The value of tags.name is the instance ID, used as a unique identifier. fields.message is a string serialized in JSON format.

X. Appendix

Please refer to the official Alibaba Cloud documentation: