Skip to content

Collector "AWS-CloudWatchLogs" Configuration Manual

Before reading this article, please read the following first:

Before using this collector, you must install the 'Integration Core Package' and its associated third-party dependency packages

1. Configuration Structure

The configuration structure of this collector is as follows:

Field Type Required Description
region_id str Required Region ID, for example: 'cn-north-1'
targets list Required CloudWatchLogs collection object configuration list
The logical relationship between multiple configurations of the same log group is "AND".
targets[#].log_group_name str Required The required CloudWatch namespace to collect. For example: 'AWS/EC2'. Refer to the appendix for the full table.
targets[#].log_streams list Required List of CloudWatch log streams to collect
targets[#].log_streams[#] str Required Log stream name pattern, supports "NOT" and wildcard matching
Normally, the logical relationship between multiple patterns is "OR". When including "NOT", the logical relationship is "AND". See below for details.

2. Configuration Examples

Specifying Specific Log Streams

Collecting the log stream named database-1 from /aws/rds/instance/database-1/slowquery

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['database-1']
      }
    ]
  }
]

Collecting All Log Streams

Collect all log streams under the /aws/rds/instance/database-1/slowquery log group.

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['ALL']
      }
    ]
  }
]

Wildcard Matching Log Streams

Log stream names can use * as a wildcard for matching.

In the following example, the following log streams will be collected:

  • Log stream name test

  • Log stream names starting with database

  • Log stream names ending with query

  • Log stream names containing slow

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['test', 'database*', '*query', '*slow*'],
      }
    ]
  }
]

Excluding Certain Log Streams

Adding the "NOT" marker at the beginning indicates that the subsequent log streams will be excluded.

In the following example, the following log streams will not be collected:

  • Log stream name test

  • Log stream names starting with database

  • Log stream names ending with query

  • Log stream names containing slow

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['NOT', 'test', 'database*', '*query', '*slow*'],
      }
    ]
  }
]

Multi-Filtering to Specify Desired Log Streams

The same log group can be specified multiple times, filtering log stream names sequentially from top to bottom.

In this example, it is equivalent to performing the following filtering steps on the log stream names:

  1. Select all log streams whose names contain database

  2. From the result of the previous step, remove the log stream named database-2

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['*database*'],
      },
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['NOT', 'database-2'],
      }
    ]
  }
]

3. Data Reporting Format

After data is synchronized normally, it can be viewed in the "LOGS" section of TrueWatch.

For example, consider the following collector configuration:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
config = [{
  'region_id': 'cn-north-1',
  'targets': [
      {
        'log_group_name': '/aws/rds/instance/database-1/slowquery',
        'log_streams'   : ['database-1']
      }
    ]
  }
]

An example of reported data is as follows:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "measurement": "aws_cloudwatch_logs_/aws/rds/instance/database-1/slowquery",
  "tags": {
    "logStreamName"       : "database-1",
    "errorCode"           : "InvalidRequestException",
    "accessKeyId"         : "xxx",
    "userIdentityUserName": "xxx"
  },
  "fields": {
    "message"     : "# Time: 2023-02-28T02:37:44.959003Z\n# User@Host: root[root] @  [xxx.xxx.xxx.xxx]  Id:     9\n# Query_time: 1.995580  Lock_time: 0.000002 Rows_sent: 100000  Rows_examined: 100000\nuse aa;\nSET timestamp=1677551862;\nselect * from t1 limit 100000;",
    "timestamp"   : 1677551864959,
    "errorMessage": "DataCatalog AwsDataCatalog was not found"
  }
}

Fields in tags and fields may change with subsequent updates

The value of tags.name is the instance ID, serving as a unique identifier, and fields.message is a JSON serialized string

4. Coordination with Custom Object Collector

Currently only supports RDS custom object supplements. This collector will obtain the DBInstanceIdentifier field based on the log group name, attempting to match the tags.DBInstanceIdentifier field in custom objects.

When a successful match occurs, additional fields from the matched custom object's tags are added to the tags of the CloudWatchlogs data. The specific effect is as follows:

Assume the original data collected by CloudWatchlogs is as follows:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  "measurement": "aws_cloudwatch_logs_/aws/rds/instance/database-1/slowquery",
  "tags": {
    "logStreamName"       : "database-1",
    "errorCode"           : "InvalidRequestException",
    "accessKeyId"         : "xxx",
    "userIdentityUserName": "xxx"
  },
  "fields": {
    "message"     : "# Time: 2023-02-28T02:37:44.959003Z\n# User@Host: root[root] @  [xxx.xxx.xxx.xxx]  Id:     9\n# Query_time: 1.995580  Lock_time: 0.000002 Rows_sent: 100000  Rows_examined: 100000\nuse aa;\nSET timestamp=1677551862;\nselect * from t1 limit 100000;",
    "timestamp"   : 1677551864959,
    "errorMessage": "DataCatalog AwsDataCatalog was not found"
  }
}

At the same time, the custom object data collected by the AWS RDS collector is as follows:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
  "measurement": "aws_rds",
  "tags": {
    "DBInstanceIdentifier": "database-1",
    "DBInstanceClass"     : "xxx",
    "{other fields}"      : "{omitted}"
  },
  "fields": {
    "{other fields}": "{omitted}"
  }
}

Then, the final reported CloudWatchlogs data is as follows:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "measurement": "aws_cloudwatch_logs_/aws/rds/instance/database-1/slowquery",
  "tags": {
    "logStreamName"       : "database-1",     // Original field from CloudWatchlogs
    "DBInstanceIdentifier": "database-1",     // Field from the custom object RDS
    "DBInstanceClass"     : "xxx",            // Field from the custom object RDS
    "{other fields}"      : "{omitted}"
  },
  "fields": {
    "{other fields}": "{omitted}"
  }
}

6. IAM Policy Permissions

If the user collects resources by assuming an IAM role, certain operational permissions need to be enabled

This collector requires the following operational permissions:

logs:DescribeLogStreams

logs:FilterLogEvents

X. Appendix

AWS CloudWatchLogs

Refer to the official AWS documentation: