Skip to content

Collector "Logstash" Configuration Manual

Before using this collector, you must install 'Logstash' and ensure it is running properly.

1. Configuration Example

1.1 Configure the logstash collector URL in the script file

Python
1
2
# Configure logstash collector URL
LOGSTASH_URL = "http://localhost:9600/_node/stats/pipelines?pretty"

The user needs to configure the pipeline.id field in pipelines.yml to distinguish the pipelines that need to be collected

Access address:

Text Only
1
2
# The user needs to configure the corresponding port; multiple instances share one collector
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'

By default, the monitoring API will try to bind to tcp:9600. If this port is used by another Logstash instance, start Logstash with the --api.http.port flag to bind to a different port.

1.2 Create a Guance connector

In the section 「Manage / Experimental Features」, enable 「Guance Connector」

Go to 「Development / Connectors / Add Connector」, select type 「Guance」, and fill in the relevant configuration parameters as prompted (this script requires the ID value to be "Guance").

For more information on selecting nodes for the Guance connector, please refer to: Manual / Interface and Operations / Guance Node

2. Data Reporting Format

After data synchronization is normal, you can view the data in the 「Metrics / Metrics Analysis」section of TrueWatch.

Example of reported data:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "fields": {
      "logstash_codecs_queue_push_duration_in_millis" : 207,
      "logstash_events_duration_in_millis"            : 349273,
      "logstash_events_failed"                        : 2,
      "logstash_events_in"                            : 345833,
      "logstash_events_out"                           : 345831,
      "logstash_filters_queue_push_duration_in_millis": 1194,
      "logstash_inputs_queue_push_duration_in_millis" : 0,
      "logstash_outputs_queue_push_duration_in_millis": 13177,
      "logstash_reloads_failures"                     : 0
    },
    "measurement": "logstash",
    "tags": {
      "host"     : "xxx",
      "pipeline" : "xxx",
      "port"     : "9600"
    }
}

Descriptions of some fields are as follows:

Field Type Description
logstash_events_in Integer Number of input events in Logstash
logstash_events_out Integer Number of output events in Logstash
logstash_events_failed Integer Number of failed events in Logstash
logstash_events_duration_in_millis Integer Logstash event delay (unit: ms)
logstash_inputs_queue_push_duration_in_millis Integer Queue stage delay in Logstash (unit: ms)
logstash_codecs_queue_push_duration_in_millis Integer Codecs stage delay in Logstash (unit: ms)
logstash_filters_queue_push_duration_in_millis Integer Filter stage delay in Logstash (unit: ms)
logstash_outputs_queue_push_duration_in_millis Integer Output stage delay in Logstash (unit: ms)
logstash_reloads_failures Integer Number of reload failures in Logstash (unit: times)

3. Common Errors

Text Only
1
2
[2023-xx-xx xx:xx:xx.xxx] [+0ms] >>>>>>>>>>>>> Message: Connector configuration exception! Please modify the connector configuration <<<<<<<<<<<<<<<<
[2023-xx-xx xx:xx:xx.xxx] [+0ms] Connector not found: `Guance`

Please check if the connector configuration is correct; refer to 1.2 to create a Guance connector.