Integration with Guance Advanced Functions

Added in version 2.5.2

1. Background

DQL queries support the selection of advanced functions. By using advanced functions, users can conduct deeper analysis and processing of data according to their needs and expected goals.

Advanced functions are mainly used for further function calculations on the data retrieved by DQL, and provide an intuitive display of time series graphs. In addition to offering official time series processing functions, Guance also supports custom data processing functions reported through local Funcs. The time series charts in Guance support selecting custom functions for secondary data processing and returning processed data for display.

The core functions are continuously updated by the Guance algorithm team. Currently, the v1 version already supports the use of DBSCAN advanced functions.

2. Preparations

To implement advanced functions, the following preparations need to be made:

Preparation	Reference Documentation
Upgrade DataFlux Func to version 2.5.2 or above	Deployment and Maintenance / Daily Maintenance / Upgrade System
Create an API Key in Guance	Guance Documentation / Workspace Management / API Key Management TrueWatch Documentation / Workspace Management / API Key Management

3. Specific Steps

3.1 Create a Guance Connector

In 「Manage / Experimental Features」, enable 「Guance Connector」

Go to 「Development / Connectors / Add Connector」, select type 「Guance」, and fill in the relevant configuration parameters.

3.2 Write Advanced Functions

Go to 「Development / Script Library」, create a script to store advanced functions.

Advanced functions have the following fixed requirements

It is necessary to add category='guance.dqlFunc' when using the @DFF.API(...) decorator.

Create a new local function script in Func. Below is an example of an algorithm implementation:

'''
Example algorithm for advanced functions

Example processing content:
1. Data passed into DQL statement
2. Returns the result after algorithm processing

Note:

Entry function: AlgorithmScriptName(data,**kwargs)
    `data` is the structure of 'series' inside the DQL query data; local Func uses Dataway to retrieve DQL data, while central Func uses Kodo to retrieve DQL data.
    Example structure as follows:
        [
            {
              'name': 'cpu',
              'tags': {'image': 'nginx'},
              'columns': ['usage_total', 'last'],
              'values': [[1681200000000, 8],[1681202880000, 23],......]
              }
              ...
           ]
    **kwargs are optional parameters for the algorithm

Output example:
    Adds an algorithm marker field 'status' to the queried data `data`, for example, returns the following in outlier detection algorithms:
        [
            {
                'status'        : "abnormal_series",
                                # Marks this time series data as normal or abnormal; abnormal status is "abnormal_series", normal status is "normal_series"
                'name'          : 'cpu',
                "tags"          : {'image': 'nginx'},
                'colums'        : ['usage_total', 'last'],
                "values"        : [[1681274880000,8],[1681277760000,20],……],
            },
            ...
        ]
'''

@DFF.API('Advanced Function_A', category='guance.dqlFunc')
def A(data, **kwargs):
    '''
    Args:
        data: Input data.
    '''
    # Add data retrieval logic
    # Enter algorithm processing logic
    return data

After completing the code writing, do not forget to publish it.

3.3 Confirm the Advanced Function in Guance and Select It

3.4 Examples of Time Series Chart Displays in Guance

3.4.1 Outlier Display

Effect after configuring outliers:

If there are no outliers, the original data will be displayed on the frontend:

3.4.2 Prediction Display

TODO Not yet completed

3.4.3 Anomaly Display

TODO Not yet completed

4. Central Advanced Functions

4.1 Outlier Algorithm

DBSCAN Algorithm Introduction

DBSCAN is a density-based clustering algorithm. It divides regions where the density reaches a certain threshold into clusters and views low-density regions as noise. The DBSCAN algorithm does not require specifying the number of clusters in advance, can discover clusters of arbitrary shapes, and has good robustness to noisy data.

• Detection objects: Multiple time series data;
• Parameters: Detection interval T, distance parameter;
• Return: Returns 1 to n outlier timelines.

Core Parameters:

Distance (eps): float, default=0.5

The distance parameter indicates the maximum distance between two samples that are adjacent to each other, not the upper limit of distances within the cluster.

You can configure any floating-point value in the range(0-3.0). If not configured, the default distance parameter is 0.5. Larger distance settings result in fewer anomalies detected, and very small distance values may detect too many outliers, while overly large distance values may lead to no outliers being detected at all. Therefore, appropriate distance parameters should be set based on different data characteristics.

Use Cases

In scenarios, choose a time series chart, click the add function button under the query, select advanced function > DBSCAN, set the algorithm parameters, and the view will display the outlier effects of multiple time series.

DQL Syntax Example

DBSCAN(`M::cpu:(usage_idle) by host`, 0.5)

4.2 Prediction Algorithm

Prophet Algorithm Introduction

Prophet is a program for predicting time series data based on additive models, where nonlinear trends match annual, weekly, and daily seasonality along with holiday effects. Prophet is robust to missing data and trend changes, and generally handles outliers well.

• Detection objects: Single or multiple time series data;
• Parameters: Detection interval T, number of points to predict, prediction frequency;
• Return: Returns the predicted value for each time series.

Core Parameters

periods: Indicates the number of points to predict. Int number of periods to forecast forward.

freq: Indicates the frequency of the time series Any valid frequency for pd.date_range.

Use Cases

In scenarios, choose a time series chart, click the add function button under the query, select advanced function > Prophet, set the algorithm parameters, and the view will predict the trend of time series data for a future period.

4.3 Anomaly Detection Algorithm

TODO Not yet completed