Deployment and Maintenance / Reporting Self-Monitoring Data

This article primarily introduces how to configure self-monitoring data reporting for DataFlux Func.

1. Introduction

By default, the self-monitoring data of DataFlux Func is stored in local Redis and MySQL.

After heavy use of DataFlux Func, the volume of content data such as Metrics and LOGs may become quite large. To limit the usage of local memory and disk space, system Metrics and task logs stored locally will be reduced, and the total amount of storage will also be restricted.

You can refer to Deployment and Maintenance / System Metrics and Task Records / Disable Local Function Task Records to disable "Local Function Task Records" and reduce the storage pressure on MySQL.

If you need a complete record of system Metrics and task logs generated by DataFlux Func, you can enable data reporting to TrueWatch through system settings.

2. Enable "Self-Monitoring Data Reporting"

The reported task logs are complete logs and will not be reduced.

In Management / System Settings / Self-Monitoring Data Reporting, users can enable "Self-Monitoring Data Reporting".

The URL address can be filled with DataWay or DataKit reporting addresses:

DataWayDataKit

http://openway.truewatch.com?token=tkn_xxxxx

http://datakit-ip-or-domain:9529

Additionally, it is generally recommended to fill in the site name to mark the current purpose of DataFlux Func, such as: "Test Func"

After enabling 'Self-Monitoring Data Reporting', you can simultaneously disable 'Local Function Task Records' to reduce local DataFlux Func storage pressure.

3. View System Metrics and Task Logs in TrueWatch

After correctly configuring "Self-Monitoring Data Reporting," you can view the reported system metrics and task logs in TrueWatch.

4. Data Reporting Description

DataFlux Func reports various types of data for troubleshooting issues or configuring Guance monitors.

Function Call Metrics

DataFlux Func records relevant Metrics for each function call after every function invocation.

Measurement	Description
DFF_func_call	Function call

Fields are as follows:

Field	Type	Description	Example Value
script_set_id	Tag	Script set ID	"demo"
script_id	Tag	Script ID	"demo__test"
func_id	Tag	Function ID	"demo__test.run"
origin	Tag	Source	"authLink"
queue	Tag	Execution queue	"8"
task_status	Tag	Task status	"success"
wait_cost	Field	Waiting time (milliseconds)	100
run_cost	Field	Execution time (milliseconds)	300
total_cost	Field	Total time (milliseconds)	400

Worker Queue Metrics

Measurement	Description
DFF_worker_queue	Worker queue

Fields are as follows:

Field	Type	Description	Example Value
queue	Tag	Queue	"8"
redis_key	Tag	Queue Redis Key name	"DataFluxFunc-worker#workerQueue@8"
length	Field	Queue length	100

Cache Database Metrics

Measurement	Description
DFF_cache_db	Cache database

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target database (HOST:PORT/DB)	"127.0.0.1:6379/5"
keys	Field	Number of Keys	100
used_memory	Field	Memory usage (bytes)	10000

Cache Database Key Metrics

Measurement	Description
DFF_cache_db_key	Cache database Key

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target database (HOST:PORT/DB)	"127.0.0.1:6379/5"
prefix	Tag	Key prefix	"DataFluxFunc-monitor#heartbeat@workerOnQueue:TAGS"
count	Field	Number of Keys	100

Database Metrics

Measurement	Description
DFF_db_table	Database

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target database (HOST:PORT/DB)	"127.0.0.1:3306/dataflux_func"
name	Tag	Table name	"biz_main_sync_api"
comment	Tag	Table name (Chinese annotation)	"Sync API"
total_length	Field	Total size used (bytes)	200
data_length	Field	Data size used (bytes)	100
index_length	Field	Index size used (bytes)	100
avg_row_length	Field	Average row size used (bytes)	100
rows	Field	Number of rows	10
auto_increment	Field	Auto-increment position	10

System Task Records / Function Task Records

After executing any internal system tasks or function tasks, DataFlux Func reports corresponding task logs, which can be viewed via a log viewer.

Data source	Description
DFF_task_record	System task records
DFF_task_record_func	Function task records

Fields are as follows:

Field	Description	System task records	Function task records
source	Data source	Fixed as DFF_task_record	Fixed as DFF_task_record_func
site_name	Site name
id	Task ID
name	Task name
kwargs	Task parameters
task_status	Task status, possible values are listed below
status	Log status, possible values are listed below
queue	Queue location
trigger_time_iso	Trigger time (ISO date format)
start_time_iso	Start time (ISO date format)
end_time_iso	End time (ISO date format)
run_cost	Execution time (milliseconds)
wait_cost	Queue waiting time (milliseconds)
total_cost	Total execution time (milliseconds)
func_id	Function ID
script_id	Script ID belonging to
script_set_id	Script set ID belonging to
func_call_kwargs	Function input parameters
origin	Origin, possible values are listed below
origin_id	Origin ID e.g., Scheduled task (Old version: Automatic trigger configuration) ID
crontab	Automatic trigger Crontab expression		Only for automatically triggered functions
workspace_uuid	Workspace ID		Only for Guance monitor functions
df_monitor_checker_id	Monitor checker ID		Only for Guance monitor functions
df_monitor_id	Alert strategy ID		Only for Guance monitor functions

5. Detailed Explanation of Relevant Fields

Detailed explanations for some fields are provided below:

Fields task_status and status

In DataFlux Func's reported logs, task_status and status have a one-to-one relationship. task_status describes the task state, while status represents the status value conforming to the Guance standard.

The specific correspondence table is as follows:

task_status value	status value	Description
success	ok	Success
failure	critical	Failure
skip	warning	Task skipped

Fields origin and origin_id

The origin and origin_id fields are used to mark the source of function tasks. Specific values are as follows:

origin value	Description	Meaning of origin_id value	Notes
syncAPI	Synchronous API	Synchronous API ID
asyncAPI	Asynchronous API	Asynchronous API ID
cronJob	Scheduled task	Scheduled task ID
direct	Directly called function e.g., Guance Studio calling within the cluster	Fixed as direct
integration	Triggered by script integration	{Integration type}.{Start method}-{Function ID}
Old version authLink	Authorization link	Authorization link ID	Replaced by syncAPI in the latest version
Old version crontab	Automatic trigger configuration	Automatic trigger configuration ID	Replaced by cronJob in the latest version
Old version batch	Batch processing	Batch processing ID	Replaced by asyncAPI in the latest version