Skip to content

Deployment and Maintenance / Reporting Self-Monitor Data

This article mainly introduces how to configure DataFlux Func to report self-monitor data.

1. Introduction

By default, the self-monitor data of DataFlux Func is stored in local Redis and MySQL.

After heavy use of DataFlux Func, the volume of Metrics, logs, and other content may be quite large. To limit local memory and disk usage, the system Metrics and task record logs saved locally will be reduced, and the total amount saved will also be limited.

You can refer to Deployment and Maintenance / System Metrics and Task Records / Disable Local Func Task Record to disable "Local Func Task Record" and reduce MySQL storage pressure.

If you need to fully record the system Metrics and task record logs generated in DataFlux Func, you can use the system settings to report the data to the data platform.

2. Enable "Self-Monitor Data Reporting"

The reported task logs are complete logs and will not be reduced.

In Management / System Settings / Self-Monitor Data Reporting, users can enable "Self-Monitor Data Reporting."

The URL address can be filled with the DataWay or DataKit reporting address:

Text Only
1
http://openway.ip-or-domain.com?token=tkn_xxxxx
Text Only
1
http://datakit.ip-or-domain:9529

In addition, it is generally recommended to fill in the site name to mark the current use of DataFlux Func, such as: "Test Func."

After enabling 'Self-Monitor Data Reporting,' you can also disable 'Local Func Task Record' to reduce local DataFlux Func storage pressure.

setup-guance-data-upload.png

3. View System Metrics and Task Record Logs in the Data Platform

After correctly configuring "Self-Monitor Data Reporting," you can view the reported system Metrics and task logs in the data platform.

dff-metric-in-guance.png

4. Reported Data Description

DataFlux Func will report various data for troubleshooting.

Func Trigger Count (Metric)

Measurement: DFF_func_trigger

Field Type Description Example Value
func_id Tag Func ID "demo__test.run"
origin Tag Origin "FuncAPI"
origin_id Tag Origin ID
queue Tag Execution Queue "8"
cron_job_exec_mode Tag Cron Job Execution Mode "cronJob"
count Field Count 1

Func Execution (Metric)

Measurement: DFF_func_run

Field Type Description Example Value
func_id Tag Func ID "demo__test.run"
origin Tag Origin "FuncAPI"
origin_id Tag Origin ID
queue Tag Execution Queue "8"
task_status Tag Task Status "success"
cron_job_exec_mode Tag Cron Job Execution Mode "cronJob"
wait_cost Field Wait Time (milliseconds) 100
run_cost Field Execution Time (milliseconds) 300
total_cost Field Total Time (milliseconds) 400
cpu_cost Field CPU Time (milliseconds) 100
non_cpu_cost Field Non-CPU Time (milliseconds) 200

Delay Queue (Metric)

Task queue that has not yet reached the trigger time.

Measurement: DFF_delay_queue

Field Type Description Example Value
queue Tag Queue "8"
length Field Queue Length 100

Worker Queue (Metric)

Task queue that has reached the trigger time.

Measurement: DFF_worker_queue

Field Type Description Example Value
queue Tag Queue "8"
length Field Queue Length 100

Cache Database (Metric)

Measurement: DFF_cache_db

Field Type Description Example Value
server Tag Target Database (HOST:PORT/DB) "127.0.0.1:6379/5"
keys Field Key Count 100
used_memory Field Memory Usage (bytes) 10000
uptime Field Service Uptime (seconds) 60

Database Table (Metric)

Measurement: DFF_db_table

Field Type Description Example Value
server Tag Target Database (HOST:PORT/DB) "127.0.0.1:3306/dataflux_func"
name Tag Table Name "biz_main_func_api"
rows Field Row Count 10
data_size Field Data Size (bytes) 100
index_size Field Index Size (bytes) 100
total_size Field Total Size (bytes) 200
avg_row_size Field Average Row Size (bytes) 100

Business Entity (Metric)

Measurement: DFF_entity

Field Type Description Example Value
entity Tag Business Entity "funcAPI"
count Field Count 100
enabled_count Field Enabled Count 99

Func Service Info (Logging)

Measurement: DFF_service_info

Field Description Example Value
name Service Name "server", "worker", "beat"
version Version Number "7.1.11"
edition Edition "GSE"
hostname Hostname "web001"
pid Process PID 1234
uptime Service Uptime (seconds) 60

System Task Record / Func Task Record (Logging)

DataFlux Func will report corresponding task logs after executing any internal system tasks or func tasks, which can be viewed through the log explorer.

Measurement Description
DFF_task_record System Task Record
DFF_task_record_func Func Task Record
Field Description System Task Record Func Task Record
source Data Source Fixed as DFF_task_record Fixed as DFF_task_record_func
site_name Site Name
id Task ID
name Task Name
queue Queue
task_status Task Status, possible values see below
root_task_id Root Task ID
script_set_id Script Set ID
script_id Script ID
func_id Func ID
func_name Func Name
origin Origin, possible values see below
origin_id Origin ID
E.g.: Cron Job ID
script_set_title Script Set Title
script_title Script Title
func_title Func Title
cron_job_exec_mode Cron Job Execution Mode Only for Cron Jobs
func_call_kwargs Func Call Arguments
cron_expr Cron Job Crontab Expression Only for Cron Jobs
call_chain Call Chain
return_value Return Value
message Task Log
kwargs Task Parameters
eta Estimated Execution Time
delay Delay Execution Duration (seconds)
timeout Timeout Duration (seconds)
expires Expiration Duration (seconds)
ignore_result Whether to Ignore Result
result Result Content
exception_from Exception Source
exception_type Exception Type
exception Exception Content
origin_exception_type Original Exception Type
origin_exception Original Exception Content
traceback Error Stack
trigger_time_iso Trigger Time (ISO Date Format)
start_time_iso Start Time (ISO Date Format)
end_time_iso End Time (ISO Date Format)
wait_cost Queue Time (milliseconds)
run_cost Execution Time (milliseconds)
total_cost Total Time (milliseconds)
cpu_cost CPU Time (milliseconds)
cpu_cost_percent CPU Time Percentage
non_cpu_cost Non-CPU Time (milliseconds)
non_cpu_cost_percent Non-CPU Time Percentage
sys_db_query_count System DB Query Count
sys_db_query_details System DB Query Details
sys_cache_db_query_count System Cache Query Count
sys_cache_db_query_details System Cache Query Details
status Log Status, possible values see below
workspace_uuid Workspace ID Only for Monitor Funcs
df_monitor_checker_id Monitor ID Only for Monitor Funcs
df_monitor_id Alert Strategy ID Only for Monitor Funcs

Some fields are explained in detail below.

Fields task_status and status

In the logs reported by DataFlux Func, task_status and status have a one-to-one relationship. task_status is the task status description, and status is the status value that meets the data platform specifications.

The specific correspondence table is as follows:

task_status Value status Value Description
success ok Success
failure critical Failure
skip warning Task Skipped

Fields origin and origin_id

The origin and origin_id fields are used to mark the execution source of the func task. The specific values are as follows:

origin Value Description origin_id Value Meaning Notes
funcAPI Func API Func API ID
cronJob Cron Job Cron Job ID
direct Direct Func Call
E.g.: Data Platform Studio calls within the cluster		 Fixed as direct
integration Triggered by Script Integration {Integration Type}.{Start Method}-{Func ID}
Old Version syncAPI Sync API Sync API ID In the latest version, replaced by funcAPI
Old Version asyncAPI Async API Async API ID In the latest version, replaced by funcAPI
Old Version authLink Auth Link Auth Link ID In the latest version, replaced by syncAPI
Old Version crontab Auto Trigger Configuration Auto Trigger Configuration ID In the latest version, replaced by cronJob
Old Version batch Batch Batch ID In the latest version, replaced by asyncAPI