Deployment and Maintenance / Reporting Self-Monitor Data

This article mainly describes how to configure the Self-Monitor data reporting of DataFlux Func.

1. Preface

By default, DataFlux Func's Self-Monitor data is stored in local Redis and MySQL.

After heavy use of DataFlux Func, the volume of data such as metrics and logs may be quite large. To limit local memory and disk usage, the system metrics and task record logs stored locally will be reduced, and the total amount stored will also be limited.

You can refer to Deployment and Maintenance / System Metrics and Task Records / Disable Local Function Task Records to disable "Local Function Task Records" and reduce MySQL storage pressure.

If you need to completely record the system metrics and task record logs generated in DataFlux Func, you can report the data to the data platform through system settings.

2. Enabling Self-Monitor Data Reporting

The reported task logs are complete logs and will not be reduced

In Management / System Settings / Self-Monitor Data Reporting, users can enable "Self-Monitor Data Reporting".

The URL address can be filled with the DataWay or DataKit reporting address:

DataWayDataKit

http://openway.ip-or-domain.com?token=tkn_xxxxx

http://datakit.ip-or-domain:9529

In addition, it is generally recommended to fill in the site name to indicate the purpose of the current DataFlux Func, such as "Test Func".

After enabling "Self-Monitor Data Reporting", you can also disable "Local Function Task Records" to reduce local DataFlux Func storage pressure

3. View System Metrics and Task Record Logs in the Data Platform

After correctly configuring "Self-Monitor Data Reporting", you can view the reported system metrics and task logs in the data platform.

4. Description of Reported Data

DataFlux Func reports various data for troubleshooting.

Self-Monitor may be adjusted at any time

Self-Monitor data may be adjusted at any time with version updates according to actual conditions. This document only provides the description of the latest Self-Monitor data.

Add a service filter to improve query efficiency

In the latest version, all Logging Self-Monitor data will additionally include a service Tag, whose value is the same as source in Logging, to adapt to ScopeDB's service partition storage.

Therefore, when querying Func's Logging Self-Monitor data, adding service:xxxxx on the basis of the filter condition source:xxxxx can significantly improve query efficiency.

Delay Queue (Metric)

Task queue whose trigger time has not yet arrived

Metric set: DFF_delay_queue

Field	Type	Description	Example Value
queue	Tag	Queue	"8"
length	Field	Queue length	100

Worker Queue (Metric)

Task queue whose trigger time has arrived

Metric set: DFF_worker_queue

Field	Type	Description	Example Value
queue	Tag	Queue	"8"
length	Field	Queue length	100
worker_count	Field	Number of worker units listening to the queue	5
process_count	Field	Number of processes listening to the queue	25

Function Trigger Count (Metric)

Metric set: DFF_func_trigger

Field	Type	Description	Example Value
func_id	Tag	Function ID	"demo__test.run"
trigger_count_per_minute	Field	Trigger count per minute	100

Function Run Count (Metric)

Metric set: DFF_func_run

Field	Type	Description	Example Value
func_id	Tag	Function ID	"demo__test.run"
run_count_per_minute	Field	Run count per minute	100

Function Run Status Count (Metric)

Metric set: DFF_func_status

Field	Type	Description	Example Value
status	Tag	Function run status	"success"
status_count_per_minute	Field	Status count per minute	100

Function Run Cost (Metric)

Metric set: DFF_func_cost

Field	Type	Description	Example Value
wait_cost_sum_per_minute	Field	Total wait time per minute (milliseconds)	10000
queue_cost_sum_per_minute	Field	Total queue time per minute (milliseconds)	10000
run_cost_sum_per_minute	Field	Total run time per minute (milliseconds)	10000
total_cost_sum_per_minute	Field	Total time consumed per minute (milliseconds)	10000
cpu_cost_sum_per_minute	Field	Total CPU time per minute (milliseconds)	10000
non_cpu_cost_sum_per_minute	Field	Total non-CPU time per minute (milliseconds)	10000

Cache Database (Metric)

Metric set: DFF_cache_db

Field	Type	Description	Example Value
server	Tag	Target database (HOST:PORT/DB)	"127.0.0.1:6379/5"
keys	Field	Number of keys	100
used_memory	Field	Memory usage (bytes)	10000
connected_clients	Field	Number of connected clients	100
uptime	Field	Service uptime (seconds)	60

Database Table (Metric)

Metric set: DFF_db_table

Field	Type	Description	Example Value
server	Tag	Target database (HOST:PORT/DB)	"127.0.0.1:3306/dataflux_func"
name	Tag	Table name	"biz_main_func_api"
rows	Field	Number of rows	10
data_size	Field	Data size (bytes)	100
index_size	Field	Index size (bytes)	100
total_size	Field	Total size (bytes)	200
avg_row_size	Field	Average row size (bytes)	100

Cron Job Schedule (Metric)

Metric set: DFF_cron_job_scheduled

Field	Type	Description	Example Value
scheduled_count_per_week	Field	Scheduled count per week	604800
scheduled_count_per_day	Field	Scheduled count per day	86400
scheduled_count_per_hour	Field	Scheduled count per hour	3600
scheduled_count_per_minute	Field	Scheduled count per minute	60
scheduled_count_per_second	Field	Scheduled count per second	1

Business Entity (Metric)

Metric set: DFF_entity

Field	Type	Description	Example Value
entity	Tag	Business entity	"funcAPI"
count	Field	Count	100
enabled_count	Field	Enabled count	99

Self-Monitor Data Reporting Panic Level (Metric)

Metric set: DFF_self_monitor_upload_panic_level

Field	Type	Description	Example Value
panic_level	Field	Panic level	10

Func Service Info (Logging)

Metric set: DFF_service_info

Field	Type	Description	Example Value
name	Tag	Service name	"server", "worker", "beat"
version	Tag	Version	"7.1.11"
edition	Tag	Edition	"GSE"
hostname	Tag	Hostname	"web001"
pid	Tag	Process PID	1234
uptime	Field	Service uptime (seconds)	60

System Task Record / Function Task Record (Logging)

After DataFlux Func executes any internal system task or function task, it reports the corresponding task log, which can be viewed in the Log Viewer.

Metric Set	Description
DFF_task_record	System task record
DFF_task_record_func	Function task record

Field	Description	System Task Record	Function Task Record
source	Data source	Fixed to DFF_task_record	Fixed to DFF_task_record_func
site_name	Site name
id	Task ID
name	Task name
queue	Queue
task_status	Task status; see below for possible values
root_task_id	Root task ID
script_set_id	Script Set ID
script_id	Script ID
func_id	Function ID
func_name	Function name
origin	Origin; see below for possible values
origin_id	Origin ID e.g., Cron Job ID
script_set_title	Script Set title
script_title	Script title
func_title	Function title
cron_job_exec_mode	Cron Job execution mode		Cron Job only
func_call_kwargs	Function call arguments
cron_expr	Cron Job Crontab expression		Cron Job only
call_chain	Call chain
return_value	Return value
message	Task log
kwargs	Task parameters
eta	Expected execution time
delay	Delay duration (seconds)
timeout	Timeout duration (seconds)
expires	Expiration duration (seconds)
ignore_result	Whether to ignore the result
result	Result content
exception_from	Exception source
exception_type	Exception type
exception	Exception content
origin_exception_type	Original exception type
origin_exception	Original exception content
traceback	Stack trace
trigger_time_iso	Trigger time (ISO date format)
start_time_iso	Start time (ISO date format)
end_time_iso	End time (ISO date format)
wait_cost	Queue wait time (ms)
run_cost	Execution time (ms)
total_cost	Total time (ms)
cpu_cost	CPU time (ms)
cpu_cost_percent	CPU time percentage
non_cpu_cost	Non-CPU time (ms)
non_cpu_cost_percent	Non-CPU time percentage
sys_db_query_count	System DB query count
sys_db_query_details	System DB query details
sys_cache_db_query_count	System cache query count
sys_cache_db_query_details	System cache query details
status	Log status; possible values are listed below
workspace_uuid	Workspace ID		Only for monitor functions in the attached version of Func
df_monitor_checker_id	Monitor ID		Only for monitor functions in the attached version of Func
df_monitor_id	Alert policy ID		Only for monitor functions in the attached version of Func
df_wksp_id	2026-08-05Workspace ID		Only for functions in the attached version of Func that involve the wksp_xxxxx field
df_rul_id	2026-08-05Monitor / metric generation rule ID		Only for functions in the attached version of Func that involve the rul_xxxxx field

5. Details of Related Fields

Details of some fields are as follows:

Fields task_status and status

In logs reported by DataFlux Func, task_status and status have a one-to-one relationship. task_status is the task status description, and status is the status value that conforms to the data platform specification.

The specific mapping is as follows:

task_status value	status value	Description
success	ok	Success
failure	critical	Failure
skip	warning	Task skipped

Fields origin and origin_id

The origin and origin_id fields are used to mark the execution source of function tasks. The specific values are as follows:

origin value	Description	Meaning of origin_id value	Remarks
funcAPI	Func API	Func API ID
cronJob	Cron Job	Cron Job ID
direct	Direct function call e.g., Data Platform Studio calls it within the cluster	Fixed to direct
integration	Triggered by Script Set integration	{integration type}.{startup method}-{function ID}
Legacy syncAPI	Sync API	Sync API ID	Replaced by funcAPI in the latest version
Legacy asyncAPI	Async API	Async API ID	Replaced by funcAPI in the latest version
Legacy authLink	Auth Link	Auth Link ID	Replaced by syncAPI in the latest version
Legacy crontab	Automatic trigger configuration	Automatic trigger configuration ID	Replaced by cronJob in the latest version
Legacy batch	Batch	Batch ID	Replaced by asyncAPI in the latest version