Deployment and Maintenance / Uploading Self-Monitor Data

This guide explains how to configure DataFlux Func to upload its self-monitor data.

1. Introduction

By default, DataFlux Func's self-monitor data is stored in the local Redis and MySQL.

After heavy usage of DataFlux Func, the volume of Metrics, logs, and other content may become quite large. To limit local memory and disk usage, the locally stored system Metrics and task logs are reduced, and the total amount saved is also restricted.

You can refer to Deployment and Maintenance / System Metrics and Task Records / Disable Local Func Task Record to disable the "Local Func Task Record" and reduce MySQL storage pressure.

If you need to fully record the system Metrics and task logs generated in DataFlux Func, you can configure the system to upload this data to the data platform.

2. Enable "Self-Monitor Data Upload"

The uploaded task logs are complete logs and will not be reduced.

In Management / System Settings / Self-Monitor Data Upload, users can enable "Self-Monitor Data Upload."

The URL address can be filled with either DataWay or DataKit upload addresses:

DataWayDataKit

http://openway.ip-or-domain.com?token=tkn_xxxxx

http://datakit.ip-or-domain:9529

Additionally, it is generally recommended to fill in the site name to mark the current DataFlux Func usage, such as: "Test Func."

After enabling 'Self-Monitor Data Upload,' you can also disable 'Local Func Task Record' to reduce local DataFlux Func storage pressure.

3. View System Metrics and Task Logs in the Data Platform

After correctly configuring "Self-Monitor Data Upload," you can view the uploaded system Metrics and task logs in the data platform.

4. Uploaded Data Description

DataFlux Func uploads various data for troubleshooting.

Func Call Metrics

DataFlux Func records relevant Metrics after each Func call.

Measurement	Description
DFF_func_call	Func Call

Fields are as follows:

Field	Type	Description	Example Value
script_set_id	Tag	Script Set ID	"demo"
script_id	Tag	Script ID	"demo__test"
func_id	Tag	Func ID	"demo__test.run"
origin	Tag	Origin	"authLink"
queue	Tag	Execution Queue	"8"
task_status	Tag	Task Status	"success"
wait_cost	Field	Wait Time (milliseconds)	100
run_cost	Field	Execution Time (milliseconds)	300
total_cost	Field	Total Time (milliseconds)	400

Worker Queue Metrics

Measurement	Description
DFF_worker_queue	Worker Queue

Fields are as follows:

Field	Type	Description	Example Value
queue	Tag	Queue	"8"
redis_key	Tag	Queue Redis Key Name	"DataFluxFunc-worker#workerQueue@8"
length	Field	Queue Length	100

Cache Database Metrics

Measurement	Description
DFF_cache_db	Cache Database

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target Database (HOST:PORT/DB)	"127.0.0.1:6379/5"
keys	Field	Key Count	100
used_memory	Field	Memory Usage (bytes)	10000

Cache Database Key Metrics

Measurement	Description
DFF_cache_db_key	Cache Database Key

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target Database (HOST:PORT/DB)	"127.0.0.1:6379/5"
prefix	Tag	Key Prefix	"DataFluxFunc-monitor#heartbeat@workerOnQueue:TAGS"
count	Field	Key Count	100

Database Metrics

Measurement	Description
DFF_db_table	Database

Fields are as follows:

Field	Type	Description	Example Value
target	Tag	Target Database (HOST:PORT/DB)	"127.0.0.1:3306/dataflux_func"
name	Tag	Table Name	"biz_main_sync_api"
comment	Tag	Table Comment	"Func API"
total_length	Field	Total Size (bytes)	200
data_length	Field	Data Size (bytes)	100
index_length	Field	Index Size (bytes)	100
avg_row_length	Field	Average Row Size (bytes)	100
rows	Field	Row Count	10
auto_increment	Field	Auto Increment Position	10

System Task Records / Func Task Records

DataFlux Func uploads corresponding task logs after executing any internal system tasks or Func tasks, which can be viewed via the Log Explorer.

Data Source	Description
DFF_task_record	System Task Records
DFF_task_record_func	Func Task Records

Fields are as follows:

Field	Description	System Task Records	Func Task Records
source	Data Source	Fixed as DFF_task_record	Fixed as DFF_task_record_func
site_name	Site Name
id	Task ID
name	Task Name
kwargs	Task Parameters
task_status	Task Status, possible values see below
status	Log Status, possible values see below
queue	Queue
trigger_time_iso	Trigger Time (ISO Date Format)
start_time_iso	Start Time (ISO Date Format)
end_time_iso	End Time (ISO Date Format)
run_cost	Execution Time (milliseconds)
wait_cost	Queue Time (milliseconds)
total_cost	Total Time (milliseconds)
func_id	Func ID
script_id	Script ID
script_set_id	Script Set ID
func_call_kwargs	Func Call Parameters
origin	Origin, possible values see below
origin_id	Origin ID e.g., Cron Job ID
crontab	Cron Job Crontab Expression		Only for Cron Job Funcs
workspace_uuid	Workspace ID		Only for Monitor Funcs
df_monitor_checker_id	Monitor ID		Only for Monitor Funcs
df_monitor_id	Alert Strategy ID		Only for Monitor Funcs

5. Detailed Explanation of Related Fields

Some fields are explained in detail below.

Fields task_status and status

In DataFlux Func uploaded logs, task_status and status have a one-to-one relationship. task_status describes the task status, and status is the status value that complies with the data platform's specifications.

The specific correspondence is as follows:

task_status Value	status Value	Description
success	ok	Success
failure	critical	Failure
skip	warning	Task Skipped

Fields origin and origin_id

The origin and origin_id fields are used to mark the execution source of Func tasks. The specific values are as follows:

origin Value	Description	origin_id Value Meaning	Notes
funcAPI	Func API	Func API ID
cronJob	Cron Job	Cron Job ID
direct	Direct Func Call e.g., Data Platform Studio via cluster call	Fixed as direct
integration	Triggered by Script Integration	{Integration Type}.{Launch Method}-{Func ID}
Legacy syncAPI	Sync API	Sync API ID	Replaced by funcAPI in the latest version
Legacy asyncAPI	Async API	Async API ID	Replaced by funcAPI in the latest version
Legacy authLink	Auth Link	Auth Link ID	Replaced by syncAPI in the latest version
Legacy crontab	Auto Trigger Configuration	Auto Trigger Configuration ID	Replaced by cronJob in the latest version
Legacy batch	Batch	Batch ID	Replaced by asyncAPI in the latest version