Deployment and Maintenance / Architecture, Scaling, and Resource Limiting

This article mainly introduces the overall architecture of DataFlux Func and how to scale it to improve processing capacity.

1. Architecture

Internally, the system is a typical "producer -> consumer" model. Every execution of a Python function goes through the process of "create task -> enqueue -> dequeue -> execute -> return result".

Any Python function is first wrapped into a "task" and enters its "work queue" (numbered starting from #0), and is then retrieved from the queue and executed by the corresponding "worker" (numbered starting from worker-0).

flowchart TB
    USER[User]
    FUNC_SERVER[Func Server Service]
    REDIS_QUEUE_N[Redis Queue #N]
    FUNC_WORKER_N[Func Worker-N Service]
    FUNC_BEAT[Func Beat Service]

    USER --HTTP Request--> FUNC_SERVER

    FUNC_SERVER --Enqueue function execution task--> REDIS_QUEUE_N

    REDIS_QUEUE_N --Dequeue function execution task--> FUNC_WORKER_N

    FUNC_BEAT --"Enqueue function execution task
    (Cron Job)"--> REDIS_QUEUE_N

1.1 Services and Their Purposes

DataFlux Func contains multiple services, each with different responsibilities, as follows:

Service	Purpose
server	Web service, providing the following functions: 1. Web interface 2. API endpoints 3. Subscriber maintenance
worker-{queue index}	Worker, used to execute user Scripts, including: 1. Func API 2. Func API 3. Cron Job It also handles some system-level background tasks See the queue description for details
beat	Cron Job trigger
mysql	Database
redis	Cache / Function execution task queue

1.2 Worker and Queue Listening Relationship

For the service worker-{queue index} (worker), each Worker service only listens to specific queues:

Queues and workers do not have to correspond one-to-one

Queues and workers are not required to correspond one-to-one; for example, the worker worker-0 is not limited to listening to tasks in queue #0. Each worker can listen to any one or more queues.

Moreover, the same queue can be listened to by multiple workers at the same time, or it can be left unmonitored (not recommended).

Standalone Func and Data Platform-affiliated Func have different queues

Since most standalone Func deployments are relatively lightly used, to reduce unnecessary resource consumption, the number of workers in standalone Func is smaller than the number of queues.

In contrast, Data Platform-affiliated Func handles heavy business such as monitors and the message sending module (Message Desk), so workers correspond one-to-one with queues, and there are more numbered workers and queues than in standalone Func.

6.0.4 and later versions3.2.0 (1.77.145) and later versions3.1.0 (1.79.155) and earlier versions

Worker	Queue Standalone deployment	Queue Data Platform-affiliated
worker-0	#0, #4, #7, #8, #9	#0
worker-1	#1	#1
worker-2	#2	#2
worker-3	#3	#3
worker-4	-	#4
worker-5	#5	#5
worker-6	#6	#6
worker-7	-	#7
worker-8	-	#8
worker-9	-	#9
worker-10	-	#10
worker-11	-	#11
worker-12	-	#12
worker-13	-	#13
worker-14	-	#14
worker-15	-	#15

Worker	Queue Standalone deployment	Queue Data Platform-affiliated
worker-0	#0, #4, #7, #8, #9	#0
worker-1	#1	#1
worker-2	#2	#2
worker-3	#3	#3
worker-4	-	#4
worker-5	#5	#5
worker-6	#6	#6
worker-7	-	#7
worker-8	-	#8
worker-9	-	#9

Worker	Queue
worker-0	#0
worker-1-6	#1, #2, #3, #4, #5, #6
worker-7	#7
worker-8-9	#8, #9

2. Services / Queues and Their Responsibilities and Scaling Recommendations

Scaling requires more hardware investment

Scaling imposes higher performance requirements on the server hosting it, including but not limited to the server itself, database services, Redis, etc.

Generally speaking, scaling DataFlux Func actually only requires increasing the number of replicas of the corresponding services. Therefore, users should first understand their actual business situation in order to scale in a targeted manner.

The complete services, queues, their responsibilities, and scaling recommendations are as follows:

6.0.4 and later versions3.2.0 (1.77.145) and later versions3.1.0 (1.79.155) and earlier versions

Service / Queue	Responsibilities Standalone deployment	Responsibilities Data Platform-affiliated	Default Pod Count Data Platform-affiliated	Scaling Recommendations
server	Web service, providing the following functions: 1. Web interface 2. API interface 3. Maintaining subscribers	← Same as left	1	Generally, no scaling is required
server-inner	(No such service)	Web service, exclusively for calling APIs within the cluster	1	Generally, no scaling is required
worker-0 Queue #0	System worker, does not directly participate in user code processing	← Same as left	2	Generally, no scaling is required
worker-1 Queue #1	Executes function tasks from the synchronously executed Func API	← Same as left	1	Can be scaled when the concurrency of the synchronously executed Func API needs to be increased
worker-2 Queue #2	Executes function tasks from Cron Jobs	← Same as left	1	Can be scaled when the concurrency of Cron Jobs needs to be increased
worker-3 Queue #3	Executes function tasks from the asynchronously executed Func API	← Same as left	1	Can be scaled when the concurrency of the asynchronously executed Func API needs to be increased
worker-4 Queue #4	(Reserved)	(Reserved)	0	No scaling required
worker-5 Queue #5	Debug code execution i.e., directly running functions in the Web interface	← Same as left	1	Scale when more users need to develop Scripts simultaneously
worker-6 Queue #6	Executes function tasks from Connector subscription message processing	← Same as left	1	Can be scaled when the concurrency of Connector subscription message processing needs to be increased
worker-7 Queue #7	(Reserved)	Executes function tasks for data platform system business e.g., logging in as a data platform backend administrator, updating various caches, releasing message aggregation pools, etc.	2	Scale when the total number of monitors is large
worker-8 Queue #8	(Reserved)	Executes function tasks related to ordinary monitors such as data platform threshold detection, metric generation, etc.	5	Scale when the number of ordinary monitors is large
worker-9 Queue #9	(Reserved)	Executes function tasks for data platform advanced detection and intelligent monitoring	3	Scale when the number of advanced detection and intelligent monitors is large
worker-10 Queue #10	(No such service)	Executes function tasks for receiving user-reported events on the data platform	1	Scale when the volume of user-reported events is large
worker-11 Queue #11	(No such service)	Executes Message Desk message sending tasks	3	Scale when the message sending volume is large
worker-12 Queue #12	(No such service)	(Reserved)	0	No scaling needed
worker-13 Queue #13	(No such service)	(Reserved)	0	No scaling needed
worker-14 Queue #14	(No such service)	Executes AI-related processing that requires immediate response to user operations e.g., invoking "Auto-Write Pipeline", etc.	2	Scale when more users need to write Pipelines concurrently
worker-15 Queue #15	(No such service)	Executes AI-related processing that does not require immediate response to user operations e.g., handling "Alert Compression and Merging"	2	Scale when more monitors using AI to aggregate alerts exist
beat	Cron Job trigger	← Same as left	1	Do not scale; keep a single global replica
mysql	Database	(No such service)	-	No scaling needed; for higher requirements, consider self-hosted or cloud services
redis	Cache / function execution task queue	(No such service)	-	No scaling needed; for higher requirements, consider self-hosted or cloud services

Service / Queue	Responsibility Standalone deployment	Responsibility Data platform attached	Scaling recommendation
server	Web service, providing the following functions: 1. Web interface 2. API endpoints 3. Maintenance subscriber	← Same as left	Generally no scaling needed
server-inner	(No such service)	Web service, exclusively for calling APIs within the cluster	Generally no scaling needed
worker-0 Queue #0	System worker unit, not directly involved in processing user code	← Same as left	Generally no scaling needed
worker-1 Queue #1	Executes function tasks from synchronously executed Func API	← Same as left	Can scale out when needing to increase the concurrency of synchronously executed Func API
worker-2 Queue #2	Executes function tasks from Cron Job	← Same as left	Can scale out when needing to increase Cron Job concurrency
worker-3 Queue #3	Executes function tasks from asynchronously executed Func API	← Same as left	Can scale out when needing to increase the concurrency of asynchronously executed Func API
worker-4 Queue #4	(Reserved)	(Reserved)	No scaling needed
worker-5 Queue #5	Debug code execution i.e., directly running functions in the Web interface	← Same as left	Scale out when needing to support more users developing Scripts simultaneously
worker-6 Queue #6	Executes function tasks from Connector subscription message processing	← Same as left	Can scale out when needing to increase the concurrency of Connector subscription message processing
worker-7 Queue #7	(Reserved)	Executes function tasks for data platform system business and message sending e.g., logging in as the data platform backend administrator, updating various caches, releasing message aggregation pools, and sending Message Desk messages	Scale out when message sending volume is large
worker-8 Queue #8	(Reserved)	Executes function tasks related to regular monitors such as data platform threshold detection	Can scale out when there are many regular monitors
worker-9 Queue #9	(Reserved)	Executes function tasks for data platform advanced detection and smart monitoring	Can scale out when there are many regular advanced detection and smart monitors
beat	Trigger for Cron Jobs	← Same as left	Must not scale out; ensure a single global replica
mysql	Database	(No such service)	No scaling needed; if there are higher requirements, you can choose self-hosted or cloud services
redis	Cache / function execution task queue	(No such service)	No scaling needed; if there are higher requirements, you can choose self-hosted or cloud services

Service	Responsibility	Scaling Recommendation
server	Web service, providing the following functions: 1. Web interface 2. API endpoints 3. Subscription maintenance	Generally no scaling needed
worker-0 Queue #0	System worker unit, does not directly participate in processing user code	Generally no scaling needed
worker-1-6 Queues #1, #2, #3, #4, #5, #6	By default, responsible for handling synchronous function calls, such as: 1. Synchronously executed Func API 2. Subscription message processing	Can scale out when you need to increase the concurrency of synchronously executed Func API and subscription message processing
worker-7 Queue #7	By default, responsible for debug code processing (i.e., running functions directly in the Web interface)	Scale out when you need to support more users developing Scripts simultaneously
worker-8-9 Queues #8, #9	By default, responsible for handling asynchronous function calls, such as: 1. Asynchronously executed Func API 2. Cron Jobs	Can scale out when you need to increase the concurrency of Cron Jobs and asynchronously executed Func API
beat	Trigger for Cron Jobs	Must not scale out; ensure a single global replica
mysql	Database	No scaling needed; if there are higher requirements, you can choose self-hosted or cloud services
redis	Cache / function execution task queue	No scaling needed; if there are higher requirements, you can choose self-hosted or cloud services

Example: When you need to enhance the processing capability of Cron Jobs...

From the above, Cron Jobs are in Queue #8, and Queue #8 corresponds to Service worker-8. Therefore, scaling out Service worker-8 is sufficient.

Estimating Scaling Capacity

Taking the common worker-8 as an example:

In the Data Platform-affiliated version, worker-8 is mainly responsible for executing monitor tasks. Assuming a detection task takes T milliseconds, 1 minute can execute 60 × 1,000 ÷ T detections. By default, each worker-8 Pod starts 5 processes.

That is, the detection capacity of a single worker-8 Pod is 5 × (60 × 1,000 ÷ T) monitors.

Formula

A = 5 × (60 × 1,000 ÷ T)
  = 300,000 ÷ T

A: Detection capacity

T: Detection task execution time (milliseconds)

Depending on the execution time of each monitor task, the following table can be listed:

Time per detection	Detection capacity per Pod	Compared to baseline
300	1,000	167%
500	600	baseline
800	375	63%
1,000	300	50%
2,000	150	25%
3,000	100	17%

Conversely, assuming the total number of monitors is M, the required number of Pods can be calculated as M ÷ (5 × (60 × 1,000 ÷ T)).

Formula

P = M ÷ (300,000 ÷ T)
  = M × T ÷ 300,000

P: Required number of Pods

M: Number of monitors

T: Detection task execution time (milliseconds)

Depending on the number of monitors and the execution time of each task, the following table can be listed:

1,000 monitors5,000 monitors10,000 monitors

Number of monitors	Time per detection	Required number of Pods	Compared to baseline
1,000	300	1	50%
1,000	500	2	baseline
1,000	800	3	150%
1,000	1,000	4	200%
1,000	2,000	7	350%
1,000	3,000	10	500%

Number of monitors	Time per detection	Required number of Pods	Compared to baseline
5,000	300	5	56%
5,000	500	9	baseline
5,000	800	14	156%
5,000	1,000	17	189%
5,000	2,000	34	378%
5,000	3,000	50	556%

Number of monitors	Time per detection	Required number of Pods	Compared to baseline
10,000	300	10	59%
10,000	500	17	baseline
10,000	800	27	159%
10,000	1,000	34	200%
10,000	2,000	67	394%
10,000	3,000	100	588%

How to Operate

DataFlux Func deployed on a single machine can be scaled out by modifying the configuration ({installation-directory}/docker-stack.yaml) and adding deploy.replicas to the corresponding service.

Please refer to the official documentation

For complete information about the deploy.replicas option, please refer to the Docker official documentation: Docker Documentation / Compose file deploy reference / replicas

Taking improving the processing capacity of worker-8 as an example, the specific modified parts are as follows:

The example is only an excerpt

The example only shows the key modified parts. In actual operation, please ensure the configuration is complete.

services:
  worker-8:
    deploy:
      # Start 2 worker units that process queue 8 simultaneously
      replicas: 2

3. Resource Limiting

Resource limiting should be adjusted reasonably according to actual business needs

Please adjust resource limits reasonably according to actual business needs.

Excessively limiting resources may lead to longer task execution, or insufficient memory, preventing code execution from being completed.

How to Operate

DataFlux Func deployed on a single machine can limit resources by modifying the configuration ({installation-directory}/docker-stack.yaml) and adding deploy.resources to the corresponding service.

Please refer to the official documentation

For complete information about the deploy.resources option, please refer to the Docker official documentation: Docker Documentation / Compose file deploy reference / resources

By default, each worker-N replica can occupy up to 5 CPU cores (i.e., each worker unit has 5 worker processes).

Taking limiting the resource usage of worker-8 as an example, the specific modified parts are as follows:

The example is only an excerpt

The example only shows the key modified parts. In actual operation, please ensure the configuration is complete.

services:
  worker-8:
    deploy:
      resources:
        limits:
          cpus  : '2.50' # Limit CPU to a maximum of 2.5 cores
          memory: 4G     # Limit memory to a maximum of 4 GB

4. Splitting Worker Units

In the new version, all worker units have already been split

In standalone versions of Func 3.2.0 and later, all non-reserved worker units are split by default, and users can enable reserved queues as needed.

In versions of Func 1.77.145 and later deployed with the Data Platform, all worker units are already split by default, and users no longer need to split them manually.

In special cases, the default merged worker units (such as worker-1-6) can be split to achieve finer-grained task scheduling, and to enable scaling and resource limiting for worker units responsible for specific queues.

Assume that, based on business requirements, DataFlux Func has high performance requirements for subscription processing, and it is desired that subscription message processing and synchronously executed Func API processing do not interfere with each other. Then worker-1-6 can be split into worker-1-5 and worker-6.

How to Operate

DataFlux Func deployed on a single machine can split worker units by modifying the configuration ({installation-directory}/docker-stack.yaml), adding or modifying the corresponding services, and modifying the queue numbers specified in command.

The queues that a worker unit listens to are specified through the parameters following ./run-worker-by-queue.sh. The service name itself is mainly used as a label, and it is recommended to keep it consistent with the actual queues it listens to, to avoid confusion.

The example is only an excerpt

The example only shows the key modified parts. In actual operation, please ensure the configuration is complete.

services:
  # Delete the original "worker-1-6" and replace it with the following

  worker-1-5:
    # Specify that this worker unit handles queues 1 to 5
    command: ./run-worker-by-queue.sh 1 2 3 4 5

  worker-6:
    # Specify that this worker unit handles queue 6
    command: ./run-worker-by-queue.sh 6