Skip to content

Script Development / Function Execution Process

This article mainly describes the specific execution process when a function is called.

1. Basic Process

Overall, when a function in DataFlux Func is called, the specific execution flow is as follows:

  1. User sends an HTTP request to Server
  2. Server generates a function execution task and pushes it to the Redis queue
  3. Worker pops the function execution task from the Redis queue and executes it

For tasks executed on a schedule, the "user sends an HTTP request" step does not exist; instead, tasks are generated directly by the Beat service. The specific execution flow is as follows:

  1. Beat periodically generates function execution tasks and pushes them to the Redis queue
  2. Worker pops the function execution task from the Redis queue and executes it

2. Single-Machine Deployment

For a single-machine deployment of DataFlux Func, the entire process is very simple:

flowchart TB
    USER[User]
    SERVER[Server]
    WORKER[Worker]
    REDIS_QUEUE[Redis Queue]

    USER --"HTTP request"--> SERVER
    SERVER --"enqueue function execution task"--> REDIS_QUEUE
    REDIS_QUEUE --"dequeue function execution task"--> WORKER

    Beat --"enqueue function execution task\n(scheduled)"--> REDIS_QUEUE

2. Multi-Replica Deployment

For a multi-replica deployment of DataFlux Func, due to the presence of an SLB (or other reverse proxy service), any Server may receive requests.

At the same time, since each replica connects to the same Redis, each task will only be fetched and executed by any one Worker:

flowchart TB
    USER[User]
    SERVER_1[Server 1]
    SERVER_2[Server 2]
    WORKER_1[Worker 1]
    WORKER_2[Worker 2]
    REDIS_QUEUE[Redis Queue]

    USER --"HTTP request"--> SLB
    SLB --"HTTP forwarding"--> SERVER_1
    SLB -.-> SERVER_2
    SERVER_1 --"enqueue function execution task"--> REDIS_QUEUE
    SERVER_2 -.-> REDIS_QUEUE
    REDIS_QUEUE -.-> WORKER_1
    REDIS_QUEUE --"dequeue function execution task"--> WORKER_2

    Beat --"enqueue function execution task\n(scheduled)"--> REDIS_QUEUE

3. Fully Independent Primary/Standby Deployment

In some cases, if a "fully independent primary/standby deployment" is needed, you can further split Redis and set the weight ratio of primary/standby nodes to 100:0 in the SLB (or other reverse proxy server).

At this point, since the primary and standby nodes are completely independent, and each runs a fully independent DataFlux Func, if DataFlux Func on both nodes is started at the same time, it will cause Cron Jobs to be executed repeatedly. Therefore, during normal operation, you can shut down DataFlux Func on the standby node, or write your own logic in the Script to prevent duplicate execution of tasks.

flowchart TB
    USER[User]
    MAIN_NODE_SERVER[Primary Node Server]
    MAIN_NODE_WORKER[Primary Node Worker]
    MAIN_NODE_BEAT[Primary Node Beat]
    MAIN_NODE_REDIS_QUEUE[Primary Node Redis Queue]
    BACKUP_NODE_SERVER[Standby Node Server]
    BACKUP_NODE_WORKER[Standby Node Worker]
    BACKUP_NODE_BEAT[Standby Node Beat]
    BACKUP_NODE_REDIS_QUEUE[Standby Node Redis Queue]

    USER --"HTTP request"--> SLB
    SLB --"HTTP forwarding"--> MAIN_NODE_SERVER
    SLB -.-> BACKUP_NODE_SERVER

    subgraph "Standby Node"
        direction TB
        BACKUP_NODE_SERVER --"enqueue function execution task"--> BACKUP_NODE_REDIS_QUEUE
        BACKUP_NODE_REDIS_QUEUE --"dequeue function execution task"--> BACKUP_NODE_WORKER

        BACKUP_NODE_BEAT --"enqueue function execution task\n(scheduled)"--> BACKUP_NODE_REDIS_QUEUE
    end

    subgraph "Primary Node"
        direction TB
        MAIN_NODE_SERVER --"enqueue function execution task"--> MAIN_NODE_REDIS_QUEUE
        MAIN_NODE_REDIS_QUEUE --"dequeue function execution task"--> MAIN_NODE_WORKER

        MAIN_NODE_BEAT --"enqueue function execution task\n(scheduled)"--> MAIN_NODE_REDIS_QUEUE
    end