Skip to content

Script Development / Function Execution Process

This article mainly introduces the specific task process when a function is called.

1. Basic Process

Overall, when calling a function in DataFlux Func, the specific execution flow is as follows:

  1. User initiates an HTTP request to the Server.
  2. Server generates a function execution task and pushes it to the Redis queue.
  3. Worker pops the function execution task from the Redis queue and executes it.

For scheduled tasks, there is no "user initiates an HTTP request" step. The tasks are directly generated by the Beat service. The specific execution flow is as follows:

  1. Beat periodically generates function execution tasks and pushes them to the Redis queue.
  2. Worker pops the function execution task from the Redis queue and executes it.

2. Single-Instance Deployment

For a single-instance deployment of DataFlux Func, the entire process is very simple:

flowchart TB
    USER[User]
    SERVER[Server]
    WORKER[Worker]
    REDIS_QUEUE[Redis Queue]

    USER --HTTP Request--> SERVER
    SERVER --Enqueue Function Task--> REDIS_QUEUE
    REDIS_QUEUE --Dequeue Function Task--> WORKER

    Beat --"Enqueue Function Task\n(Scheduled)"--> REDIS_QUEUE

2. Multi-Replica Deployment

For a multi-replica deployment of DataFlux Func, due to the presence of an SLB (or other reverse proxy service), any Server might receive the request.

At the same time, since each replica connects to the same Redis, each task will only be fetched and executed by any one Worker:

flowchart TB
    USER[User]
    SERVER_1[Server 1]
    SERVER_2[Server 2]
    WORKER_1[Worker 1]
    WORKER_2[Worker 2]
    REDIS_QUEUE[Redis Queue]

    USER --HTTP Request--> SLB
    SLB --HTTP Forward--> SERVER_1
    SLB -.-> SERVER_2
    SERVER_1 --Enqueue Function Task--> REDIS_QUEUE
    SERVER_2 -.-> REDIS_QUEUE
    REDIS_QUEUE -.-> WORKER_1
    REDIS_QUEUE --Dequeue Function Task--> WORKER_2

    Beat --"Enqueue Function Task\n(Scheduled)"--> REDIS_QUEUE

3. Fully Independent Active-Standby Deployment

In some cases, if a "fully independent active-standby deployment" needs to be implemented, the Redis can be further split, and the weight ratio of the active and standby nodes in the SLB (or other reverse proxy server) can be set to 100:0.

At this point, since the active and standby nodes are completely independent from each other, each runs as a completely separate DataFlux Func. If the DataFlux Func on both the active and standby nodes are started simultaneously, it will cause duplicate execution of scheduled tasks. Therefore, you can usually shut down the DataFlux Func on the standby node, or write your own handling in the scripts to avoid duplicate task execution.

flowchart TB
    USER[User]
    MAIN_NODE_SERVER[Active Node Server]
    MAIN_NODE_WORKER[Active Node Worker]
    MAIN_NODE_BEAT[Active Node Beat]
    MAIN_NODE_REDIS_QUEUE[Active Node Redis Queue]
    BACKUP_NODE_SERVER[Standby Node Server]
    BACKUP_NODE_WORKER[Standby Node Worker]
    BACKUP_NODE_BEAT[Standby Node Beat]
    BACKUP_NODE_REDIS_QUEUE[Standby Node Redis Queue]

    USER --HTTP Request--> SLB
    SLB --HTTP Forward--> MAIN_NODE_SERVER
    SLB -.-> BACKUP_NODE_SERVER

    subgraph Standby Node
        direction TB
        BACKUP_NODE_SERVER --Enqueue Function Task--> BACKUP_NODE_REDIS_QUEUE
        BACKUP_NODE_REDIS_QUEUE --Dequeue Function Task--> BACKUP_NODE_WORKER

        BACKUP_NODE_BEAT --"Enqueue Function Task\n(Scheduled)"--> BACKUP_NODE_REDIS_QUEUE
    end

    subgraph Active Node
        direction TB
        MAIN_NODE_SERVER --Enqueue Function Task--> MAIN_NODE_REDIS_QUEUE
        MAIN_NODE_REDIS_QUEUE --Dequeue Function Task--> MAIN_NODE_WORKER

        MAIN_NODE_BEAT --"Enqueue Function Task\n(Scheduled)"--> MAIN_NODE_REDIS_QUEUE
    end