Skip to content

Script Development / Function Execution Process

This article mainly introduces the specific task process when a function is called.

1. Basic Process

In general, when calling a function in DataFlux Func (Automata), the specific execution flow is as follows:

  1. The user initiates an HTTP request to the Server
  2. The Server generates a function execution task and pushes it to the Redis queue
  3. The Worker pops out the function execution task from the Redis queue and executes it

For scheduled tasks, there is no step of "the user initiating an HTTP request", as the task is directly generated by the Beat service. The specific execution flow is as follows:

  1. Beat periodically generates function execution tasks and pushes them to the Redis queue
  2. The Worker pops out the function execution task from the Redis queue and executes it

2. Single-Machine Deployment

For single-machine deployment of DataFlux Func, the entire process is very simple:

flowchart TB
    User --HTTP Request--> Server
    Server --Function Execution Task Enqueue--> Redis Queue
    Redis Queue --Function Execution Task Dequeue--> Worker

    Beat --"Function Execution Task Enqueue\n(Scheduled)"--> Redis Queue

3. Multi-Replica Deployment

For multi-replica deployments of DataFlux Func, due to the presence of SLB (or other reverse proxy services), any Server may receive requests.

At the same time, since each replica connects to the same Redis, each task will only be fetched and executed by one Worker:

flowchart TB
    User --HTTP Request--> SLB
    SLB --HTTP Forwarding--> Server1
    SLB -.-> Server2
    Server1 --Function Execution Task Enqueue--> Redis Queue
    Server2 -.-> Redis Queue
    Redis Queue -.-> Worker1
    Redis Queue --Function Execution Task Dequeue--> Worker2

    Beat --"Function Execution Task Enqueue\n(Scheduled)"--> Redis Queue

4. Fully Independent Primary/Backup Deployment

In some cases, if you need to achieve "fully independent primary/backup deployment," Redis can be further split, and the weight ratio of primary and backup nodes in SLB (or other reverse proxy servers) can be set to 100:0.

At this point, since the primary and backup nodes are completely independent, each runs its own fully independent DataFlux Func. If DataFlux Func on both the primary and backup nodes is started simultaneously, scheduled tasks may execute repeatedly. Therefore, the DataFlux Func on the backup node can be turned off during normal times, or logic can be written in the script to avoid repeated task execution.

flowchart TB
    User --HTTP Request--> SLB
    SLB --HTTP Forwarding--> Primary Node Server
    SLB -.-> Backup Node Server

    subgraph Backup Node
        direction TB
        Backup Node Server --Function Execution Task Enqueue--> Backup Node Redis Queue
        Backup Node Redis Queue --Function Execution Task Dequeue--> Backup Node Worker

        Backup Node Beat --"Function Execution Task Enqueue\n(Scheduled)"--> Backup Node Redis Queue
    end

    subgraph Primary Node
        direction TB
        Primary Node Server --Function Execution Task Enqueue--> Primary Node Redis Queue
        Primary Node Redis Queue --Function Execution Task Dequeue--> Primary Node Worker

        Primary Node Beat --"Function Execution Task Enqueue\n(Scheduled)"--> Primary Node Redis Queue
    end