Skip to content

Deployment and Maintenance / High Availability Deployment

DataFlux Func supports multiple deployments to meet high availability requirements.

This article mainly describes how to directly install and deploy a high availability DataFlux Func on servers.

When selecting a Redis high availability solution, do not use 'cluster edition Redis'; you may use 'master-replica edition Redis'

If you have previously installed DataFlux Func in standalone mode, when switching to high availability deployment, please refer to Deployment and Maintenance / Backup and Migration / Database Migration for migration

1. Multi-Replica Deployment

DataFlux Func's Server and Worker services both support multiple deployments to meet requirements such as high availability and scaling.

Generally, the execution efficiency bottleneck is on the Worker service (i.e., Python code). Therefore, the Server service only needs to avoid a single point of failure, while the Worker service needs to increase the number of replicas according to actual business volume.

When deploying multiple replicas, ensure that the contents of the user-config.yaml file on all services are exactly the same, and that they all connect to the same MySQL and Redis instance, with the resource directory mounted on the same storage.

At the same time, the Beat service, as the trigger for Cron Jobs, can run one and only one replica; otherwise, duplicate scheduled tasks may occur.

flowchart TB
    USER[User]
    SERVER_1[Server 1]
    SERVER_2[Server 2]
    WORKER_1[Worker 1]
    WORKER_2[Worker 2]
    WORKER_3[Worker 3]
    BEAT[Beat]
    REDIS[Redis]

    USER --HTTP Request--> SLB
    SLB --HTTP Forwarding--> SERVER_1
    SLB --HTTP Forwarding--> SERVER_2
    SERVER_1 --Enqueue function execution task--> REDIS
    SERVER_2 --> REDIS
    REDIS --Dequeue function execution task--> WORKER_1
    REDIS --Dequeue function execution task--> WORKER_2
    REDIS --Dequeue function execution task--> WORKER_3

    BEAT --"Enqueue function execution task\n(scheduled)"--> REDIS

2. Fully Independent Active-Standby Deployment

For now, let's not consider whether this deployment mode really counts as 'high availability'; assume such a deployment need does exist

A fully independent active-standby deployment is actually a deployment mode in which 2 sets of independent DataFlux Func instances are deployed separately (the Secret, MySQL, and Redis settings in the configuration file user-config.yaml are exactly the same).

Since the active and standby DataFlux Func instances run independently, the Beat services on the active and standby servers will trigger scheduled tasks in their respective environments, which will cause scheduled tasks to be triggered repeatedly.

To avoid this problem, you can shut down DataFlux Func on the standby node during normal operation, or write your own handling in a Script to prevent tasks from being executed repeatedly.

flowchart TB
    USER[User]
    MAIN_NODE_SERVER[Active Node Server]
    MAIN_NODE_WORKER[Active Node Worker]
    MAIN_NODE_BEAT[Active Node Beat]
    MAIN_NODE_REDIS_QUEUE[Active Node Redis Queue]
    BACKUP_NODE_SERVER[Standby Node Server]
    BACKUP_NODE_WORKER[Standby Node Worker]
    BACKUP_NODE_BEAT[Standby Node Beat]
    BACKUP_NODE_REDIS_QUEUE[Standby Node Redis Queue]

    USER --HTTP Request--> SLB
    SLB --HTTP Forwarding--> MAIN_NODE_SERVER
    SLB -.-> BACKUP_NODE_SERVER

    subgraph "Standby Node - Off"
        direction TB
        BACKUP_NODE_SERVER --Enqueue function execution task--> BACKUP_NODE_REDIS_QUEUE
        BACKUP_NODE_REDIS_QUEUE --Dequeue function execution task--> BACKUP_NODE_WORKER

        BACKUP_NODE_BEAT --"Enqueue function execution task\n(scheduled)"--> BACKUP_NODE_REDIS_QUEUE
    end

    subgraph "Active Node"
        direction TB
        MAIN_NODE_SERVER --Enqueue function execution task--> MAIN_NODE_REDIS_QUEUE
        MAIN_NODE_REDIS_QUEUE --Dequeue function execution task--> MAIN_NODE_WORKER

        MAIN_NODE_BEAT --"Enqueue function execution task\n(scheduled)"--> MAIN_NODE_REDIS_QUEUE
    end