Skip to content

Deployment and Maintenance / Backup and Migration

This article mainly introduces how to backup and migrate DataFlux Func data.

1. Database Backup

DataFlux Func automatically backs up the database into .sql files periodically and saves them in the DataFlux Func Database Backup Directory.

This feature uses mysqldump to backup the database. The MySQL user connecting to the database requires sufficient permissions for normal backup, including:

  • SELECT
  • RELOAD
  • LOCK TABLES
  • REPLICATION CLIENT
  • SHOW VIEW
  • PROCESS

The automatic backup includes:

  1. Complete database table creation structure
  2. All data except logs and operation records

By default, the database backup files are saved in the following locations:

Environment Location
Inside Container /data/sqldump/
On Host Machine {Installation Directory}/data/sqldump/

By default, the database is backed up every hour on the hour, with a maximum retention of 7 days (168 files in total). The file naming rule is as follows:

Text Only
1
dataflux-func-sqldump-YYYYMMDD-hhmmss.sql

2. Export / Import Database

In addition to the periodic database backup provided by DataFlux Func, you can also directly use mysqldump to export and import data.

2.1 Export Data

The reference command for the export operation is as follows:

Bash
1
docker exec {MySQL Container ID} sh -c 'exec mysqldump --user=root --password="$MYSQL_ROOT_PASSWORD" --hex-blob --default-character-set=utf8mb4 --skip-extended-insert --databases dataflux_func' > {Backup file on the host machine}

2.2 Import Data

The reference command for the import operation is as follows:

Bash
1
docker exec -i {MySQL Container ID} sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < {Backup file on the host machine}

3. Migrate Database

If, after the initial single-machine verification phase post-installation, you need to switch the database to an external database (e.g., Alibaba Cloud RDS, Redis), you can follow the steps below:

When using an external database, ensure the MySQL version is 5.7 or above, and the Redis version is 5.0 or above

DataFlux Func does not support Redis deployed in cluster mode

3.1 Confirm Environment

Create a database in the external database instance and ensure the following configuration items are set:

  • innodb-large-prefix=on
  • character-set-server=utf8mb4
  • collation-server=utf8mb4_unicode_ci

3.2 Import Data

Find the most recent MySQL database backup file based on the "Database Backup" section above and import it into the external database.

3.3 Modify DataFlux Func Configuration

Locate the DataFlux Func Configuration user-config.yaml, and add/modify the following fields according to the actual situation:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# MySQL connection configuration
MYSQL_HOST    : '127.0.0.1'    # MySQL connection address
MYSQL_PORT    : 3306           # MySQL port number
MYSQL_USER    : root           # MySQL username
MYSQL_PASSWORD: ''             # MySQL password
MYSQL_DATABASE: dataflux_func  # MySQL database name

# Redis connection configuration
REDIS_HOST    : '127.0.0.1'    # Redis connection address
REDIS_PORT    : 6379           # Redis port number
REDIS_DATABASE: 5              # Redis database number (Suggestion: To avoid conflicts with other applications, do not use the default database 0)
REDIS_PASSWORD: ''             # Redis password (leave empty if no password)
REDIS_USE_TLS : false          # Redis connection TLS support (true / false)

3.4 Modify Docker Stack Configuration

Locate the Docker Stack Configuration docker-stack.yaml, and delete (or comment out) the MySQL and Redis related sections.

3.5 Restart DataFlux Func

Restart the entire DataFlux Func according to Deployment and Maintenance / Upgrade and Restart.

4. Change Installation Directory

In some cases, you might encounter insufficient disk capacity where DataFlux Func was originally installed and need to change the installation directory to a new, larger-capacity disk.

You can follow the steps below to change the installation directory.

4.1 Stop DataFlux Func

  1. Use the command docker stack rm dataflux-func to stop DataFlux Func (this step may take some time).
  2. Use docker ps to confirm all containers have exited.
  3. Use the command cp {Installation Directory} {New Installation Directory} to copy the entire DataFlux Func installation directory to the new location.
  4. Open /etc/dataflux-func and change the recorded installation directory to the new one.
  5. Modify Docker Stack Configuration

Locate the Docker Stack Configuration docker-stack.yaml, and change all volumes entries involving the installation directory to the new installation directory, for example:

Diff
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
 version: '3.1'

 services:
   # Other content omitted
   server:
     image: dataflux-func.com/dataflux-func:x.y.z
     labels:
       - server
     volumes:
-      - "/usr/local/dataflux-func/data:/data"
+      - "/my-workspace/dataflux-func/data:/data"
     networks:
       - datafluxfunc
       - default
     ports:
       - "8088:8088"
     command: ./run-server.sh

 # Other content omitted
  1. Use docker stack deploy dataflux-func -c {Installation Directory}/docker-stack.yaml --resolve-image never to restart all services.

Since the image files in the installation package have already been imported locally, adding the --resolve-image never parameter can prevent Docker from performing unnecessary image checks when starting containers.