Skip to content

Deployment and Maintenance / Backup and Migration

This article mainly introduces how to backup and migrate DataFlux Func data.

1. Database Backup

DataFlux Func automatically backs up the database regularly as .sql files, which are saved in the DataFlux Func Database Backup Directory.

This feature uses mysqldump to back up the database, and the MySQL user connecting to the database requires sufficient permissions to perform the backup, including:

  • SELECT
  • RELOAD
  • LOCK TABLES
  • REPLICATION CLIENT
  • SHOW VIEW
  • PROCESS

The automatic backup includes:

  1. Complete database table structures
  2. All data except logs and operation records

By default, the database backup files are saved in the following locations:

Environment Location
Inside container /data/sqldump/
On host machine {installation directory}/data/sqldump/

By default, the database backup files are backed up every hour on the hour, and are retained for up to 7 days (a total of 168 files). The file naming convention is as follows:

Text Only
1
dataflux-func-sqldump-YYYYMMDD-hhmmss.sql

2. Export / Import Database

In addition to the regular database backup provided by DataFlux Func, you can also directly use mysqldump to export and import data.

2.1 Export Data

The reference command for exporting data is as follows:

Bash
1
docker exec {MySQL container ID} sh -c 'exec mysqldump --user=root --password="$MYSQL_ROOT_PASSWORD" --hex-blob --default-character-set=utf8mb4 --skip-extended-insert --databases dataflux_func' > {backup file on host machine}

2.2 Import Data

The reference command for importing data is as follows:

Bash
1
docker exec -i {MySQL container ID} sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < {backup file on host machine}

3. Migrate Database

If the system has passed the initial single-machine verification phase after installation and needs to switch the database to an external database (e.g., Alibaba Cloud RDS, Redis), you can follow the steps below:

When using an external database, ensure that the MySQL version is 5.7 or higher, and the Redis version is 5.0 or higher

DataFlux Func does not support Redis cluster deployment

3.1 Confirm Environment

Create a database in the external database instance and ensure the following configurations:

  • innodb-large-prefix=on
  • character-set-server=utf8mb4
  • collation-server=utf8mb4_unicode_ci

3.2 Import Data

Find the most recent MySQL database backup file based on the "Database Backup" section above and import it into the external database.

3.3 Modify DataFlux Func Configuration

Locate the DataFlux Func Configuration user-config.yaml and add/modify the following fields based on the actual situation:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# MySQL connection configuration
MYSQL_HOST    : '127.0.0.1'    # MySQL connection address
MYSQL_PORT    : 3306           # MySQL port number
MYSQL_USER    : root           # MySQL username
MYSQL_PASSWORD: ''             # MySQL password
MYSQL_DATABASE: dataflux_func  # MySQL database name

# Redis connection configuration
REDIS_HOST    : '127.0.0.1'    # Redis connection address
REDIS_PORT    : 6379           # Redis port number
REDIS_DATABASE: 5              # Redis database number (recommended: to avoid conflicts with other applications, do not use the default database 0)
REDIS_PASSWORD: ''             # Redis password (leave empty if no password)
REDIS_USE_TLS : false          # Redis connection TLS support (true / false)

3.4 Modify Docker Stack Configuration

Locate the Docker Stack Configuration docker-stack.yaml and delete the MySQL and Redis related parts (you can comment them out).

3.5 Restart DataFlux Func

Restart the entire DataFlux Func according to Deployment and Maintenance / Upgrade and Restart.

4. Change Installation Directory

In some cases, you may encounter insufficient disk space where DataFlux Func was originally installed, and you need to move the installation directory to a new, larger disk.

You can follow the steps below to change the installation directory.

4.1 Shut Down DataFlux Func

  1. Use the command docker stack rm dataflux-func to shut down DataFlux Func (this step may take some time).
  2. Use docker ps to confirm that all containers have exited.

4.2 Move Installation Directory

Use the command cp {installation directory} {new installation directory} to copy the entire DataFlux Func installation directory to the new location.

4.3 Modify Installation Directory Record

Open /etc/dataflux-func and change the recorded installation directory to the new installation directory.

4.4 Modify Docker Stack Configuration

Locate the Docker Stack Configuration docker-stack.yaml and change all volumes related to the installation directory to the new installation directory, such as:

Diff
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
 version: '3.1'

 services:
   # Other content omitted
   server:
     image: dataflux-func.com/dataflux-func:x.y.z
     labels:
       - server
     volumes:
-      - "/usr/local/dataflux-func/data:/data"
+      - "/my-workspace/dataflux-func/data:/data"
     networks:
       - datafluxfunc
       - default
     ports:
       - "8088:8088"
     command: ./run-server.sh

 # Other content omitted
  1. Use docker stack deploy dataflux-func -c {installation directory}/docker-stack.yaml --resolve-image never to restart all services.

Since the image files in the installation package have already been imported locally, adding the --resolve-image never parameter can prevent Docker from performing unnecessary image checks when starting containers.