Skip to content

Deployment and Maintenance / Backup and Migration

This article mainly describes how to back up and migrate DataFlux Func data.

1. Database Backup

DataFlux Func automatically backs up the database to .sql files on a regular basis and stores them in the DataFlux Func database backup directory.

This feature backs up the database via mysqldump. The MySQL user connecting to the database needs sufficient permissions to back up properly, including:

  • SELECT
  • RELOAD
  • LOCK TABLES
  • REPLICATION CLIENT
  • SHOW VIEW
  • PROCESS

The automatic backup includes:

  1. The complete database table structure
  2. All data except logs and operation records

By default, database backup files are saved in the following locations:

Environment Location
Inside container /data/sqldump/
On host machine {installation directory}/data/sqldump/

By default, the database backup is performed once every hour on the hour, and up to 7 days (168 files in total) are retained. The file naming convention is as follows:

Text Only
1
dataflux-func-sqldump-YYYYMMDD-hhmmss.sql

2. Export / Import Database

In addition to the regular database backup built into DataFlux Func, you can also use mysqldump directly to export and import data.

2.1 Export Data

The reference command for the export operation is as follows:

Bash
1
docker exec {MySQL Container ID} sh -c 'exec mysqldump --user=root --password="$MYSQL_ROOT_PASSWORD" --hex-blob --default-character-set=utf8mb4 --skip-extended-insert --databases dataflux_func' > {backup file on host machine}

2.2 Import Data

The reference command for the import operation is as follows:

Bash
1
docker exec -i {MySQL Container ID} sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < {backup file on host machine}

3. Migrate Database

If the system has passed the initial standalone verification stage after installation, and you need to switch the database to an external database (such as Alibaba Cloud RDS, Redis), you can follow the steps below:

When using an external database, make sure the MySQL version is 5.7 or above, and the Redis version is 5.0 or above

DataFlux Func does not support cluster-deployed Redis

3.1 Confirm Environment

Create a database in the external database instance and ensure the following configuration items:

  • innodb-large-prefix=on
  • character-set-server=utf8mb4
  • collation-server=utf8mb4_unicode_ci

3.2 Import Data

According to the "Database Backup" section above, find the most recent MySQL database backup file and import it into the external database.

3.3 Modify DataFlux Func Configuration

Locate the DataFlux Func configuration user-config.yaml, and add / modify the following fields according to the actual situation:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# MySQL connection configuration
MYSQL_HOST    : '127.0.0.1'    # MySQL connection address
MYSQL_PORT    : 3306           # MySQL port number
MYSQL_USER    : root           # MySQL username
MYSQL_PASSWORD: ''             # MySQL password
MYSQL_DATABASE: dataflux_func  # MySQL database name

# Redis connection configuration
REDIS_HOST    : '127.0.0.1'    # Redis connection address
REDIS_PORT    : 6379           # Redis port number
REDIS_DATABASE: 5              # Redis database number (recommended: to avoid conflicts with other applications, do not use the default database 0)
REDIS_PASSWORD: ''             # Redis password (leave blank if no password)
REDIS_USE_TLS : false          # Redis connection TLS support (true / false)

3.4 Modify Docker Stack Configuration

Locate the Docker Stack configuration docker-stack.yaml, and remove (comment out) the MySQL and Redis related sections.

3.5 Restart DataFlux Func

According to Deployment and Maintenance / Upgrade and Restart, restart the entire DataFlux Func.

4. Change Installation Directory

In some cases, the disk where DataFlux Func was originally installed may run out of capacity, and you need to change the installation directory to a new disk with larger capacity.

In this case, you can follow the steps below to change the installation directory.

4.1 Shut Down DataFlux Func

  1. Use the docker stack rm dataflux-func command to shut down DataFlux Func (this step may take some time)
  2. Use docker ps to confirm that all containers have exited
  3. Use the cp {installation directory} {new installation directory} command to copy the entire DataFlux Func installation directory to the new location.
  4. Open /etc/dataflux-func and change the installation directory recorded therein to the new installation directory.
  5. Modify the Docker Stack configuration

Locate the Docker Stack configuration docker-stack.yaml, and change all volumes entries that involve the installation directory to the new installation directory, for example:

Diff
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
 version: '3.1'

 services:
   # Other content omitted
   server:
     image: dataflux-func.com/dataflux-func:x.y.z
     labels:
       - server
     volumes:
-      - "/usr/local/dataflux-func/data:/data"
+      - "/my-workspace/dataflux-func/data:/data"
     networks:
       - datafluxfunc
       - default
     ports:
       - "8088:8088"
     command: ./run-server.sh

 # Other content omitted
  1. Use the docker stack deploy dataflux-func -c {installation directory}/docker-stack.yaml --resolve-image never command to restart all services

Since the image files in the installation package have already been imported locally, adding the --resolve-image never parameter can prevent Docker from performing unnecessary image checks when starting containers