Skip to content

Deployment and Maintenance / Standalone Deployment

This article mainly introduces how to directly install and deploy the standalone version of DataFlux Func on a server.

For installing DataFlux Func using Helm in k8s, please refer to Deployment and Maintenance / Installation and Deployment / Helm Deployment

1. Download the Installation Package

DataFlux Func supports downloading the required installation files and bringing them into a non-public network environment via USB drives or other mobile devices for installation.

The downloaded "installation package" comes with an automatic installation script, Docker, etc., and can be executed to perform the installation (details below).

1.1 One-Command Download

For Linux, macOS, and other systems, it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the required installation files for DataFlux Func. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Please confirm the system requirements and server configuration before installation

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)"

If the architecture of the download differs from the final installation server architecture, you can specify the architecture by adding an architecture option:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=x86_64
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=aarch64

After the command is executed, all installation files are saved in the automatically created dataflux-func-portable-{architecture}-{version} directory.

  • When installing DataFlux Func on a server without public network access, you can first download it on your local machine and then copy the entire directory to the target machine via USB drives or other mobile storage devices, or using the scp tool.
  • When installing DataFlux Func on a server with public network access, you can directly download it on the server.

1.2 Manual Download

For systems where using shell commands is inconvenient, you can manually download the required installation files.

If manual download is needed, here is the complete list of files:

# Content File Name x86_64 Architecture (Original) aarch64 Architecture (Original)
1 Docker Binary Program docker-24.0.9.tgz Download Download
2 DataFlux Func Image dataflux-func.tar.gz Download Download
3 MySQL/MariaDB Image mysql.tar.gz Download Download
4 Redis Image redis.tar.gz Download Download
5 Docker Service Configuration File docker.service Download Download
6 DataFluxFunc Installation Script run-portable.sh Download Download
7 Docker Stack Configuration File Template docker-stack.example.yaml Download Download
8 Image List image-list Download Download
9 Version Information version Download Download

After manually downloading all installation files, place them in the same directory.

1.3 Download Options

When executing the download script, you can specify download options to meet personalized needs.

Example:

Note: Before the specified download options, there is a -- (two hyphens)

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

Supported download options are as follows:

--arch={architecture} Specify Architecture

The download script defaults to downloading the installation package files that match the local machine's architecture. If you need to download installation files for a different architecture, you can specify this parameter.

Available architectures are as follows:

Architecture Description Parameter Example
Intel Intel / AMD 64-bit Processor --arch=x86_64
ARM ARM64v8 Processor --arch=aarch64

--download-dir={download directory} Specify Download Directory

Added in version 2.6.1

This parameter is suitable for automated script deployment of DataFlux Func

The download script defaults to creating/clearing the dataflux-func-portable-{architecture}-{version} directory in the current directory and downloading the installation files there.

If you need to download to a specified directory, you can specify this parameter, such as:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

2. Execute Installation

Enter the installation package directory downloaded in the previous section and run the one-click installation script run-portable.sh:

Please confirm the system requirements and server configuration before installation

DataFlux Func does not support macOS, Windows. Please copy it to a Linux system before running the installation

Bash
1
sudo /bin/bash {installation file directory}/run-portable.sh

Using the automatic installation script, you can quickly install and run it within a few minutes. The automatic configuration includes:

  • Running MySQL, Redis, DataFlux Func (including Server, Worker, Beat)
  • Automatically creating and saving all data in the /usr/local/dataflux-func/ directory (including MySQL data, Redis data, DataFlux Func configuration, log files, etc.)
  • Randomly generating MySQL root user password and system Secret, and saving them in the DataFlux Func configuration file
  • Redis has no password
  • MySQL and Redis are not accessible externally

After execution, you can use a browser to access http://{server IP address/domain}:8088 for the initialization operation interface.

If the running environment has poor performance, you should use the docker ps command to confirm that all components have successfully started before accessing (see the list below)

  1. dataflux-func_server
  2. dataflux-func_worker-0
  3. dataflux-func_worker-1
  4. dataflux-func_worker-2
  5. dataflux-func_worker-3
  6. dataflux-func_worker-5
  7. dataflux-func_worker-6
  8. dataflux-func_beat
  9. dataflux-func_mysql
  10. dataflux-func_redis
  1. dataflux-func_server
  2. dataflux-func_worker-0
  3. dataflux-func_worker-1-6
  4. dataflux-func_worker-7
  5. dataflux-func_worker-8-9
  6. dataflux-func_beat
  7. dataflux-func_mysql
  8. dataflux-func_redis

2.1 Installation Options

When executing the installation script, you can specify installation options to meet personalized needs, such as:

Bash
1
sudo /bin/bash run-portable.sh --port=80

--mini Install Mini Version

For low-configuration environments where resources need to be saved.

When enabled:

  • Only a single Worker is started to listen to all queues
  • Heavy load tasks are more likely to cause queue blocking and lag
  • System tasks and function tasks share the processing queue and will affect each other
  • System requirements are reduced to:
    • CPU cores >= 1
    • Memory capacity >= 2GB
  • If built-in MySQL and Redis are not used, system requirements can be further reduced

--port={port number} Specify Listening Port Number

DataFlux Func defaults to using port 8088 for access. If this port is occupied by another program, you can choose another port, such as 9000.

--install-dir={installation directory} Specify Installation Directory

When you need to install to a different path than the default /usr/local/dataflux-func, you can specify this parameter.

--no-mysql Disable Built-in MySQL

When using an existing MySQL database, you can specify this parameter to prevent MySQL from starting on the local machine.

After enabling this option, you need to specify the correct MySQL connection information on the configuration page after installation

--no-redis Disable Built-in Redis

When using an existing Redis database, you can specify this parameter to prevent Redis from starting on the local machine.

After enabling this option, you need to specify the correct Redis connection information on the configuration page after installation

--auto-setup Automatically Execute Configuration

Added in version 2.6.0

This parameter is suitable for automated script deployment of DataFlux Func

When enabled, the configuration and database initialization are automatically performed, and there will be no configuration page, such as:

Bash
1
sudo /bin/bash run-portable.sh --auto-setup

Additionally, when the --auto-setup option is enabled, you can add other --auto-setup-* options to adjust the automatic configuration. The extra options for automatic configuration are as follows:

Extra Options for Automatic Configuration Default Value Description
--auto-setup-admin-username={username} admin Specify the administrator username
--auto-setup-admin-password={password} admin Specify the administrator password
--auto-setup-ak-secret={AK Secret} Automatically generated Automatically create AK
and use the specified value as the AK Secret
--auto-setup-ak-id={AK ID} ak-auto-setup AK ID when automatically creating AK
(requires the --auto-setup-ak-secret option)

If you need to automatically configure and specify the administrator password as AdminPass, the complete command is as follows:

Bash
1
sudo /bin/bash run-portable.sh --auto-setup --auto-setup-admin-password='AdminPass'

3. Verify Installation

By default, DataFlux Func comes with some example scripts after installation.

Perform the following steps to verify the installation:

  1. Click the "Script Editor" in the top navigation bar, and select "Script Lib" - "Examples" - "Basic Demo" in the left sidebar.
  2. In the right script editor, click "Edit" to enter edit mode, select the "hello_world" function, and click the "Execute" button to run the function.
  3. At this point, if you can see the function's return value in the "Script Output" at the bottom.

The installation is now verified.

3.1 Service Descriptions

By default, after DataFlux Func starts normally, the following services are running:

Queues #7, #8, #9 are reserved for the Data Platform attached version of Func. In the standalone version of Func, they are reserved queues.

Name Description
dataflux-func_server The front-end service of DataFlux Func.
Mainly provides the web interface, API interfaces, etc.
dataflux-func_worker-0 Python worker unit listening to queues #0, #4, #7, #8, #9.
Mainly handles internal tasks of DataFlux Func (#0 queue) and tasks that mistakenly enter reserved queues.
dataflux-func_worker-1 Python worker unit listening to queue #1.
Mainly handles general-purpose user function tasks
dataflux-func_worker-2 Python worker unit listening to queue #2.
Mainly handles automatically triggered user function tasks
dataflux-func_worker-3 Python worker unit listening to queue #3.
Mainly handles batch processing user function tasks
dataflux-func_worker-5 Python worker unit listening to queue #5.
Mainly handles debugging tasks for scripts run in the web interface
dataflux-func_worker-6 Python worker unit listening to queue #6.
Mainly handles tasks executed by connector subscription messages
dataflux-func_beat Trigger for automatically triggered tasks, only one can be enabled globally
dataflux-func_mysql Built-in MySQL of DataFlux Func
dataflux-func_redis Built-in Redis of DataFlux Func
Name Description
dataflux-func_server The front-end service of DataFlux Func.
Mainly provides the web interface, API interfaces, etc.
dataflux-func_worker-0 Python worker unit listening to queue #0.
Mainly handles internal tasks of DataFlux Func
dataflux-func_worker-1-6 Python worker unit listening to queues #1, #2, #3, #4, #5, #6.
Mainly handles Sync API (old version: Auth Link)
dataflux-func_worker-7 Python worker unit listening to queue #7.
Mainly handles debugging tasks for scripts run in the web interface
dataflux-func_worker-8-9 Python worker unit listening to queues #8, #9.
Mainly handles asynchronous tasks (automatically triggered, batch processing)
dataflux-func_beat Trigger for automatically triggered tasks, only one can be enabled globally
dataflux-func_mysql Built-in MySQL of DataFlux Func
dataflux-func_redis Built-in Redis of DataFlux Func

3.2 Data Storage Locations

DataFlux Func requires storing various types of data during operation. The main content and storage locations are as follows:

Storage Save Path Storage Content
MySQL {installation directory}/mysql/ Most of the data generated during UI operations, including but not limited to:
1. Scripts, Connector configurations, environment variables
2. User information, Sync API (old version: Auth Link), Cron Job (old version: Automatically Triggered Configuration), Async API (old version: Batch)
3. Operation records, script execution records, import/export records, etc.
Redis {installation directory}/redis/ Mainly used for caching and queues, including but not limited to:
1. User login information
2. Various caches established during script execution
3. Script execution task queue
4. Func's own monitoring data, etc.
Directory {installation directory}/data/ Mainly used for data that must exist in file form, see below for details
Directory {installation directory}/data/resources/ Resource folder (i.e., the root directory of "File Service")
Directory {installation directory}/data/resources/extra-python-packages/ Third-party packages installed by PIP
Directory {installation directory}/data/resources/script-market/ Script Market download data
Directory {installation directory}/data/sqldump/ Database automatic backups
Directory {installation directory}/data/logs/ System logs
File {installation directory}/data/user-config.yaml DataFlux Func system configuration
File /etc/dataflux-func DataFlux Func installation directory record
Used to correctly obtain the current installation location of DataFlux Func during upgrades

4. Using External Databases

When installing DataFlux Func for the first time, if you need to use external databases (MySQL, Redis), you can specify to disable the built-in MySQL and Redis when executing the installation script, such as:

Bash
1
sudo /bin/bash {installation file directory}/run-portable.sh --no-mysql --no-redis

After installation, on the first run interface, click "Show More Configuration" to modify the MySQL and Redis configurations.

For MySQL and Redis configuration requirements, please refer to Deployment and Maintenance / System Requirements

If you need to migrate the existing DataFlux Func database to an external one, please refer to Deployment and Maintenance / Daily Operations / Migrating Databases