Deployment and Maintenance / Standalone Deployment

This article mainly introduces how to install and deploy the standalone version of DataFlux Func directly on a server.

For installing DataFlux Func with Helm in k8s, please refer to Deployment and Maintenance / Installation and Deployment / Helm Deployment

1. Download the Installation Package

DataFlux Func supports an "installation package" that allows you to download the required installation files and then use a USB drive or other portable storage device to bring them into an environment without public network access for installation.

The downloaded "installation package" itself includes the automatic installation Script, Docker, etc. You can run it to perform the installation (see below for details).

1.1 One-Command Download

For systems such as Linux and macOS, it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the required installation files for DataFlux Func. The download Script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Before installation, please confirm the system requirements and server configuration

bash -c "$(curl -fsSL docs.dataflux-func.com/download)"

If the architecture you need to download is different from the architecture of the server where it will eventually be installed, you can add an architecture option to specify the architecture:

x86 ArchitectureARM64 Architecture

bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=x86_64

bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=aarch64

After the command is executed, all installation files are saved in the automatically created dataflux-func-portable-{architecture}-{version} directory.

• If you need to install DataFlux Func on a server without public network access, you can first download it on your local machine, and then copy the entire directory to the target machine using a USB drive or other portable storage device, or tools such as scp
• If you need to install DataFlux Func on a server with public network access, simply download it directly on the server

1.2 Manual Download

For systems where using shell commands is inconvenient, you can manually download the required installation files.

If you need to download manually, the following is the complete list of files:

#	Content	File Name	x86_64 Architecture (Original)	aarch64 Architecture (Original)
1	Docker binary	docker-24.0.9.tgz	Download	Download
2	DataFlux Func image	dataflux-func.tar.gz	Download	Download
3	MySQL/MariaDB image	mysql-5.7.26.tar.gz	Download	Download
4	Redis image	redis-6.2.20.tar.gz	Download	Download
5	Docker service configuration file	docker.service	Download	Download
6	DataFlux Func installation Script	run-portable.sh	Download	Download
7	Docker Stack configuration file template	docker-stack.example.yaml	Download	Download
8	image list	image-list	Download	Download
9	version information	version	Download	Download

After manually downloading all installation files, place them in the same directory.

1.3 Download Options

When running the download Script, you can specify download options to meet individual needs.

For example:

Note: there is a -- (two hyphens) before the specified download options

bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

The supported download options are as follows:

--arch={architecture} Specify the architecture

By default, the download Script downloads installation package files with the same architecture as the local machine. If you need to download installation files for a different architecture, you can specify this parameter.

The available architectures are as follows:

Architecture	Description	Parameter Example
Intel	Intel / AMD 64-bit processor	--arch=x86_64
ARM	ARM64v8 processor	--arch=aarch64

--download-dir={download-directory} Specify the download directory

Added in version 2.6.1

This parameter is suitable for deploying DataFlux Func via automated Scripts

By default, the download Script will create/clear the dataflux-func-portable-{architecture}-{version-number} directory in the current directory and download the installation files to it.

If you need to download to a specified directory, you can specify this parameter, for example:

bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

2. Execute Installation

Enter the installation package directory downloaded above and run the one-click installation Script run-portable.sh:

Before installation, please confirm the system requirements and server configuration

DataFlux Func does not support macOS or Windows. Please copy it to a Linux system and then run the installation

sudo bash {installation-file-directory}/run-portable.sh

Using the automatic installation Script, you can quickly install and start running within a few minutes. The contents automatically configured are as follows:

• Run MySQL, Redis, DataFlux Func (including Server, Worker, Beat)
• Automatically create and save all data under the /usr/local/dataflux-func/ directory (including MySQL data, Redis data, DataFlux Func configuration, logs, and other files)
• Randomly generate the MySQL root user password and system Secret, and save them in the DataFlux Func configuration file
• Redis does not have a password set
• MySQL and Redis do not provide external access

After execution, you can use a browser to access http://{server IP address/domain}:8088 to enter the initialization page.

If the runtime environment has poor performance, use the docker ps command to confirm that all components have started successfully before accessing (see the list below)

3.2.0 and later versions3.1.0 and earlier versions

1. dataflux-func_server
2. dataflux-func_worker-0
3. dataflux-func_worker-1
4. dataflux-func_worker-2
5. dataflux-func_worker-3
6. dataflux-func_worker-5
7. dataflux-func_worker-6
8. dataflux-func_beat
9. dataflux-func_mysql
10. dataflux-func_redis

1. dataflux-func_server
2. dataflux-func_worker-0
3. dataflux-func_worker-1-6
4. dataflux-func_worker-7
5. dataflux-func_worker-8-9
6. dataflux-func_beat
7. dataflux-func_mysql
8. dataflux-func_redis

2.1 Installation Options

When executing the installation Script, you can specify installation options to meet individual requirements, for example:

sudo bash run-portable.sh --port=80

--mini Install the mini version

An installation mode for low-configuration environments where resources need to be conserved.

After enabling:

• Only a single Worker is started to listen to all queues
• Heavy-load tasks are more likely to cause queue blocking and stalling
• System tasks and function tasks share the processing queue and affect each other
• The system requirements are reduced to:
  • CPU cores >= 1
  • Memory capacity >= 2GB
• If the built-in MySQL and Redis are not used, the system requirements can be further reduced

--port={port-number} Specify the listening port number

By default, DataFlux Func uses port 8088 for access. If this port is occupied by another program, you can choose another port, for example, 9000.

--install-dir={installation-directory} Specify the installation directory

If you need to install to a path different from the default path /usr/local/dataflux-func, you can specify this parameter.

--no-mysql Disable the built-in MySQL

When you need to use an existing MySQL database, you can specify this parameter to prevent MySQL from being started on this host.

After enabling this option, you need to specify the correct MySQL connection information on the configuration page after installation is complete

--no-redis Disable the built-in Redis

When you need to use an existing Redis database, you can specify this parameter to prevent Redis from being started on this host.

After enabling this option, you need to specify the correct Redis connection information on the configuration page after installation is complete

--auto-setup Automatically perform configuration

Added in version 2.6.0

This parameter is suitable for deploying DataFlux Func via automated Scripts

After enabling, configuration and database initialization are automatically performed, and there will no longer be a configuration page. For example:

sudo bash run-portable.sh --auto-setup

In addition, after enabling the --auto-setup option, you can add other --auto-setup-* options to adjust the automatic configuration. The additional automatic configuration options are as follows:

Additional automatic configuration option	Default value	Description
--auto-setup-admin-username={username}	admin	Specify the administrator username
--auto-setup-admin-password={password}	admin	Specify the administrator password
--auto-setup-ak-secret={AK Secret}	Automatically generated	Automatically create AK and use the specified value as the AK Secret
--auto-setup-ak-id={AK ID}	ak-auto-setup	The AK ID used when automatically creating an AK (must be used together with the --auto-setup-ak-secret option)

If automatic configuration is required and the administrator password is specified as AdminPass, the complete command is as follows:

sudo bash run-portable.sh --auto-setup --auto-setup-admin-password='AdminPass'

3. Verify Installation

After the default installation of DataFlux Func is complete, it already comes with some sample Scripts.

Perform the following operations in sequence to verify the installation:

1. Click "Script Editor" in the top navigation bar, and then select "Script Lib" - "Examples" - "Basic Demo" in the left sidebar.
2. At the top of the script editor on the right, click "Edit" to enter edit mode, select the "hello_world" function, and click the "Execute" button to run the function.
3. At this point, if you can see the function's return value normally in the "Script Output" at the bottom.

At this point, the installation verification is complete.

3.1 Service Descriptions

By default, after DataFlux Func starts normally, the following services are running:

3.2.0 and later versions3.1.0 and earlier versions

Queues #7, #8, and #9 are dedicated queues for Func in the Data Platform companion version, and are reserved queues in the standalone deployment version

Name	Description
dataflux-func_server	DataFlux Func's front-end service. Mainly used to provide the Web interface, API interfaces, etc.
dataflux-func_worker-0	Python worker listening to queues #0, #4, #7, #8, and #9. Mainly handles DataFlux Func's internal tasks (#0 queue) and other tasks that accidentally enter reserved queues
dataflux-func_worker-1	Python worker listening to queue #1. Mainly handles synchronously executed Func API
dataflux-func_worker-2	Python worker listening to queue #2. Mainly handles Cron Jobs
dataflux-func_worker-3	Python worker listening to queue #3. Mainly handles asynchronously executed Func API
dataflux-func_worker-5	Python worker listening to queue #5. Mainly handles debugging tasks for running Scripts in the Web interface
dataflux-func_worker-6	Python worker listening to queue #6. Mainly handles tasks executed by Connector subscription messages
dataflux-func_beat	The trigger for Cron Jobs; only one can be enabled globally
dataflux-func_mysql	DataFlux Func's built-in MySQL
dataflux-func_redis	DataFlux Func's built-in Redis

Name	Description
dataflux-func_server	DataFlux Func's front-end service. Mainly used to provide the Web interface, API interfaces, etc.
dataflux-func_worker-0	Python worker listening to queue #0. Mainly handles internal tasks of DataFlux Func
dataflux-func_worker-1-6	Python worker listening to queues #1, #2, #3, #4, #5, #6. Mainly handles synchronously executed Func API
dataflux-func_worker-7	Python worker listening to queue #7. Mainly handles debugging tasks for running Scripts in the Web interface
dataflux-func_worker-8-9	Python worker listening to queues #8, #9. Mainly handles asynchronous tasks (Cron Jobs, etc.)
dataflux-func_beat	The trigger for Cron Jobs; only one can be enabled globally
dataflux-func_mysql	DataFlux Func's built-in MySQL
dataflux-func_redis	DataFlux Func's built-in Redis

3.2 Data Storage Locations

DataFlux Func needs to store various types of data during operation. The general contents and storage locations are as follows:

Storage	Path	Contents
MySQL	{installation-directory}/mysql/	Most of the data generated from UI operations, including but not limited to: 1. Scripts, Connector configurations, and environment variables 2. User information, Func API, and Cron Jobs 3. Operation logs, Script execution logs, import/export logs, etc.
Redis	{installation-directory}/redis/	Mainly used for caching and queues, including but not limited to: 1. User login information 2. Various caches created during Script runtime 3. Script execution task queues 4. Func's own monitoring data, etc.
Directory	{installation-directory}/data/	Used mainly for data that must exist in file form; see below for details.
Directory	{installation-directory}/data/resources/	Resource folder (i.e., the root directory of the "File Manager")
Directory	{installation-directory}/data/resources/extra-python-packages/	Third-party packages installed by the PIP tool
Directory	{installation-directory}/data/resources/script-market/	Data downloaded from the Script Market
Directory	{installation-directory}/data/sqldump/	Automatic database backups
Directory	{installation-directory}/data/logs/	System logs
File	{installation-directory}/data/user-config.yaml	DataFlux Func system configuration
File	/etc/dataflux-func	DataFlux Func installation directory record Used to correctly obtain the installation location of the current DataFlux Func during upgrades

4. Using an External Database

When performing a fresh installation of DataFlux Func, if you need to use an external database (MySQL, Redis), you can specify disabling the built-in MySQL and Redis when executing the installation Script, for example:

sudo bash {installation-file-directory}/run-portable.sh --no-mysql --no-redis

After installation is complete, on the first-run screen, click "Show more configurations" and modify the MySQL and Redis configurations as needed.

For MySQL and Redis configuration requirements, please refer to Deployment and Maintenance / System Requirements

If you need to migrate the existing DataFlux Func database to an external database, please refer to Deployment and Maintenance / Daily Maintenance / Database Migration

4. Nginx Reverse Proxy

You can use Nginx to configure a reverse proxy.

Please note that DataFlux Func uses WebSocket technology, so you need to add the relevant configuration to Nginx.

The reference configuration is as follows:

HTTPHTTPS

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name your-domain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_http_version 1.1;

    location / {
        proxy_pass http://127.0.0.1:8088;
    }
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl;
    server_name your-domain.com;

    ssl_certificate     /etc/nginx/ssl/your.pem;
    ssl_certificate_key /etc/nginx/ssl/your.key;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_http_version 1.1;

    location / {
        proxy_pass http://127.0.0.1:8088;
    }
}