Skip to content

Deployment and Maintenance / Standalone Deployment

This article mainly introduces how to directly install and deploy the standalone version of DataFlux Func on a server.

For installing DataFlux Func using Helm in k8s, please refer to Deployment and Maintenance / Installation and Deployment / Helm Deployment

1. Download the Installation Package

DataFlux Func supports downloading the required installation files and then installing them in an environment without public network access using a USB drive or other mobile devices.

The downloaded "installation package" comes with an automatic installation script, Docker, etc., and can be executed to install (details below)

1.1 One-Command Download

For Linux, macOS, and other systems, it is recommended to use the official provided shell command to download the installation package.

Run the following command to automatically download the required installation files for DataFlux Func. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Please confirm the system requirements and server configuration before installation

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)"

If the architecture of the download is different from the server architecture where it will be installed, you can specify the architecture with the architecture option:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=x86_64
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --arch=aarch64

After the command is executed, all installation files are saved in the automatically created dataflux-func-portable-{architecture}-{version} directory.

  • If you need to install DataFlux Func on a server without public network access, you can download it on your local machine first, then copy the entire directory to the target machine using a USB drive or other mobile storage devices, or using the scp tool.
  • If you need to install DataFlux Func on a server with public network access, you can download it directly on the server.

1.2 Manual Download

For systems where it is inconvenient to use shell commands, you can manually download the required installation files.

If you need to manually download, here is the complete file list:

# Content File Name x86_64 Architecture (Original) aarch64 Architecture (Original)
1 Docker Binary docker-24.0.9.tgz Download Download
2 DataFlux Func Image dataflux-func.tar.gz Download Download
3 MySQL/MariaDB Image mysql-5.7.26.tar.gz Download Download
4 Redis Image redis-6.2.20.tar.gz Download Download
5 Docker Service Config File docker.service Download Download
6 DataFluxFunc Installation Script run-portable.sh Download Download
7 Docker Stack Config Template docker-stack.example.yaml Download Download
8 Image List image-list Download Download
9 Version Info version Download Download

After manually downloading all installation files, place them in the same directory.

1.3 Download Options

When executing the download script, you can specify download options to meet personalized needs.

Example as follows:

Note: There is a -- (two hyphens) before the specified download options

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

Supported download options are as follows:

--arch={architecture} Specify Architecture

The download script defaults to downloading the installation package files of the same architecture as the local machine. If you need to download installation files of a different architecture, you can specify this parameter.

Available architectures are as follows:

Architecture Description Parameter Example
Intel Intel / AMD 64-bit Processors --arch=x86_64
ARM ARM64v8 Processors --arch=aarch64

--download-dir={download directory} Specify Download Directory

Added in version 2.6.1

This parameter is suitable for automated script deployment of DataFlux Func

The download script will create/clear the dataflux-func-portable-{architecture}-{version} directory in the current directory by default and download the installation files to it.

If you need to download to a specified directory, you can specify this parameter, for example:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/download)" -- --download-dir=func-download

2. Execute Installation

Enter the installation package directory downloaded in the previous section and run the one-click installation script run-portable.sh:

Please confirm the system requirements and server configuration before installation

DataFlux Func does not support macOS, Windows, please copy it to a Linux system before running the installation

Bash
1
sudo /bin/bash {installation file directory}/run-portable.sh

Using the automatic installation script, you can quickly install and run it within minutes. The automatic configuration includes:

  • Running MySQL, Redis, DataFlux Func (including Server, Worker, Beat)
  • Automatically creating and saving all data in the /usr/local/dataflux-func/ directory (including MySQL data, Redis data, DataFlux Func configuration, log files, etc.)
  • Randomly generating MySQL root user password, system Secret, and saving them in the DataFlux Func configuration file
  • Redis has no password
  • MySQL, Redis do not provide external access

After execution, you can use a browser to access http://{server IP address/domain}:8088 for the initialization operation interface.

If the running environment has poor performance, you should use the docker ps command to confirm that all components have started successfully before accessing (see the list below)

  1. dataflux-func_server
  2. dataflux-func_worker-0
  3. dataflux-func_worker-1
  4. dataflux-func_worker-2
  5. dataflux-func_worker-3
  6. dataflux-func_worker-5
  7. dataflux-func_worker-6
  8. dataflux-func_beat
  9. dataflux-func_mysql
  10. dataflux-func_redis
  1. dataflux-func_server
  2. dataflux-func_worker-0
  3. dataflux-func_worker-1-6
  4. dataflux-func_worker-7
  5. dataflux-func_worker-8-9
  6. dataflux-func_beat
  7. dataflux-func_mysql
  8. dataflux-func_redis

2.1 Installation Options

When executing the installation script, you can specify installation options to meet personalized needs, for example:

Bash
1
sudo /bin/bash run-portable.sh --port=80

--mini Install Mini Version

For low-configuration environments where resources need to be saved.

After enabling:

  • Only a single Worker is started to listen to all queues
  • Heavy load tasks are more likely to cause queue blocking and stuttering
  • System tasks and function tasks share the processing queue and will affect each other
  • System requirements are reduced to:
    • CPU cores >= 1
    • Memory capacity >= 2GB
  • If the built-in MySQL, Redis are not used, system requirements can be further reduced

--port={port number} Specify Listening Port Number

DataFlux Func defaults to using port 8088 for access. If this port is occupied by other programs, you can choose another port, for example: 9000.

--install-dir={installation directory} Specify Installation Directory

When you need to install to a different path than the default path /usr/local/dataflux-func, you can specify this parameter

--no-mysql Disable Built-in MySQL

When you need to use an existing MySQL database, you can specify this parameter to prevent MySQL from being started on the local machine.

After enabling this option, you need to specify the correct MySQL connection information on the configuration page after installation is complete

--no-redis Disable Built-in Redis

When you need to use an existing Redis database, you can specify this parameter to prevent Redis from being started on the local machine.

After enabling this option, you need to specify the correct Redis connection information on the configuration page after installation is complete

--auto-setup Automatically Execute Configuration

Added in version 2.6.0

This parameter is suitable for automated script deployment of DataFlux Func

After enabling, it will automatically configure and initialize the database, and there will be no configuration page, for example:

Bash
1
sudo /bin/bash run-portable.sh --auto-setup

In addition, after enabling the --auto-setup option, you can also add other --auto-setup-* options to adjust the automatic configuration. The additional options for automatic configuration are as follows:

Additional Auto-Setup Options Default Value Description
--auto-setup-admin-username={username} admin Specify the administrator username
--auto-setup-admin-password={password} admin Specify the administrator password
--auto-setup-ak-secret={AK Secret} Automatically generated Automatically create AK
and use the specified value as the AK Secret
--auto-setup-ak-id={AK ID} ak-auto-setup AK ID when automatically creating AK
(requires --auto-setup-ak-secret option)

If you need to automatically configure and specify the administrator password as AdminPass, the complete command is as follows:

Bash
1
sudo /bin/bash run-portable.sh --auto-setup --auto-setup-admin-password='AdminPass'

3. Verify Installation

By default, DataFlux Func comes with some example Scripts after installation.

Execute the following operations in sequence to verify the installation:

  1. Click the "Script Editor" in the top navigation bar, select "Script Lib" - "Examples" - "Basic Demo" in the left sidebar
  2. At the top of the right script editor, click "Edit" to enter edit mode, select the "hello_world" function and click the "Execute" button to execute the function
  3. At this time, if you can see the function return value normally in the "Script Output" at the bottom

At this point, the installation verification is complete

3.1 Service Descriptions

By default, after DataFlux Func starts normally, the following services are running:

Queues #7, #8, #9 are reserved for the data platform's affiliated version of Func, and are reserved queues in the standalone version of Func

Name Description
dataflux-func_server The front-end service of DataFlux Func.
Mainly provides Web interface, API interfaces, etc.
dataflux-func_worker-0 Python worker unit listening to queues #0, #4, #7, #8, #9.
Mainly handles internal tasks of DataFlux Func (#0 queue) and tasks mistakenly entered into reserved queues
dataflux-func_worker-1 Python worker unit listening to queue #1.
Mainly handles synchronously executed Func APIs
dataflux-func_worker-2 Python worker unit listening to queue #2.
Mainly handles Cron Jobs
dataflux-func_worker-3 Python worker unit listening to queue #3.
Mainly handles asynchronously executed Func APIs
dataflux-func_worker-5 Python worker unit listening to queue #5.
Mainly handles debugging tasks of running Scripts in the Web interface
dataflux-func_worker-6 Python worker unit listening to queue #6.
Mainly handles tasks executed by Connector subscription messages
dataflux-func_beat Trigger for Cron Jobs, only one can be enabled globally
dataflux-func_mysql Built-in MySQL of DataFlux Func
dataflux-func_redis Built-in Redis of DataFlux Func
Name Description
dataflux-func_server The front-end service of DataFlux Func.
Mainly provides Web interface, API interfaces, etc.
dataflux-func_worker-0 Python worker unit listening to queue #0.
Mainly handles internal tasks of DataFlux Func
dataflux-func_worker-1-6 Python worker unit listening to queues #1, #2, #3, #4, #5, #6.
Mainly handles synchronously executed Func APIs
dataflux-func_worker-7 Python worker unit listening to queue #7.
Mainly handles debugging tasks of running Scripts in the Web interface
dataflux-func_worker-8-9 Python worker unit listening to queues #8, #9.
Mainly handles asynchronous tasks (Cron Jobs, etc.)
dataflux-func_beat Trigger for Cron Jobs, only one can be enabled globally
dataflux-func_mysql Built-in MySQL of DataFlux Func
dataflux-func_redis Built-in Redis of DataFlux Func

3.2 Data Storage Locations

DataFlux Func needs to store various data during operation. The main content and storage locations are as follows:

Storage Storage Path Storage Content
MySQL {installation directory}/mysql/ Most of the data generated during UI operations, including but not limited to:
1. Scripts, Connector configurations, environment variables
2. User information, Func APIs, Cron Jobs
3. Operation records, Script execution records, import and export records, etc.
Redis {installation directory}/redis/ Mainly used for caching and queues, including but not limited to:
1. User login information
2. Various caches established during Script execution
3. Script execution task queues
4. Func self-monitoring data, etc.
Directory {installation directory}/data/ Mainly used for data that must exist in file form, see below for details
Directory {installation directory}/data/resources/ Resource folder (i.e., the root directory of "File Service")
Directory {installation directory}/data/resources/extra-python-packages/ Third-party packages installed by PIP
Directory {installation directory}/data/resources/script-market/ Script Market download data
Directory {installation directory}/data/sqldump/ Database automatic backup
Directory {installation directory}/data/logs/ System logs
File {installation directory}/data/user-config.yaml DataFlux Func system configuration
File /etc/dataflux-func DataFlux Func installation directory record
Used to correctly obtain the current installation location of DataFlux Func during upgrades

4. Using External Databases

When installing DataFlux Func for the first time, if you need to use an external database (MySQL, Redis), you can specify to disable the built-in MySQL, Redis when executing the installation script, for example:

Bash
1
sudo /bin/bash {installation file directory}/run-portable.sh --no-mysql --no-redis

After installation, on the first run interface, click "Show More Configuration" and modify the MySQL, Redis configuration.

For MySQL, Redis configuration requirements, please refer to Deployment and Maintenance / System Requirements

If you need to migrate the existing DataFlux Func database to an external one, please refer to Deployment and Maintenance / Daily Maintenance / Migrate Database