Skip to content

Sidecar

Sidecar is an add-on component of DataFlux Func.

Since DataFlux Func typically runs within a container, it cannot directly execute Shell commands on the host machine. Sidecar is a program that runs on the host machine, acting as a proxy for DataFlux Func to execute Shell commands directly on the host.

Download Command

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

0. Before You Read

This feature requires root privileges

All shell commands mentioned in this document can be run directly under the root user. For non-root users, you need to add sudo to run them.

1. System and Environment Requirements

Any host capable of running DataFlux Func can normally run Sidecar.

1.1 DataFlux Func Version Requirements

Sidecar must be used with DataFlux Func version 1.3.5 or higher.

2. Quick Installation

Normally, Sidecar is installed on the same host as DataFlux Func. The following operations assume they are performed on a host where DataFlux Func is already installed.

If Sidecar and DataFlux Func run on different hosts, the corresponding configurations need to be modified

2.1 Offline Installation

Before installing Sidecar, you need to download the required resources.

For hosts without internet access, you can copy the files to the host via USB drives or other removable media.

The downloaded resource files include an automatic installation script. Execute it to proceed with the installation (details below).

2.1.1 One-Click Command to Download Resource Files

For systems like Linux, macOS, etc., it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the necessary files for Sidecar. The download script will automatically choose the x86_64 or aarch64 architecture version based on the current environment:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

If you need to download a version for a specific architecture, you can use the following commands:

  • For Intel x86_64 processors
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=x86_64
  • For ARM aarch64 processors (i.e., ARM64v8, e.g., Raspberry Pi)
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=aarch64

After the command finishes executing, all required files are saved in a newly created dataflux-func-sidecar directory under the current directory.

  • To install Sidecar on a server without internet access, you can download the files locally first, then copy the entire directory to the target machine via USB drives, scp tool, etc.
  • To install Sidecar on a server with internet access, you can download directly on the server.

2.1.2 Manual Download of Resource Files

For systems where using shell commands is inconvenient, you can manually download the required resource files.

If manual download is needed, here is the complete file list:

# Content Filename x86_64 Architecture aarch64 Architecture
1 Sidecar binary dataflux-func-sidecar.tar.gz Download Download
2 Sidecar service config dataflux-func-sidecar.service Download Download
3 Sidecar installation script run-sidecar.sh Download Download
4 Version info version Download Download

After manually downloading all files, place them into a newly created dataflux-func-sidecar directory.

If there is an update, 【re-download all files】. Do not guess which files have changed and which have not.

When manually downloading, if using a browser, be careful not to download cached old content!!

2.1.3 Using the Included Script to Perform Installation

In the already downloaded dataflux-func-sidecar directory, run the following command to automatically configure and finally start the entire Sidecar:

Sidecar does not support Mac. Please copy the files to a Linux system before running the installation.

Bash
1
sudo /bin/bash run-sidecar.sh

Using the automatic installation script enables quick installation and startup within seconds. The automatic configuration includes:

  • Creating the /usr/local/bin/dataflux-func-sidecar executable file
  • Creating the /etc/dataflux-func-sidecar configuration file
  • Creating the dffs user to run the Sidecar program
  • Creating the /etc/systemd/system/dataflux-func-sidecar Systemd configuration file and setting it to start on boot

After installation, the following installation information will be displayed:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

The meaning of the output is as follows:

Item Description Corresponding Configuration Item
Bind Listening addresses. Supports multiple, separated by commas. BIND
Secret Key Secret key. Used to verify requests. SECRET_KEY

By default, after installation, Bind will have 2 addresses:

  • 127.0.0.1:8099: Local network
  • 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verifying Installation

After Sidecar is installed by default, you can use the following command to verify the installation:

Bash
1
curl http://127.0.0.1:8099

If the following information is returned, it indicates Sidecar is running normally:

Text Only
1
2
3
Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation script supports some installation options to adapt to different installation needs.

During installation, simply add --{parameter}[ parameter configuration (if any)] after the automatic installation command to specify the installation option.

For example, to specify the listening address:

Bash
1
sudo /bin/bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

For specific parameter details, see below.

--bind: Specify Listening Address

Sidecar listens on 127.0.0.1:8099,172.17.0.1:8099 by default. If the port is occupied, you can choose other listening addresses.

--secret-key: Specify Secret Key

DataFlux Func Sidecar automatically generates a random secret key during default installation. You can use this parameter to manually specify a secret key.

3. Calling Sidecar from DataFlux Func

DataFlux Func provides a DataFlux Func Sidecar Connector for operating Sidecar.

3.1. Creating a Sidecar Connector

Go to "Connector / Add Connector / DataFlux Func Sidecar (HTTP)", fill in the Secret Key correctly, and create it.

If you specified a different listening address when installing Sidecar (modified the Bind parameter), you need to make corresponding changes when creating the Connector.

add-datasource.png

3.2. Writing Code

The following is example code:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/sync/sidecar-callback')
    # res content:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # kwargs content:
    # {'stderr': '', 'stdout': 'my_host\n'}

3.3. Configuring Functions

In the example code above:

"Test Sidecar" is the main execution function, which can be configured as a "Func API" or "Cron Job".

"Test Sidecar callback" is the function that receives the post-execution callback and needs to be configured as a "Func API".

In the example code, the function API address ID part in the callback_url parameter is sidecar-callback. Therefore, when configuring the "Func API" for the callback function, you must specify the same ID to ensure the URL addresses match.

When Sidecar and DataFlux Func are installed on the same host, they can access each other via docker0, i.e., the IP address is 172.17.0.1.

For details about the "Sidecar Connector operation object", see below.

4. Sidecar Connector Operation Object API

Using the Sidecar Connector operation object allows users to call Sidecar to execute Shell commands.

Parameters for DFF.CONN(...) are as follows:

Parameter Type Required / Default Description
data_source_id str Required Connector ID

SidecarHelper.shell(...)

Calls Sidecar to execute a Shell command. Parameters are as follows:

Parameter Type Required / Default Description
cmd str Required The Shell command to execute.
e.g., "ls -l"
wait bool True Whether to wait for execution to complete.
When set to False, this function returns immediately and will not return terminal output.
workdir str None The working directory for the Shell command execution.
e.g., "/home/dev"
envs dict None Environment variables, both keys and values are strings.
e.g., {"MY_NAME": "Tom"}
callback_url str None Callback URL. After the command executes, stdout and stderr are sent to the specified URL via POST.
Typically used with the wait=False parameter for asynchronous callbacks.
timeout int 3 Request timeout.
Note: This parameter is not the timeout for the Shell command, but the timeout for Func's request to Sidecar.
Meaning Func's request to Sidecar might timeout, but the executed Shell command will not stop because of it.

Post-Execution Callback

After calling SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar will send the standard output stdout and standard error stderr to this address via POST after executing the Shell command.

The specific structure is as follows:

Text Only
1
2
3
4
5
6
7
8
9
POST {callback_url}
Content-Type: application/json

{
  "kwargs": {
    "stdout": "<standard_output_text>",
    "stderr": "<standard_error_text>"
  }
}

This structure matches DataFlux Func's "Func API" standard POST method and can be directly used with a "Func API" to receive post-execution callbacks.

5. Daily Maintenance

By default, the executable program is installed at /usr/local/bin/dataflux-func-sidecar.

5.1 Upgrading the System

Repeat the installation process. The automatic installation script will automatically replace the executable program and restart the service.

At the same time, previous configuration file contents will be preserved.

5.2 Starting/Stopping/Restarting the Service

The Sidecar service is managed by systemd, so you can directly use systemctl to operate it:

Bash
1
2
3
sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 Viewing Configuration

The configuration file is located at /etc/dataflux-func-sidecar.

6. Security Guarantees

Since the usage pattern of Sidecar essentially involves sending arbitrary executable code to the host machine, it is inherently dangerous.

Therefore, Sidecar has the following restrictions in its implementation and deployment:

  1. The Sidecar service runs under the dffs user (i.e., DataFlux Func Sidecar initials).
  2. Sidecar must have a SecretKey configured to normally call and execute Shell commands.
  3. SidecarHelper.shell(...) internally implements HmacSha1 signing to prevent tampering and replay attacks.

Since the Sidecar service runs as the dffs user, it cannot execute commands requiring root privileges or operate on files belonging to other users. If needed, you can add the dffs user to user groups or modify relevant file permissions.