Skip to content

Sidecar

Sidecar is an additional component of DataFlux Func.

Since DataFlux Func typically runs inside a container, it cannot directly execute shell commands on the host machine. Sidecar is a program that runs on the host machine, acting as a proxy for DataFlux Func to execute shell commands directly on the host.

Download Command

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

0. Before You Read

This feature requires root privileges

All shell commands mentioned in this document can be run directly under the root user. For non-root users, you need to add sudo to run them.

1. System and Environment Requirements

Hosts capable of running DataFlux Func can generally run Sidecar.

1.1 DataFlux Func Version Requirements

Sidecar must be used with DataFlux Func version 1.3.5 or higher.

2. Quick Installation

Normally, Sidecar is installed on the same host as DataFlux Func. The following operations assume they are performed on a host where DataFlux Func is already installed.

If Sidecar and DataFlux Func run on different hosts, corresponding configurations need to be modified.

2.1 Offline Installation

Before installing Sidecar, you need to download the required resources.

For hosts without internet access, you can copy the files to the host via USB drives or other removable media.

The downloaded resource files include an automatic installation script. Execute it to proceed with the installation (details below).

2.1.1 One-Command Download for Resource Files

For systems like Linux, macOS, etc., it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the required files for Sidecar. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

If you need to download a version for a specific architecture, you can use the following commands:

  • For Intel x86_64 processors
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=x86_64
  • For ARM aarch64 processors (i.e., ARM64v8, such as Raspberry Pi)
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=aarch64

After the command finishes, all required files are saved in a newly created dataflux-func-sidecar directory under the current directory.

  • To install Sidecar on a server without internet access, you can download it locally first, then copy the entire directory to the target machine via USB drives, scp, or other methods.
  • To install Sidecar on a server with internet access, download it directly on the server.

2.1.2 Manual Download of Resource Files

For systems where using shell commands is inconvenient, you can manually download the required resource files.

If manual download is needed, here is the complete file list:

# Content Filename x86_64 Architecture aarch64 Architecture
1 Sidecar Binary Program dataflux-func-sidecar.tar.gz Download Download
2 Sidecar Service Config dataflux-func-sidecar.service Download Download
3 Sidecar Install Script run-sidecar.sh Download Download
4 Version Information version Download Download

After manually downloading all files, place them in a newly created dataflux-func-sidecar directory.

If there is an update, you should re-download all files. Do not guess which files have changed and which have not.

When downloading manually, if using a browser or similar, be careful not to download cached old content!!

2.1.3 Using the Included Script for Installation

In the already downloaded dataflux-func-sidecar directory, run the following command to automatically configure and finally start Sidecar:

Sidecar does not support Mac. Please copy it to a Linux system before running the installation.

Bash
1
sudo /bin/bash run-sidecar.sh

Using the automatic installation script enables quick installation and startup within seconds. The automatic configuration includes:

  • Creating the executable file /usr/local/bin/dataflux-func-sidecar
  • Creating the configuration file /etc/dataflux-func-sidecar
  • Creating the dffs user to run the Sidecar program
  • Creating the Systemd configuration file /etc/systemd/system/dataflux-func-sidecar and setting it to start on boot

After installation, the following installation information will be displayed:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

The meaning of the output is as follows:

Item Description Corresponding Config Item
Bind Listening addresses. Supports multiple, separated by commas. BIND
Secret Key Secret key. Used for request verification. SECRET_KEY

By default, after installation, Bind will have 2 addresses:

  • 127.0.0.1:8099: Local network
  • 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verifying Installation

After Sidecar is installed by default, you can use the following command to verify the installation:

Bash
1
curl http://127.0.0.1:8099

If the following information is returned, it indicates Sidecar is running normally:

Text Only
1
2
3
Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation script supports some installation options to adapt to different installation needs.

During installation, simply add --{parameter}[ parameter configuration (if any)] after the automatic installation command to specify installation options.

For example, to specify the listening address:

Bash
1
sudo /bin/bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

For specific parameter details, see below.

--bind: Specify Listening Address

Sidecar listens on 127.0.0.1:8099,172.17.0.1:8099 by default. If the port is occupied, you can choose other listening addresses.

--secret-key: Specify Secret Key

DataFlux Func Sidecar automatically generates a random secret key during default installation. You can use this parameter to manually specify a secret key.

3. Calling Sidecar in DataFlux Func

DataFlux Func provides a DataFlux Func Sidecar connector for operating Sidecar.

3.1. Creating a Sidecar Connector

Go to "Connectors / Add Connector / DataFlux Func Sidecar (HTTP)", fill in the Secret Key correctly to create it.

If a different listening address was specified during Sidecar installation (the Bind parameter was modified), then the connector creation also needs to be modified accordingly.

add-datasource.png

3.2. Writing Code

The following is example code:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/sync/sidecar-callback')
    # res content is:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # kwargs content is:
    # {'stdout': 'my_host\n', 'stderr': ''}
    print(kwargs['stdout'])
    print(kwargs['stderr'])

3.3. Configuring Functions

In the example code above:

"Test Sidecar" is the main execution function and can be configured to execute as a "Function API" or "Scheduled Task".

"Test Sidecar callback" is the function that receives the post-execution callback and needs to be configured as a "Function API".

In the example code, the function API address ID part in the callback_url parameter is sidecar-callback. Therefore, when configuring the "Function API" for the callback function, you must specify the same ID to ensure the URL addresses match.

For cases where Sidecar and DataFlux Func are installed on the same host, they can access each other via docker0, i.e., the IP address is 172.17.0.1.

For details about the "Sidecar Connector Operation Object", see below.

4. Sidecar Connector Operation Object API

Using the Sidecar connector operation object allows users to call Sidecar to execute shell commands.

DFF.CONN(...) parameters are as follows:

Parameter Type Required / Default Description
data_source_id str Required Connector ID

SidecarHelper.shell(...)

Calls Sidecar to execute a shell command. Parameters are as follows:

Parameter Type Required / Default Description
cmd str Required The shell command to execute.
Example: "ls -l"
wait bool True Whether to wait for execution to complete.
When set to False, this function returns immediately and will not return terminal output.
workdir str None The working directory for shell command execution.
Example: "/home/dev"
envs dict None Environment variables, both keys and values are strings.
Example: {"MY_NAME": "Tom"}
callback_url str None Callback URL. After command execution, stdout and stderr are sent to the specified URL via POST.
Typically used with the wait=False parameter to achieve asynchronous callback.
timeout int 3 Request timeout (in seconds).
Note: This parameter is not the timeout for the shell command, but the timeout for Func's request to Sidecar.
Meaning Func's request to Sidecar may timeout, but the executed shell command will not stop.

Post-Execution Callback

After calling SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar will send the standard output stdout and standard error stderr to this address via POST after executing the shell command.

The specific structure is as follows:

Text Only
1
2
3
4
5
6
7
8
9
POST {callback_url}
Content-Type: application/json

{
  "kwargs": {
    "stdout": "<standard output text>",
    "stderr": "<standard error text>"
  }
}

This structure matches DataFlux Func's "Function API" standard POST method, and can directly use a "Function API" to receive the post-execution callback.

5. Daily Maintenance

By default, the executable program is installed at /usr/local/bin/dataflux-func-sidecar.

5.1 Upgrading the System

Repeat the installation process. The automatic installation script will automatically replace the executable program and restart the service.

Also, previous configuration file contents will be preserved.

5.2 Starting/Stopping/Restarting the Service

The Sidecar service is managed by systemd. Use systemctl directly to operate it:

Bash
1
2
3
sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 Viewing Configuration

The configuration file is located at /etc/dataflux-func-sidecar.

6. Security Guarantees

Since Sidecar's usage pattern essentially involves sending arbitrary executable code to the host machine, it is inherently risky.

Therefore, Sidecar has the following restrictions in its implementation and deployment:

  1. The Sidecar service runs under the dffs user (abbreviation for DataFlux Func Sidecar).
  2. Sidecar must be configured with a SecretKey to normally execute shell commands.
  3. SidecarHelper.shell(...) internally implements HmacSha1 signature to prevent tampering and replay attacks.

Since the Sidecar service runs under the dffs user, it cannot execute commands requiring root privileges or operate on files belonging to other users. If needed, you can add the dffs user to user groups or modify relevant file permissions.