Skip to content

Sidecar

Sidecar is an additional component of DataFlux Func.

Since DataFlux Func usually runs inside a container, it cannot directly execute shell commands on the host machine. Sidecar is a program that runs on the host machine, acting as a proxy for DataFlux Func to execute shell commands directly on the host.

Download Command

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

0. Before You Read

All shell commands mentioned in this Guide can be run directly under the root user. For non-root users, add sudo before running.

1. System and Environment Requirements

Any host that can run DataFlux Func should be able to run Sidecar.

1.1 DataFlux Func Version Requirements

Sidecar must be used with DataFlux Func version 1.3.5 or higher

2. Quick Installation

Normally, Sidecar is installed on the same host as DataFlux Func. The following operations assume that DataFlux Func is already installed on the host.

If Sidecar and DataFlux Func are running on different hosts, the corresponding configuration needs to be modified.

2.1 Offline Installation

Before installing Sidecar, the required resources need to be downloaded.

For hosts without internet access, the resources can be copied to the host via a USB drive or other portable devices.

The downloaded resource files come with an automatic installation script, which can be executed to install Sidecar (details below).

2.1.1 One-Command Download Resource Files

For Linux, macOS, and other systems, it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the required files for Sidecar. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

If you need to download a specific architecture version, use the following commands:

  • Intel x86_64 Processor
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=x86_64
  • ARM aarch64 Processor (i.e., ARM64v8, such as Raspberry Pi)
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=aarch64

After the command is executed, all required files are saved in the newly created dataflux-func-sidecar directory under the current directory.

  • If you need to install Sidecar on a server without internet access, you can download the files locally first, then copy the entire directory to the target machine via a USB drive or scp tool.
  • If you need to install Sidecar on a server with internet access, you can download the files directly on the server.

2.1.2 Manual Download Resource Files

For systems where using shell commands is inconvenient, you can manually download the required resource files.

If you need to manually download, here is the list of all files:

# Content File Name x86_64 Architecture aarch64 Architecture
1 Sidecar Binary dataflux-func-sidecar.tar.gz Download Download
2 Sidecar Service Config dataflux-func-sidecar.service Download Download
3 Sidecar Install Script run-sidecar.sh Download Download
4 Version Info version Download Download

After manually downloading all files, place them in the newly created dataflux-func-sidecar directory.

If there is an update, [re-download all files]. Do not guess which files have changed and which have not.

When manually downloading, if using a browser or other tools, be careful not to download cached old content!!

2.1.3 Use the Included Script to Install

In the downloaded dataflux-func-sidecar directory, run the following command to automatically configure and start Sidecar:

Sidecar does not support Mac. Please copy it to a Linux system before running the installation.

Bash
1
sudo /bin/bash run-sidecar.sh

The automatic installation script allows for quick installation and running within seconds. The automatic configuration includes:

  • Creating the /usr/local/bin/dataflux-func-sidecar executable
  • Creating the /etc/dataflux-func-sidecar configuration file
  • Creating the dffs user to run the Sidecar program
  • Creating the /etc/systemd/system/dataflux-func-sidecar Systemd configuration file and setting it to start on boot

After installation, the following installation information will be displayed:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

The output information is as follows:

Item Description Corresponding Configuration Item
Bind Listening addresses. Supports multiple, separated by commas BIND
Secret Key Secret key. Used to verify requests SECRET_KEY

By default, there are 2 addresses in Bind:

  • 127.0.0.1:8099: Local network
  • 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verify Installation

After the default installation of Sidecar, you can use the following command to verify the installation:

Bash
1
curl http://127.0.0.1:8099

If the following information is returned, it means Sidecar is running normally:

Text Only
1
2
3
Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation script supports some installation options to adapt to different installation needs.

During installation, you can specify installation options by adding --{parameter}[ parameter configuration (if any)] after the automatic installation command.

For example, specify the listening address:

Bash
1
sudo /bin/bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

For specific parameter details, see below.

--bind: Specify Listening Address

Sidecar listens to 127.0.0.1:8099,172.17.0.1:8099 by default. If the port is occupied, you can choose another listening address.

--secret-key: Specify Secret Key

DataFlux Func Sidecar generates a random secret key by default during installation. You can manually specify the secret key using this parameter.

3. Calling Sidecar in DataFlux Func

DataFlux Func provides a Sidecar Connector, which can be used to operate Sidecar.

3.1. Create Sidecar Connector

Go to 'Connector / Add Connector / DataFlux Func Sidecar (HTTP)' , fill in the Secret Key correctly to create it.

If a different listening address is specified during Sidecar installation (modified the Bind parameter), the corresponding modification is also needed when creating the Connector.

add-datasource.png

3.2. Write Code

The following is an example code:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/sync/sidecar-callback')
    # res content is:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # kwargs content is:
    # {'stderr': '', 'stdout': 'my_host\n'}

3.3. Configure Function

In the example code above:

'Test Sidecar' is the main execution function, which can be configured as 'Sync API (Legacy: Auth Link)' or 'Cron Job (Legacy: Auto Trigger Configuration)' .

'Test Sidecar callback' is the function that receives the post-execution callback, which needs to be configured as 'Sync API (Legacy: Auth Link)' .

In the example code, the Sync API (Legacy: Auth Link) address ID part in the callback_url parameter is sidecar-callback, so when configuring the 'Sync API (Legacy: Auth Link)' for the callback function, the same ID must be specified to ensure the URL address is consistent.

If Sidecar and DataFlux Func are installed on the same host, they can access each other via docker0, i.e., the IP address is 172.17.0.1.

For details about the 'Sidecar Connector Operation Object' , see below.

4. Sidecar Connector Operation Object API

Using the Sidecar Connector Operation Object allows users to call Sidecar to execute shell commands.

DFF.CONN(...) parameters are as follows:

Parameter Type Required / Default Description
data_source_id str Required Connector ID

SidecarHelper.shell(...)

Execute a shell command by calling Sidecar. Parameters are as follows:

Parameter Type Required / Default Description
cmd str Required The shell command to execute
Example: "ls -l"
wait bool True Whether to wait for execution to complete
If set to False, this function will return immediately and will not return terminal output
workdir str None The working directory for the shell command
Example: "/home/dev"
envs dict None Environment variables, both keys and values are strings
Example: {"MY_NAME": "Tom"}
callback_url str None Callback URL, after the command is executed, stdout and stderr will be sent to the specified URL via POST
Usually used with the wait=False parameter for async callback
timeout int 3 Request timeout
Note: This parameter is not the timeout for the shell command, but the timeout for Func to request Sidecar
Func's request to Sidecar may timeout, but the executed shell command will not stop

Post-Execution Callback

When calling SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar will send the standard output stdout and standard error stderr to this URL via POST after executing the shell command.

The specific structure is as follows:

Text Only
1
2
3
4
5
6
7
8
9
POST {callback_url}
Content-Type: application/json

{
    "kwargs": {
        "stdout": "<Standard Output Text>",
        "stderr": "<Standard Error Text>"
    }
}

This structure matches DataFlux Func's 'Sync API (Legacy: Auth Link) Standard POST Method' and can be directly used with 'Sync API (Legacy: Auth Link)' to receive post-execution callbacks.

5. Daily Maintenance

By default, the executable is installed at /usr/local/bin/dataflux-func-sidecar.

5.1 Upgrade System

Repeat the installation process. The automatic installation script will automatically replace the executable and restart the service.

At the same time, previous configuration file contents will be retained.

5.2 Start/Stop/Restart Service

Sidecar service is managed by systemd. Use systemctl directly to operate:

Bash
1
2
3
sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 View Configuration

The configuration file is located at /etc/dataflux-func-sidecar.

6. Security Assurance

Since the usage pattern of Sidecar essentially sends arbitrary executable code to the host machine, it is inherently dangerous.

Therefore, Sidecar has the following restrictions in implementation and deployment:

  1. Sidecar service runs as the dffs user (i.e., DataFlux Func Sidecar initials)
  2. Sidecar must be configured with a SecretKey to execute shell commands normally
  3. SidecarHelper.shell(...) internally implements HmacSha1 signature to prevent tampering and replay attacks

Since the Sidecar service runs as the dffs user, it cannot execute commands that require root privileges or operate on files of other users. If needed, you can add the dffs user to a group or modify file permissions.