Skip to content

Sidecar

Sidecar is an additional component of DataFlux Func.

Since DataFlux Func usually runs inside a container, it cannot directly execute shell commands on the host machine. Sidecar is a program that runs on the host machine and acts as a proxy for DataFlux Func to execute shell commands directly on the host.

Download Command

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

0. Reading Guide

All shell commands mentioned in this Guide can be executed directly under the root user. Non-root users need to add sudo to run them.

1. System and Environment Requirements

Any host that can run DataFlux Func should be able to run Sidecar.

1.1 DataFlux Func Version Requirements

Sidecar must be used with DataFlux Func 1.3.5 or higher

2. Quick Installation

Normally, Sidecar is installed on the same host as DataFlux Func. The following operations assume that DataFlux Func is already installed on the host.

If Sidecar and DataFlux Func are running on different hosts, the corresponding configurations need to be modified.

2.1 Offline Installation

Before installing Sidecar, you need to download the required resources.

For hosts without internet access, you can copy the files to the host using a USB drive or other mobile devices.

The downloaded resource files include an automatic installation script, which can be executed to install Sidecar (details below).

2.1.1 One-Command Download of Resource Files

For Linux, macOS, and other systems, it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the required files for Sidecar. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

If you need to download a specific architecture version, you can use the following commands:

  • Intel x86_64 Processor
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=x86_64
  • ARM aarch64 Processor (i.e., ARM64v8, such as Raspberry Pi)
Bash
1
/bin/bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=aarch64

After the command is executed, all required files are saved in the newly created dataflux-func-sidecar directory in the current directory.

  • If you need to install Sidecar on a server without internet access, you can download it locally first, and then copy the entire directory to the target machine using a USB drive or scp tool.
  • If you need to install Sidecar on a server with internet access, you can download it directly on the server.

2.1.2 Manual Download of Resource Files

For systems where shell commands are inconvenient, you can manually download the required resource files.

If you need to manually download, here is the complete list of files:

# Content File Name x86_64 Architecture aarch64 Architecture
1 Sidecar Binary Program dataflux-func-sidecar.tar.gz Download Download
2 Sidecar Service Config dataflux-func-sidecar.service Download Download
3 Sidecar Installation Script run-sidecar.sh Download Download
4 Version Information version Download Download

After manually downloading all files, place them in a newly created dataflux-func-sidecar directory.

If there is an update, [re-download all files]. Do not guess which files have changed and which have not.

When manually downloading, if using a browser, be careful not to download cached old content!!

2.1.3 Using the Included Script to Perform Installation

In the already downloaded dataflux-func-sidecar directory, run the following command to automatically configure and finally start Sidecar:

Sidecar does not support Mac, please copy it to a Linux system before running the installation.

Bash
1
sudo /bin/bash run-sidecar.sh

Using the automatic installation script allows for quick installation and running within seconds. The automatic configuration includes the following:

  • Create the /usr/local/bin/dataflux-func-sidecar executable file
  • Create the /etc/dataflux-func-sidecar configuration file
  • Create the dffs user to run the Sidecar program
  • Create the /etc/systemd/system/dataflux-func-sidecar Systemd configuration file and set it to start on boot

After installation, the following installation information will be displayed:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

The output content has the following meanings:

Item Description Corresponding Configuration Item
Bind Listening address. Supports multiple, separated by commas BIND
Secret Key Secret key. Used to verify requests SECRET_KEY

By default, Bind has 2 addresses:

  • 127.0.0.1:8099: Local network
  • 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verify Installation

After Sidecar is installed by default, you can use the following command to verify the installation:

Bash
1
curl http://127.0.0.1:8099

If the following information is returned, Sidecar is running normally:

Text Only
1
2
3
Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation script supports some installation options to adapt to different installation needs.

During installation, you only need to add --{parameter}[ parameter configuration (if any)] after the automatic installation command to specify the installation options.

For example, specify the listening address:

Bash
1
sudo /bin/bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

Specific parameter details are as follows.

--bind: Specify Listening Address

Sidecar listens to 127.0.0.1:8099,172.17.0.1:8099 by default. If the port is occupied, you can choose other listening addresses.

--secret-key: Specify Secret Key

DataFlux Func Sidecar automatically generates a random secret key during default installation. You can use this parameter to manually specify the secret key.

3. Calling Sidecar in DataFlux Func

DataFlux Func provides a Connector for DataFlux Func Sidecar, which can be used to operate Sidecar.

3.1. Create Sidecar Connector

Go to 'Connector / Add Connector / DataFlux Func Sidecar (HTTP)' , fill in the Secret Key correctly to create it.

If you specified a different listening address when installing Sidecar (modified the Bind parameter), you also need to make corresponding modifications when creating the Connector.

add-datasource.png

3.2. Write Code

The following is an example code:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/sync/sidecar-callback')
    # res content is:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # kwargs content is:
    # {'stderr': '', 'stdout': 'my_host\n'}

3.3. Configure Function

In the above code example:

'Test Sidecar' is the main execution function, which can be configured as a 'Func API' or 'Cron Job' to execute.

'Test Sidecar callback' is the function that receives the callback after execution, which needs to be configured as a 'Func API' .

In the example code, the Func API address ID part in the callback_url parameter is sidecar-callback, so when configuring the 'Func API' for the callback function, you also need to specify the same ID to ensure the URL address is consistent.

If Sidecar and DataFlux Func are installed on the same host, they can access each other through docker0, i.e., the IP address is 172.17.0.1.

For details about the 'Sidecar Connector Operation Object' , see below.

4. Sidecar Connector Operation Object API

Using the Sidecar Connector Operation Object allows users to call Sidecar to execute shell commands.

DFF.CONN(...) parameters are as follows:

Parameter Type Required / Default Description
data_source_id str Required Connector ID

SidecarHelper.shell(...)

Execute a call to Sidecar to execute a shell command. Parameters are as follows:

Parameter Type Required / Default Description
cmd str Required The shell command to execute
e.g., "ls -l"
wait bool True Whether to wait for execution to complete
If set to False, this function will return immediately and will not return terminal output
workdir str None The working directory for the shell command execution
e.g., "/home/dev"
envs dict None Environment variables, both keys and values are strings
e.g., {"MY_NAME": "Tom"}
callback_url str None Callback URL, after the command is executed, stdout and stderr will be sent to the specified URL using POST
Generally used with the wait=False parameter to achieve asynchronous callback
timeout int 3 Request timeout
Note: This parameter is not the timeout for the shell command, but the timeout for Func to request Sidecar
i.e., Func's request to Sidecar may timeout, but the executed shell command will not stop because of this

Post-Execution Callback

After calling SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar will send the standard output stdout and standard error stderr to this URL using POST after executing the shell command.

The specific structure is as follows:

Text Only
1
2
3
4
5
6
7
8
9
POST {callback_url}
Content-Type: application/json

{
    "kwargs": {
        "stdout": "<Standard Output Text>",
        "stderr": "<Standard Error Text>"
    }
}

This structure matches the standard POST method of DataFlux Func's 'Func API' , and can directly use 'Func API' to receive the callback after execution.

5. Daily Maintenance

By default, the executable program is installed at /usr/local/bin/dataflux-func-sidecar.

5.1 Upgrade System

Repeat the installation process. The automatic installation script will automatically replace the executable program and restart the service.

At the same time, the previous configuration file content will be retained.

5.2 Start/Stop/Restart Service

Sidecar service is managed by systemd, directly use systemctl to operate:

Bash
1
2
3
sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 View Configuration

The configuration file is located at /etc/dataflux-func-sidecar.

6. Security Assurance

Since the usage mode of Sidecar is essentially sending arbitrary executable code to the host machine, it is dangerous.

Therefore, Sidecar has the following restrictions in implementation and deployment:

  1. Sidecar service runs as the dffs user (i.e., the abbreviation of DataFlux Func Sidecar)
  2. Sidecar must be configured with a SecretKey to normally execute shell commands
  3. SidecarHelper.shell(...) internally implements HmacSha1 signature to prevent tampering and replay attacks

Since Sidecar service runs as the dffs user, it cannot execute commands that require root privileges or operate files of other users. If needed, you can add the dffs user to a user group or modify file-related permissions.