Skip to content

Sidecar

Sidecar is an auxiliary component of DataFlux Func.

Since DataFlux Func usually runs inside a container, it cannot directly execute Shell commands on the host machine. Sidecar, on the other hand, is a program running on the host machine, acting as a proxy for DataFlux Func to directly execute Shell commands on the host machine.

Download Command

Bash
1
bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

0. Notes Before Reading

This feature requires root privileges

All shell commands mentioned in this article can be run directly as the root user; non-root users need to add sudo to run them

1. System and Environment Requirements

Hosts that can run DataFlux Func can normally run Sidecar as well.

1.1 DataFlux Func Version Requirements

Sidecar must be used with DataFlux Func 1.3.5 and later versions

2. Quick Installation

Under normal circumstances, Sidecar is installed on the same host as DataFlux Func. The following operations assume that they are performed on a host where DataFlux Func has already been installed.

If Sidecar and DataFlux Func run on different hosts, you need to modify the corresponding configuration

2.1 Offline Installation

Before installing Sidecar, you need to download the required resources.

For hosts without public network access, you can copy the resources to the host using a USB flash drive or other removable storage devices.

The downloaded resource files include an automatic installation Script. Run it to perform the installation (see details below).

2.1.1 Download Resource Files with One Command

For Linux, macOS, and other systems, it is recommended to use the official shell command to download the installation package.

Run the following command to automatically download the files required by Sidecar. The download Script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

Bash
1
bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)"

If you need to download the version for a specific architecture, you can use the following commands to download:

  • Intel x86_64 processor
Bash
1
bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=x86_64
  • ARM aarch64 processor (i.e., ARM64v8, such as Raspberry Pi, etc.)
Bash
1
bash -c "$(curl -fsSL docs.dataflux-func.com/sidecar-download)" -- --arch=aarch64

After the command is executed, all required files are saved in the newly created dataflux-func-sidecar directory under the current directory.

  • If you need to install Sidecar on a server without public network access, you can download it on your local machine first, and then copy the entire directory to the target machine using a USB flash drive or other removable storage devices, or via tools such as scp
  • If you need to install Sidecar on a server that can access the public network, just download it directly on the server.

2.1.2 Download Resource Files Manually

For systems where shell commands are inconvenient to use, you can manually download the required resource files.

If you need to download manually, the following is a list of all the files:

# Description File Name x86_64 Architecture aarch64 Architecture
1 Sidecar binary program dataflux-func-sidecar.tar.gz Download Download
2 Sidecar service configuration file dataflux-func-sidecar.service Download Download
3 Sidecar installation Script run-sidecar.sh Download Download
4 Version information version Download Download

After manually downloading all files, place them in the newly created dataflux-func-sidecar directory.

If there is an update, you should re-download all files. Do not guess which files have changed and which have not.

When downloading manually, for example with a browser, be careful not to download cached old content!!

2.1.3 Use the Included Script to Install

In the downloaded dataflux-func-sidecar directory, run the following command to automatically configure and then start the entire Sidecar:

Sidecar does not support Mac. Please copy it to a Linux system before running the installation.

Bash
1
sudo bash run-sidecar.sh

Using the automatic installation Script, you can be up and running in seconds. The following items are automatically configured:

  • Create the executable file /usr/local/bin/dataflux-func-sidecar
  • Create the configuration file /etc/dataflux-func-sidecar
  • Create the dffs user to run the Sidecar program
  • Create the systemd configuration file /etc/systemd/system/dataflux-func-sidecar and set it to start on boot

After installation is complete, the following installation information will be displayed:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

The output items are described as follows:

Item Description Corresponding Configuration File Item
Bind Listening address. Multiple addresses are supported, separated by English commas BIND
Secret Key Secret key. Used to verify requests SECRET_KEY

In the default installation, Bind has 2 addresses:

  • 127.0.0.1:8099: local network
  • 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verify Installation

After the default installation of Sidecar is complete, you can verify the installation with the following command:

Bash
1
curl http://127.0.0.1:8099

If the following information is returned, Sidecar is running normally:

Text Only
1
2
3
Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation Script supports some installation options to accommodate different installation requirements.

During installation, simply add --{parameter}[ parameter configuration (if any)] after the automatic installation command to specify installation options.

For example, specify the listening address:

Bash
1
sudo bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

See below for parameter details.

--bind: Specify the listening address

By default, Sidecar listens on 127.0.0.1:8099,172.17.0.1:8099. If the port is occupied, you can choose another listening address.

--secret-key: Specify the secret key

During default installation, DataFlux Func Sidecar automatically generates a random secret key. You can use this parameter to manually specify the secret key.

3. Invoke Sidecar in DataFlux Func

DataFlux Func provides the Connector for DataFlux Func Sidecar, which can be used to operate Sidecar.

3.1. Create a Sidecar Connector

Go to "Connector / Add Connector / DataFlux Func Sidecar (HTTP)", fill in the Secret Key correctly, and create it.

If you specified a different listening address (modified the Bind parameter) when installing Sidecar, make corresponding changes when creating the Connector.

add-datasource.png

3.2. Write Code

The following is sample code:

Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/sync/sidecar-callback')
    # The content of res is:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # The content of kwargs is:
    # {'stdout': 'my_host\n', 'stderr': ''}
    print(kwargs['stdout'])
    print(kwargs['stderr'])

3.3. Configure Functions

In the code example above:

"Test Sidecar" is the main execution function, and can be configured for execution as a Func API or a Cron Job.

"Test Sidecar callback" is the function that receives the callback after execution and needs to be configured as a Func API.

In the sample code, the ID part of the Func API address in the callback_url parameter is sidecar-callback; therefore, when configuring the Func API for the callback function, you also need to specify the same ID to keep the URL address consistent.

If Sidecar and DataFlux Func are installed on the same host, they can access each other through docker0, i.e., the IP address is 172.17.0.1

For details about the "Sidecar Connector operation object", see below.

4. Sidecar Connector Operation Object API

The Sidecar Connector operation object allows users to invoke Sidecar to execute Shell commands.

The parameters for DFF.CONN(...) are as follows:

Parameter Type Required / Default Value Description
data_source_id str Required Connector ID

SidecarHelper.shell(...)

The parameters for invoking Sidecar to execute Shell commands are as follows:

Parameter Type Required / Default Value Description
cmd str Required The Shell command to execute
For example: "ls -l"
wait bool True Whether to wait for the execution to complete
When set to False, this function returns immediately and does not return terminal output
workdir str None The working directory for Shell command execution
For example: "/home/dev"
envs dict None Environment variables, with both keys and values as strings
For example: {"MY_NAME": "Tom"}
callback_url str None Callback URL. After the command is executed, stdout and stderr are sent to the specified URL via POST
Generally used together with the wait=False parameter for asynchronous callbacks
timeout int 3 Request timeout
Note: This parameter is not the timeout for the Shell command, but the timeout for Func requests to Sidecar
That is, the Func request to Sidecar may time out, but the Shell command being executed will not stop because of this

Post-execution Callback

When calling SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar sends the standard output stdout and standard error stderr to this address via POST after the Shell command has been executed.

The specific structure is as follows:

Text Only
1
2
3
4
5
6
7
8
9
POST {callback_url}
Content-Type: application/json

{
  "kwargs": {
    "stdout": "<standard output text>",
    "stderr": "<standard error text>"
  }
}

This structure matches the standard POST method of DataFlux Func's 'Func API', so you can directly use 'Func API' to receive the callback after execution

5. Daily Maintenance

By default, the executable is installed at /usr/local/bin/dataflux-func-sidecar

5.1 Upgrade the System

Simply repeat the installation process; the automatic installation Script will automatically replace the executable and restart the service.

At the same time, the previous configuration file contents will be retained.

5.2 Start/Stop/Restart the Service

The Sidecar service is managed by systemd; simply use systemctl to operate it:

Bash
1
2
3
sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 View Configuration

The configuration file is located at /etc/dataflux-func-sidecar.

6. Uninstall

Execute the following commands in order:

Bash
1
2
3
4
5
6
sudo systemctl disable --now dataflux-func-sidecar.service   # Stop the Sidecar service and disable automatic startup on boot
sudo rm -f /etc/systemd/system/dataflux-func-sidecar.service # Remove the Sidecar systemd service configuration
sudo systemctl daemon-reload                                 # Reload the systemd configuration to apply the service removal
sudo rm -f /usr/local/bin/dataflux-func-sidecar              # Remove the Sidecar executable
sudo rm -f /etc/dataflux-func-sidecar                        # Remove the configuration file containing BIND and SECRET_KEY
sudo userdel dffs                                            # Remove the Sidecar system user created during installation

6.1 Verification

Bash
1
2
3
4
systemctl status dataflux-func-sidecar.service                      # Check whether the service has been removed; on successful uninstall, it should report that the service cannot be found
test ! -e /usr/local/bin/dataflux-func-sidecar && echo "Program removed" # Confirm that the executable has been removed
test ! -e /etc/dataflux-func-sidecar && echo "Configuration removed"           # Confirm that the configuration file has been removed
getent passwd dffs || echo "User removed"                             # Confirm that the dffs user no longer exists

/etc/dataflux-func-sidecar contains BIND and SECRET_KEY. If you want to keep the configuration for reinstallation, back up this file before uninstalling.

If the dffs user is also used by other programs, do not execute sudo userdel dffs.

7. Security Notes

Because the Sidecar usage pattern essentially sends arbitrary executable code to the host machine, it is dangerous.

Therefore, Sidecar has the following restrictions in implementation and deployment:

  1. The Sidecar service runs as the dffs user (i.e., the abbreviation for DataFlux Func Sidecar)
  2. Sidecar must be configured with a SecretKey before it can normally execute Shell commands
  3. SidecarHelper.shell(...) internally implements HmacSha1 signing to prevent tampering and replay attacks

Since the Sidecar service runs as the dffs user, it cannot execute commands that require root privileges or operate on other users' files. If necessary, you can add the dffs user to a user group or modify file-related permissions.