Sidecar

Sidecar is an accompanying component of DataFlux Func.

Since DataFlux Func typically runs within a container and cannot directly execute Shell commands on the host machine, Sidecar is a program that runs on the host machine. It acts as a proxy for DataFlux Func to directly execute Shell commands on the host machine.

Download Command

/bin/bash -c "$(curl -fsSL func.guance.com/sidecar-download)"

0. Pre-reading Tips

All shell commands mentioned in this article can be run directly under the root user. For non-root users, add sudo before running.

1. System and Environment Requirements

Any host that can run DataFlux Func normally should also be able to run Sidecar.

1.1 DataFlux Func Version Requirements

Sidecar must be used in conjunction with DataFlux Func version 1.3.5 or higher

2. Quick Installation

Under normal circumstances, Sidecar will be installed on the same host as DataFlux Func. The following operations assume they are performed on a host where DataFlux Func has already been installed.

If Sidecar and DataFlux Func are running on different hosts, the corresponding configurations need to be modified

2.1 Offline Installation

Before installing Sidecar, you need to download the required resources.

For hosts without internet access, the resources can be copied to the host via USB drives or other mobile devices.

The downloaded resource files come with an automatic installation script, which can be executed to install (see details below)

2.1.1 One-click Command to Download Resource Files

For Linux, macOS, and similar systems, it is recommended to use the official provided shell command to download the installation package.

Running the following command will automatically download all the necessary files for Sidecar. The download script will automatically select the x86_64 or aarch64 architecture version based on the current environment:

/bin/bash -c "$(curl -fsSL func.guance.com/sidecar-download)"

If you need to download a specific architecture version, use the following commands:

• Intel x86_64 Processor

/bin/bash -c "$(curl -fsSL func.guance.com/sidecar-download)" -- --arch=x86_64

• ARM aarch64 Processor (i.e., ARM64v8, such as: Raspberry Pi etc.)

/bin/bash -c "$(curl -fsSL func.guance.com/sidecar-download)" -- --arch=aarch64

After executing the command, all required files will be saved in the newly created dataflux-func-sidecar directory under the current directory.

• For installations where Sidecar needs to be placed on a non-internet-connected server, you can first download locally and then copy the entire directory to the target machine using a USB drive or other mobile storage device, or tools like scp
• For installations where Sidecar needs to be placed on a server that can access the internet, simply download directly on the server

2.1.2 Manual Download of Resource Files

For systems where it is inconvenient to use shell commands, you can manually download the required resource files.

If manual downloading is needed, here is the list of all the files:

#	Content	Filename	x86_64 Architecture	aarch64 Architecture
1	Sidecar Binary Program	dataflux-func-sidecar.tar.gz	Download	Download
2	Sidecar Service Configuration File	dataflux-func-sidecar.service	Download	Download
3	Sidecar Installation Script	run-sidecar.sh	Download	Download
4	Version Information	version	Download	Download

After manually downloading all the files, place them into a newly created dataflux-func-sidecar directory.

If there are updates, 【re-download all files】. Do not guess which files have changed and which have not.

When manually downloading, if using a browser, please ensure not to download cached old content!!

2.1.3 Use the Included Script to Execute the Installation

In the already downloaded dataflux-func-sidecar directory, run the following command to automatically configure and finally start the entire Sidecar:

Sidecar does not support Mac. Please copy it to a Linux system before running the installation.

sudo /bin/bash run-sidecar.sh

Using the automatic installation script allows quick installation and operation within seconds, automatically configuring the following:

• Create /usr/local/bin/dataflux-func-sidecar executable file
• Create /etc/dataflux-func-sidecar configuration file
• Create dffs user to run the Sidecar program
• Create /etc/systemd/system/dataflux-func-sidecar Systemd configuration file and set it to start on boot

After installation, the following installation information will be displayed:

Bind:
    127.0.0.1:8099,172.17.0.1:8099
Secret Key:
    xxxxxxxxxxxxxxxx
To shutdown:
    sudo systemctl start dataflux-func-sidecar
To start:
    sudo systemctl stop dataflux-func-sidecar

Now open 127.0.0.1:8099,172.17.0.1:8099 and have fun!

Meaning of the output content:

Item	Explanation	Corresponding Configuration File Item
Bind	Listening address. Supports multiple addresses separated by commas	BIND
Secret Key	Key. Used for request verification	SECRET_KEY

By default, during installation, Bind will have 2 addresses:

• 127.0.0.1:8099: Local network
• 172.17.0.1:8099: docker0, used for communication with DataFlux Func

2.2. Verify Installation

After default installation of Sidecar, you can use the following command to verify the installation:

curl http://127.0.0.1:8099

If the following information is returned, it indicates that Sidecar is running normally:

Welcome to DataFlux Func Sidecar
* Version: 0.0.1
* Release Date: 2021-10-17 00:00:00

2.3. Installation Options

The automatic installation script supports some installation options to adapt to different installation requirements.

During installation, simply add --{parameter}[ parameter configuration (if any)] after the automatic installation command to specify installation options.

For example, specifying the listening address:

sudo /bin/bash run-sidecar.sh --bind 0.0.0.0.8099

2.3.1 Available Installation Options

Specific parameter details are listed below

--bind: Specify the listening address

Sidecar listens by default to 127.0.0.1:8099,172.17.0.1:8099. If the port is occupied, you can choose another listening address.

--secret-key: Specify the key

DataFlux Func Sidecar generates a random key by default during installation, but you can manually specify the key using this parameter.

3. Invoking Sidecar in DataFlux Func

DataFlux Func provides a connector for DataFlux Func Sidecar, which can be used to operate Sidecar.

3.1. Create the Sidecar Connector

Go to «Connector / Add Connector / DataFlux Func Sidecar (HTTP)», correctly fill in the Secret Key to create it.

If you specified a different listening address when installing Sidecar (modified the Bind parameter), you need to make the corresponding changes when creating the connector.

3.2. Write Code

Here is an example code:

import json

@DFF.API('Test Sidecar')
def test_sidecar():
    sidecar = DFF.CONN('sidecar')
    res = sidecar.shell('hostname', wait=True, callback_url='http://172.17.0.1:8088/api/v1/al/auln-sidecar-callback')
    # res content:
    # (200, {'data': {'stderr': '', 'stdout': 'my_host\n'}, 'message': '', 'ok': True})
    return res

@DFF.API('Test Sidecar callback')
def test_sidecar_callback(**kwargs):
    # kwargs content:
    # {'stderr': '', 'stdout': 'my_host\n'}

3.3. Configure Function

In the above code example:

«Test Sidecar» is the main execution function, which can be configured as either a «Synchronous API (Old Version: Authorized Link)» or a «Scheduled Task (Old Version: Automatic Trigger Configuration)» for execution.

«Test Sidecar callback» is the function that receives the callback after execution, and needs to be configured as a «Synchronous API (Old Version: Authorized Link)»

In the example code, the ID part of the callback_url parameter for the Synchronous API (Old Version: Authorized Link) is auln-sidecar-callback. When configuring the callback function as a «Synchronous API (Old Version: Authorized Link)», the same ID must be specified to ensure the URL address matches.

For cases where Sidecar and DataFlux Func are installed on the same host, both can access each other through docker0, i.e., IP address is 172.17.0.1

Details about the «Sidecar Connector Operation Object» are listed below

4. Sidecar Connector Operation Object API

Using the Sidecar connector operation object allows users to invoke Sidecar to execute Shell commands.

Parameters for DFF.CONN(...) are as follows:

Parameter	Type	Required / Default Value	Description
data_source_id	str	Required	Connector ID

SidecarHelper.shell(...)

Invoke Sidecar to execute Shell commands. Parameters are as follows:

Parameter	Type	Required / Default Value	Description
cmd	str	Required	Shell command to be executed Example: "ls -l"
wait	bool	True	Whether to wait for completion Setting to False will cause the function to return immediately and terminal output will not be returned
workdir	str	None	Working directory for Shell command execution Example: "/home/dev"
envs	dict	None	Environment variables, both keys and values are strings Example: {"MY_NAME": "Tom"}
callback_url	str	None	Callback URL, after command execution, stdout and stderr will be sent to the specified URL using POST Generally used together with the wait=False parameter to implement asynchronous callbacks
timeout	int	3	Request timeout time Note: This parameter is not the timeout time for the Shell command, but the timeout time for Func's request to Sidecar i.e., Func's request to Sidecar may time out, but the executed Shell command will not stop due to this

Callback After Execution

After invoking SidecarHelper.shell(...) and specifying the callback_url parameter, Sidecar will send the standard output stdout and standard error stderr to the specified address using POST after executing the Shell command.

Specific structure is as follows:

POST {callback_url}
Content-Type: application/json

{
    "kwargs": {
        "stdout": "<standard output text>",
        "stderr": "<standard error text>"
    }
}

This structure matches the «Synchronous API (Old Version: Authorized Link)Standard POST Method» of DataFlux Func, so you can directly use «Synchronous API (Old Version: Authorized Link)» to receive the callback after execution

5. Routine Maintenance

By default, the executable program is installed at /usr/local/bin/dataflux-func-sidecar

5.1 System Upgrade

Re-execute the installation process. The automatic installation script will automatically replace the executable program and restart the service.

At the same time, previous configuration file contents will be retained.

5.2 Start/Stop/Restart Service

The Sidecar service is managed using systemd, and can be operated directly using systemctl:

sudo systemctl start dataflux-func-sidecar    # Start
sudo systemctl stop dataflux-func-sidecar     # Stop
sudo systemctl restart dataflux-func-sidecar  # Restart

5.3 View Configuration

Configuration file is located at /etc/dataflux-func-sidecar.

6. Security Assurance

Since the usage mode of Sidecar essentially involves sending arbitrary executable code to the host machine, it poses potential risks.

Therefore, Sidecar has the following restrictions in its implementation and deployment:

1. The Sidecar service runs under the dffs user (i.e., the first letters abbreviation of DataFlux Func Sidecar)
2. Sidecar must be configured with a SecretKey to properly call and execute Shell commands
3. SidecarHelper.shell(...) internally implements HmacSha1 signing to prevent tampering and replay attacks

Since the Sidecar service runs under the dffs user, it cannot execute commands requiring root privileges or manipulate files belonging to other users. If necessary, you can add the dffs user to user groups or modify file-related permissions.