Skip to content

Script Development / Pre-execution Scripts

Added in version 2.6.18

Important Notice

The pre-execution script directory was changed from init-scripts to pre-run-scripts in version 2.6.22.

Meanwhile, the new version will automatically copy all files from init-scripts to pre-run-scripts, so users do not need to perform any additional operations.

Although DataFlux Func already provides PIP tool installation for third-party Python packages, it may still be unusable due to missing dependency libraries.

For example, if a user wants to use OpenCV in DataFlux Func, besides installing opencv-python, they also need to install dependency libraries via apt or similar methods.

Otherwise, the following issues might occur:

dependency-error.png

1. Resolve Dependencies Using Pre-execution Scripts

To solve this problem, you can provide pre-execution scripts to DataFlux Func. These scripts will execute before DataFlux Func starts, allowing for the installation of necessary dependencies.

The specific steps are as follows:

1.1 Prepare the Script

The DataFlux Func image is based on Ubuntu:22.04, and APT uses TUNA at Tsinghua University (http://mirrors.tuna.tsinghua.edu.cn/ubuntu).

As an example to resolve the OpenCV dependency issue mentioned earlier, you can prepare the following Bash script.

Bash
1
2
3
4
5
6
7
8
9
# This script only needs to be executed on the Worker side.
# It differentiates based on the value of $1.
if [ $1 != 'worker' ]; then
    exit 0
fi

# Install OpenCV dependencies
apt-get update
apt-get install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxext6 libxrender-dev

Save it as the file prepare-for-opencv.sh.

The filename can be arbitrary but must end with .sh. To avoid unnecessary trouble, do not use Chinese characters or other unconventional symbols in the filename.

Differentiate Running Environments

DataFlux Func includes both Server and Worker sides.

  • The Server side is mainly the HTTP server, providing Web pages and HTTP APIs, and does not participate in executing Python code.
  • The Worker side is the actual Python code execution service.

Therefore, in most cases, the pre-execution script only needs to run on the Worker side.

To differentiate environments, you can read $1 and check whether its value is server or worker.

Beat services, MySQL services, and Redis services will not execute pre-execution scripts.

Example Bash code for differentiation:

Bash
1
2
3
4
# Execute only on the Worker side
if [ $1 != 'worker' ]; then
    exit 0
fi
Bash
1
2
3
4
# Execute only on the Server side
if [ $1 != 'server' ]; then
    exit 0
fi

1.2 Upload the Script

The pre-execution script storage directory is as follows:

Environment Location
Inside Container /data/resources/pre-run-scripts/
On Host Machine {Installation Directory}/data/resources/pre-run-scripts/

Users can place the pre-execution script inside the host machine's directory.

Alternatively, you can upload your own pre-execution script by navigating through the file manager in DataFlux Func:

enable-file-manager.png

upload-script.png

script-ready.png

1.3 Restart DataFlux Func and Verify

After completing all preparations, restart DataFlux Func.

Returning to the previous script and running it again, you'll see that the opencv-python library can now be correctly imported:

import-ok.png

2. Pre-execution Script Execution Details

Each time DataFlux Func starts, it first checks whether there are any pre-execution scripts.

When pre-execution scripts exist, DataFlux Func executes them in order based on their filenames, and DataFlux Func will only start normally after all scripts have been successfully executed.

If you need to observe the execution process of the pre-execution script, you can track it using the following command:

Bash
1
docker logs {DataFlux Func Container ID} -f

Using the pre-execution script from the example above, you can see output like the following:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
[PRERUN SCRIPT] prepare-for-opencv.sh
Hit:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [127 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe arm64 Packages [1528 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages [2960 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted arm64 Packages [3920 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main arm64 Packages [82.8 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe arm64 Packages [33.3 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe arm64 Packages [1230 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports jammy-security/main arm64 Packages [2656 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [3777 kB]
Fetched 16.6 MB in 18s (923 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
libxext6 is already the newest version (2:1.3.4-1build1).
libxext6 set to manually installed.
The following additional packages will be installed:
  libdrm-amdgpu1 libdrm-common libdrm-nouveau2 libdrm-radeon1 libdrm2 libgl1
  libgl1-amber-dri libgl1-mesa-dri libglapi-mesa libglib2.0-data libglvnd0
  libglx-mesa0 libglx0 libice6 libllvm15 libpthread-stubs0-dev libx11-dev
  libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
  libxcb-present0 libxcb-randr0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0
  libxcb1-dev libxdmcp-dev libxfixes3 libxrender1 libxshmfence1 libxxf86vm1
  shared-mime-info x11-common x11proto-dev xdg-user-dirs xorg-sgml-doctools
  xtrans-dev
Suggested packages:
  libx11-doc libxcb-doc
The following NEW packages will be installed:
  libdrm-amdgpu1 libdrm-common libdrm-nouveau2 libdrm-radeon1 libdrm2 libgl1
  libgl1-amber-dri libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libglib2.0-0
  libglib2.0-data libglvnd0 libglx-mesa0 libglx0 libice6 libllvm15
  libpthread-stubs0-dev libsm6 libx11-dev libx11-xcb1 libxau-dev libxcb-dri2-0
  libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-shm0
  libxcb-sync1 libxcb-xfixes0 libxcb1-dev libxdmcp-dev libxfixes3
  libxrender-dev libxrender1 libxshmfence1 libxxf86vm1 shared-mime-info
  x11-common x11proto-dev xdg-user-dirs xorg-sgml-doctools xtrans-dev
0 upgraded, 43 newly installed, 0 to remove and 27 not upgraded.
Need to get 38.4 MB of archives.
After this operation, 163 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libglib2.0-0 arm64 2.72.4-0ubuntu2.5 [1435 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libglib2.0-data all 2.72.4-0ubuntu2.5 [4656 B]
Get:3 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 shared-mime-info arm64 2.1-2 [453 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 xdg-user-dirs arm64 0.17-2ubuntu4 [53.5 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libdrm-common all 2.4.113-2~ubuntu0.22.04.1 [5450 B]

Omitted below

It can be seen that the pre-execution script executed correctly and installed the required dependency packages.