Script Development / Pre-execution Scripts
Added in version 2.6.18
Important Notice
The pre-execution script directory was changed from init-scripts to pre-run-scripts in version 2.6.22.
At the same time, the new version will automatically copy all files from init-scripts to pre-run-scripts, so users do not need to perform any additional operations.
Although DataFlux Func already provides PIP tool for installing third-party Python packages, it may fail to work properly due to missing dependency libraries.
For example, if a user wants to use OpenCV in DataFlux Func, besides installing opencv-python, they also need to install dependency libraries via apt or similar methods.
Otherwise, the following issues might arise:
1. Resolve Dependencies Using Pre-execution Scripts
To address this issue, you can provide pre-execution scripts to DataFlux Func. These scripts will run before DataFlux Func starts, allowing for the installation of necessary dependencies.
The specific steps are as follows:
1.1 Prepare the Script
DataFlux Func's image is based on Ubuntu:20.04, and the APT source uses TUNA at Tsinghua University (http://mirrors.tuna.tsinghua.edu.cn/ubuntu).
To resolve the OpenCV dependency issue mentioned above, you can prepare the following Bash script.
| Bash |
|---|
| # Script that only needs to be executed on the Worker end
# Distinguished by the value of $1
if [ $1 != 'worker' ]; then
exit 0
fi
# Install OpenCV dependencies
apt-get update
apt-get install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxext6 libxrender-dev
|
Save it as the file prepare-for-opencv.sh.
Filename is arbitrary, but it must end with .sh. To avoid unnecessary trouble, filenames should not include Chinese characters or other unconventional symbols.
Distinguish Running Environments
DataFlux Func includes both Server and Worker ends.
- The Server end is mainly an HTTP server that provides Web pages and HTTP APIs, and does not participate in executing Python code.
- The Worker end is the actual Python code execution service.
Therefore, in most cases, the pre-execution script only needs to be executed on the Worker end.
If you need to distinguish between environments, you can read $1 and check whether its value is server or worker.
Beat services, MySQL services, Redis services will not execute pre-execution scripts.
Example Bash code for distinguishing environments:
| Bash |
|---|
| # Execute only on the Worker end
if [ $1 != 'worker' ]; then
exit 0
fi
|
| Bash |
|---|
| # Execute only on the Server end
if [ $1 != 'server' ]; then
exit 0
fi
|
1.2 Upload the Script
The storage directory for pre-execution scripts is as follows:
| Environment |
Location |
| Container |
/data/resources/pre-run-scripts/ |
| Host |
{Installation Directory}/data/resources/pre-run-scripts/ |
Users can place pre-execution scripts into the host system.
You can also upload your own pre-execution scripts through DataFlux Func's file management:
1.3 Restart DataFlux Func and Verify
After completing all preparations, restart DataFlux Func.
Go back to the previous script and run it again; you'll see that the opencv-python library can now be imported correctly:
2. Details of Pre-execution Script Execution
Each time DataFlux Func starts, it first checks whether pre-execution scripts exist.
When pre-execution scripts are present, DataFlux Func executes them in alphabetical order by name, and DataFlux Func will only start normally after all scripts have been successfully executed.
If you need to observe the execution process of the pre-execution scripts, you can track it using the following command:
| Bash |
|---|
| docker logs {DataFlux Func Container ID} -f
|
Using the pre-execution script from the example above, you can see the following output:
| Text Only |
|---|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64 | Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal InRelease
Get:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates InRelease [114 kB]
Get:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-backports InRelease [108 kB]
Get:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-security InRelease [114 kB]
Get:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/restricted amd64 Packages [2277 kB]
Get:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/multiverse amd64 Packages [31.2 kB]
Get:7 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/universe amd64 Packages [1329 kB]
Get:8 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 Packages [3111 kB]
Get:9 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-security/main amd64 Packages [2629 kB]
Get:10 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-security/restricted amd64 Packages [2139 kB]
Get:11 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-security/universe amd64 Packages [1033 kB]
Get:12 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-security/multiverse amd64 Packages [28.5 kB]
Fetched 12.9 MB in 4s (3104 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
libxext6 is already the newest version (2:1.3.4-0ubuntu1).
libxext6 set to manually installed.
The following additional packages will be installed:
libdrm-amdgpu1 libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1
libdrm2 libgl1 libgl1-mesa-dri libglapi-mesa libglib2.0-data libglvnd0
libglx-mesa0 libglx0 libice6 libllvm12 libpciaccess0 libpthread-stubs0-dev
libvulkan1 libwayland-client0 libx11-dev libx11-xcb1 libxau-dev
libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0
libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcb1-dev libxdmcp-dev libxfixes3
libxrender1 libxshmfence1 libxxf86vm1 mesa-vulkan-drivers shared-mime-info
x11-common x11proto-core-dev x11proto-dev xdg-user-dirs xorg-sgml-doctools
xtrans-dev
Suggested packages:
pciutils libx11-doc libxcb-doc
The following NEW packages will be installed:
libdrm-amdgpu1 libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1
libdrm2 libgl1 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libglib2.0-0
libglib2.0-data libglvnd0 libglx-mesa0 libglx0 libice6 libllvm12
libpciaccess0 libpthread-stubs0-dev libsm6 libvulkan1 libwayland-client0
libx11-dev libx11-xcb1 libxau-dev libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
libxcb-present0 libxcb-randr0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0
libxcb1-dev libxdmcp-dev libxfixes3 libxrender-dev libxrender1 libxshmfence1
libxxf86vm1 mesa-vulkan-drivers shared-mime-info x11-common
x11proto-core-dev x11proto-dev xdg-user-dirs xorg-sgml-doctools xtrans-dev
0 upgraded, 48 newly installed, 0 to remove and 62 not upgraded.
Need to get 39.6 MB of archives.
After this operation, 538 MB of additional disk space will be used.
Get:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libglib2.0-0 amd64 2.64.6-1~ubuntu20.04.4 [1287 kB]
... omitted in the middle ...
Setting up libx11-dev:amd64 (2:1.6.9-2ubuntu1.2) ...
Setting up libxrender-dev:amd64 (1:0.9.10-1) ...
Setting up libglx-mesa0:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libglx0:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up libgl1:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up libgl1-mesa-glx:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
[YAML Resource] Config Overrided by: `/data/user-config.yaml`
Start upgrade process...
Database Timezone: +00:00
Already up to date, skip.
[YAML Resource] Config Overrided by: `/data/user-config.yaml`
Database Timezone: +00:00
Web Server is listening on port [ 8088 ] (Press CTRL+C to quit)
PID: 766
Have fun!
|
It can be seen that the pre-execution script has been correctly executed and installed the required dependency packages.