Script Development / Pre-execution Scripts
Added in version 2.6.18
Although DataFlux Func provides PIP tool for installing third-party Python packages, it may fail to work properly due to missing dependency libraries.
For example, when users need to use OpenCV in DataFlux Func, besides installing opencv-python, it is also necessary to install dependency libraries via apt or other methods.
Otherwise, issues like the following may occur:
1. Using Pre-execution Scripts to Resolve Dependencies
To solve this problem, pre-execution scripts can be provided to DataFlux Func. These scripts are executed before DataFlux Func starts, allowing the installation of necessary dependencies.
The specific steps are as follows:
1.1 Prepare the Script
The DataFlux Func image is based on Ubuntu:22.04.
Taking the OpenCV dependency issue mentioned above as an example, the following Bash script can be prepared.
| Install OpenCV Dependencies | |
|---|---|
1 2 | |
Save it as a file named prepare-for-opencv.sh.
The filename can be arbitrary, but it must end with .sh. To avoid unnecessary trouble, do not use Unicode or other non-standard symbols in the filename.
Distinguishing Runtime Environments
DataFlux Func includes a Server side and a Worker side.
- The Server side is primarily the HTTP server, providing the web interface and HTTP APIs, and does not participate in Python code execution.
- The Worker side is the actual service for executing Python code.
Therefore, in most cases, pre-execution scripts only need to be executed in the Worker.
When it is necessary to distinguish the environment, you can read $1 to determine if the value is server or worker.
The Beat service, MySQL service, and Redis service do not execute pre-execution scripts.
Reference Bash code for environment distinction:
| Execute only in Worker container | |
|---|---|
1 2 3 | |
| Execute only in Server container | |
|---|---|
1 2 3 | |
1.2 Upload the Script
The storage directory for pre-execution scripts is as follows:
| Environment | Location |
|---|---|
| Inside container | /data/resources/pre-run-scripts/ |
| On host machine | {installation directory}/data/resources/pre-run-scripts/ |
Users can place pre-execution scripts into this directory on the host machine.
Alternatively, you can enter the file management in DataFlux Func and upload your own pre-execution script:
1.3 Restart Func and Verify
After completing all preparations, restart DataFlux Func.
Return to the previous script and execute it again. You will see that the opencv-python library can now be imported correctly:
2. Details of Pre-execution Script Execution
Each time DataFlux Func starts, it first checks for the existence of pre-execution scripts.
When pre-execution scripts exist, DataFlux Func executes them in order based on their filenames. DataFlux Func will only start normally after all scripts have executed successfully.
If you need to observe the execution process of the pre-execution scripts, you can track it with the following command:
| View container logs | |
|---|---|
1 | |
Taking the pre-execution script from the example above, you can see output similar to the following:
| Execution Log | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | |
It can be seen that the pre-execution script executed correctly and installed the required dependency packages.
3. Replacing Pre-installed Python Packages in the Image
In some cases, the package version required by the user's script is incompatible with the version already included in the Func image.
In such situations, "pre-execution scripts" can also be used to replace packages, for example:
| Install the latest version of the simplejson package | |
|---|---|
1 | |
Since "pre-execution scripts" are executed before Func starts, the packages installed via PIP affect not only user scripts but the entire DataFlux Func.
Therefore, when using this method, please ensure that these operations do not affect the operation of DataFlux Func itself.