Script Development / Pre-run Scripts
New in version 2.6.18
Although DataFlux Func already provides PIP tool for installing third-party Python packages, some may not work properly due to missing dependency libraries.
For example, if users need to use OpenCV in DataFlux Func, in addition to installing opencv-python, they also need to install dependency libraries via apt or other means.
Otherwise, the following problems may occur:
1. Using Pre-run Scripts to Resolve Dependencies
To solve this problem, you can provide pre-run scripts to DataFlux Func. The pre-run scripts will be executed before DataFlux Func starts, thereby installing the necessary dependencies.
The specific steps are as follows:
1.1 Preparing the Script
The DataFlux Func image is based on Ubuntu:22.04.
Taking the OpenCV dependency issue mentioned above as an example, you can prepare the following Bash script.
| Install OpenCV Dependencies | |
|---|---|
1 2 | |
Save it as the prepare-for-opencv.sh file.
The file name can be arbitrary, but it must end with .sh. To avoid unnecessary trouble, do not use Unicode or other unconventional symbols in the file name.
Distinguishing Runtime Environments
DataFlux Func includes a Server side and a Worker side.
- The Server side is mainly an HTTP server, providing web pages and HTTP APIs, and does not participate in Python code execution.
- The Worker side is the service that actually executes Python code.
Therefore, in most cases, pre-run scripts only need to be executed in the Worker.
When you need to distinguish the current environment, you can read $1 and check whether its value is server or worker.
Beat service, MySQL service, and Redis service will not execute pre-run scripts
The reference Bash code for distinguishing the environment is as follows:
| Execute only in Worker container | |
|---|---|
1 2 3 | |
| Execute only in Server container | |
|---|---|
1 2 3 | |
1.2 Uploading the Script
The storage directory for pre-run scripts is as follows:
| Environment | Location |
|---|---|
| In container | /data/resources/pre-run-scripts/ |
| On the host machine | {installation directory}/data/resources/pre-run-scripts/ |
Users can place pre-run scripts into this directory on the host machine.
You can also go to File Management in DataFlux Func to upload the pre-run scripts you have written:
1.3 Restart Func and Verify
After completing all preparations, restart DataFlux Func
Go back to the previous script and run it again. You can see that the opencv-python library can now be imported correctly:
2. Pre-run Script Execution Details
Every time DataFlux Func starts, it first checks whether pre-run scripts exist.
When pre-run scripts exist, DataFlux Func sorts them by name and executes them in sequence, and it will start normally only after all scripts have been executed successfully.
If you need to observe the execution process of pre-run scripts, you can track it with the following command:
| View container logs | |
|---|---|
1 | |
Taking the pre-run script above as an example, you can see the following output:
| Execution Log | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | |
As you can see, the pre-run script executed correctly and installed the required dependency packages.
3. Replacing Python Packages in the Image
In some cases, the packages that users need in their scripts are incompatible with the package versions already included in the Func image.
In this case, you can also use "pre-run scripts" to replace packages, for example:
| Install the latest simplejson package | |
|---|---|
1 | |
Since "pre-run scripts" execute before Func starts, PIP-installed packages affect not only user scripts but also the entire DataFlux Func.
Therefore, when using them, please make sure these operations will not affect the operation of DataFlux Func itself.