Skip to content

Troubleshooting / Package Cannot Be Imported or Wrong Version

1. Introduction

In normal Python development, any change to a third-party package requires restarting the system.

However, in DataFlux Func, for development convenience, installing a third-party package does not require restarting the entire DataFlux Func system.

However, this does not guarantee 100% reliability.

For a third-party package to be installed and upgraded normally without restarting DataFlux Func, the following conditions must be met:

  1. A package written in pure Python (e.g., does not contain C extensions)
  2. No preloaded content (e.g., various large data models)

When a third-party package does not meet the above conditions, it can only be guaranteed to work normally at the first installation; subsequent updates require restarting the entire DataFlux Func

2. Troubleshooting Case: numpy

The commonly used numpy package is a typical package that uses C extensions. After deploying DataFlux Func, installing numpy for the first time works normally, but if you later install a different version of numpy again,

Then, you may encounter the following failures:

2.1 numpy Cannot Be Imported

Taking numpy as an example, when the Script executes import numpy, the following error is thrown:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#1 --------------------
Execute function: demo__demo.test_numpy()

Error stack:
Traceback (most recent call last):
  File "demo__demo", line 1, in <module>
    import numpy
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
    https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
  * The Python version is: Python3.8 from "/opt/python/bin/python3.8"
  * The NumPy version is: "1.22.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: libopenblas64_p-r0-2f7c42d4.3.18.so: cannot open shared object file: No such file or directory

2.2 numpy Can Be Used but Calls an Old Version

Taking numpy as an example, by printing the package object's __version___, you can see the version of the numpy package during actual code execution, but it is inconsistent with the version actually installed.

Python
1
print(numpy.__version__)

Satisfying this condition does not mean the code can still run normally. After simply restarting DataFlux Func, it will again become the problem in 1.1 above. Do not ignore it.

3. Cause Explanation

Third-party packages such as numpy may involve downloading additional resources during installation (such as code in other languages that needs to be compiled, data models, etc.). After the first installation, Python can read the third-party package normally and load external data.

However, when a different version of the same package is installed again, the resource files from the previous version are still in use and cannot actually be updated / overwritten, so the package is not installed correctly.

At this point, for DataFlux Func, the following situations exist:

  1. If a Worker process has already imported this third-party package before, this Worker process may still be able to run relying on cached data, but it appears to call the old version (that is, the version cached for the first time).
  2. If a Worker process has never imported this third-party package before, then when it performs the first import and loads related resources, since the package was not installed correctly during the second installation, it cannot be imported normally.
  3. At this point, if DataFlux Func is restarted, the above situation 1 will change to the above situation 2 because the restart releases the cache.

4. Solution

Taking numpy 1.22.1 as an example, after installation, the following folders will be generated under resource directory / extra-python-packages:

  1. numpy
  2. numpy-1.22.1.dist-info
  3. numpy.libs

The default host location of the resource directory is: /usr/local/dataflux-func/data/resources/extra-python-packages

The location of the resource directory in the container is: /data/resources/extra-python-packages

The steps to solve this problem are as follows:

  1. Completely delete the above-mentioned folders related to numpy
  2. Restart DataFlux Func
  3. Reinstall numpy