Skip to content

Script Development / Basic Concepts

In DataFlux Func, there are some concepts unique to DataFlux Func. This document will explain these concepts.

1. Script Set, Script, and Function

Script Set, Script, and Function can be created in "Development / Script Lib". They are the core concepts of DataFlux Func, and their IDs are directly specified by the user during creation/coding.

  • "Script Set" is a collection of several scripts. The ID is directly specified by the user during creation, and it can only contain scripts.
  • "Script" refers to the Python script itself, which must belong to a Script Set. The ID is directly specified by the user during creation.
  • "Function" in DataFlux Func specifically refers to the top-level function decorated by the @DFF.API(...) decorator, which can be used as the entry function for Func API, Cron Job, etc.

Script Set is not a folder

A Script Set is similar to a folder, but this "folder" is unrelated to folders in general Python coding.

When coding in DataFlux Func, you will frequently deal with the IDs of Script Set, Script, and Function, and these IDs are closely related.

Relationship Between Script Set, Script, and Function IDs

According to the hierarchical relationship of Script Set, Script, and Function, the ID of a lower-level concept must include the ID of the upper-level concept.

Assume there is a Script Set with an ID of demo, then all scripts belonging to this Script Set must start with demo__ (double underscores).

Further assume there is a script with an ID of demo__test under this Script Set, which contains a function def hello(...). Then, the ID of this function is demo__test.hello.

The ID example table is as follows:

Concept ID Example
Script Set demo
Script demo__test
Function demo__test.hello

Mutual References in Coding

In DataFlux Func scripts, it is allowed to reference another script for code reuse.

Assume there is a script demo__script_a containing a function func_a(). Then, when referencing this function in script demo__script_b, you can use the following method:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement can also be used:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

You can also use the from ... import statement to import only the required function:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between scripts belonging to the same Script Set, you can omit the Script Set ID and use the abbreviated form starting with __ (double underscores):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Prefer Using Abbreviated Form

When referencing within a Script Set, the abbreviated form (i.e., omitting the Script Set ID and starting with __) should be used as much as possible.

This way, when the entire Script Set is cloned and the Script Set ID changes, the code within the cloned new Script Set can still correctly reference scripts within this Script Set.

2. Connector

Connector can be created in "Development / Connector". It is a tool provided by DataFlux Func to connect to external systems, and the ID is directly specified by the user during creation.

In fact, writing Python code in DataFlux Func is not much different from writing Python code in general. Developers can completely ignore Connector and connect to external systems on their own in the code.

However, for external systems with connection pool concepts, Connector has a built-in connection pool that can maintain the connection during repeated function runs, avoiding repeatedly creating/closing connections to external systems.

Assume the user has configured a Connector with an ID of mysql. The code to get the operation object of this Connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different Connectors have different operation methods and parameters. For details, please refer to Script Development / Connector Object DFF.CONN

3. Environment Variable

Environment Variable can be created in "Development / Environment Variable". It is a simple Key-Value configuration reading tool provided by DataFlux Func, and the ID is directly specified by the user during creation.

Environment Variable is particularly suitable for scenarios where the same set of code runs in different environments.

For example, if the system accessed by the script distinguishes between test/production environments, you can set Environment Variables to switch between test/production environments without changing the code.

Assume the user has configured an Environment Variable with an ID of api_endpoint. The code to get the value of this Environment Variable is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Func API

Func API can be created in "Management / Func API". It is a common way to externally call functions in DataFlux Func. The call process can be synchronous or asynchronous. When executed synchronously, the function can directly return the result to the caller after execution.

After creating a Func API for a function, multiple different calling methods are supported.

Func API supports GET and POST methods. Both methods support "Simplified Form" and "Standard Form" for parameter passing.

In addition, the "Simplified" form of POST also supports file upload. The following is a list of functional support for various calling methods:

Calling Method Passing kwargs Parameters kwargs Parameter Types Passing options File Upload Submit Arbitrary Format Body
GET Simplified Form Supported Only Strings Not Supported Not Supported Not Supported
GET Standard Form Supported Data Types in JSON Supported Not Supported Not Supported
POST Simplified Form Supported Only Strings Not Supported Supported Supported
POST Standard Form Supported Data Types in JSON Supported Not Supported Not Supported

Different Passing Methods May Result in Parameter Type Restrictions

For calling methods where only strings can be passed in kwargs, type conversion of parameters needs to be done in the function. In the Func API list, you can click "API Calling Example" to view specific calling methods.

Assume there is the following function:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

Assume the "Func API" ID created for this function is func-api-xxxxx, and the parameters passed are x=100 (integer), y="hello" (string).

Then, the various different calling methods are as follows:

GET Simplified Form Parameter Passing

If the function parameters are simple, you can use the GET simplified form to pass parameters, making the interface more intuitive.

Since parameters passed in the URL cannot distinguish between the string "100" and the integer 100, the function will receive all parameters as strings when called. The function needs to perform type conversion on the parameters.

Text Only
1
GET /api/v1/al/func-api-xxxxx/simplified?x=100&y=hello

For readability, the example is the content before URLEncode. The actual URL parameters need to be URLEncoded.

GET Standard Form Parameter Passing

In some cases, if you cannot send a POST request, you can also use the GET method to call the interface.

When passing parameters in the GET standard form, the entire kwargs is serialized into JSON and passed as a URL parameter. Since the parameters are actually sent in JSON format, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

In this example, the function will receive the x parameter as an integer, and no type conversion is needed.

Text Only
1
GET /api/v1/al/func-api-xxxxx?kwargs={"x":100,"y":"hello"}

For readability, the example is the content before URLEncode. The actual URL parameters need to be URLEncoded.

POST Simplified Form Parameter Passing

In some cases, if you cannot send an HTTP request with a JSON body, you can also pass parameters in a form similar to a Form, where the field names are the parameter names.

Since Form submissions cannot distinguish between the string "100" and the integer 100, the function will receive all parameters as strings when called. The function needs to perform type conversion on the parameters.

Text Only
1
2
3
4
POST /api/v1/al/func-api-xxxxx/simplified
Content-Type: x-www-form-urlencoded

x=100&y=hello

In addition, the POST simplified form also supports file upload (the parameter/field name must be files), and the form-data/multipart method needs to be used for processing.

The page HTML code example is as follows:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Func API address (if this page and DataFlux Func are not under the same domain, you need to write the full http://domain:prot/api/v1/al/func-api-xxxxx/simplified
        // Note: File upload must use the simplified form Func API
        var API_URL = '/api/v1/al/func-api-xxxxx/simplified';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, generate a FormData object and send it as the request body
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', API_URL);
            xhr.send(data);
        });
    </script>
</html>

POST Standard Form Parameter Passing

The POST standard form parameter passing is the most common calling method. Since the parameters are sent in JSON format through the request body, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

In this example, the function will receive the x parameter as an integer, and no type conversion is needed.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/func-api-xxxxx
Content-Type: application/json

{
    "kwargs": {
        "x": 100,
        "y": "hello"
    }
}

5. Cron Job

Cron Job can be created in "Management / Cron Job". It is used to let DataFlux Func automatically call functions periodically.

After creating a Cron Job for a function, the function will be executed periodically according to the specified Crontab expression, without external calls.

Because of this, all parameters of the executed function must be satisfied, i.e.:

  1. The function does not require input parameters
  2. The function requires input parameters, but all are optional parameters
  3. The function requires mandatory parameters, and specific values are configured for them in the Cron Job

Distinguishing the Execution Function to Which the Function Belongs During Runtime

If a function is configured with both "Cron Job" and other execution functions, and you want to distinguish processing in different execution functions, you can judge the built-in variable _DFF_CRONTAB to distinguish:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRON_EXPR:
        # Only output logs during Cron Job
        print(f'x + y = {result}')

    return