Skip to content

Script Development / Basic Concepts

In DataFlux Func, there are some unique concepts specific to DataFlux Func. This document will explain these concepts.

1. Script Set, Script, and Function

Script Set, Script, and Function can be created in "Development / Script Lib". They are core concepts of DataFlux Func, and their IDs are directly specified by the user during creation/coding.

  • A "Script Set" is a collection of several scripts. Its ID is directly specified by the user during creation, and it can only contain scripts.
  • A "Script" is the Python script itself, which must belong to a Script Set. Its ID is directly specified by the user during creation.
  • A "Function" in DataFlux Func specifically refers to the top-level function decorated by the @DFF.API(...) decorator. It can be used as the entry function for calls by Func API, Cron Job, etc.

Script Set is not a folder

A Script Set is similar to a folder, but this "folder" is unrelated to folders in general Python coding.

When coding in DataFlux Func, you will frequently deal with the IDs of Script Sets, Scripts, and Functions, and these IDs are closely related.

Relationship Between Script Set, Script, and Function IDs

According to the hierarchical relationship of Script Set, Script, and Function, the ID of a lower-level concept must include the ID of the higher-level concept.

Assume there is a Script Set with an ID of demo. Then all scripts belonging to this Script Set must start with demo__ (double underscores).

Further assume that there is a Script with an ID of demo__test under this Script Set, and it contains a function def hello(...). The ID of this function will be demo__test.hello.

The ID example table is as follows:

Concept ID Example
Script Set demo
Script demo__test
Function demo__test.hello

Mutual References in Coding

In DataFlux Func scripts, it is allowed to reference another script to achieve code reuse.

Assume there is a Script demo__script_a containing a function func_a(). Then, when referencing this function in Script demo__script_b, you can use the following method:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement can also be used:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

You can also use the from ... import statement to import only the required function:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between scripts belonging to the same Script Set, you can omit the Script Set ID and use the abbreviated form starting with __ (double underscores):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Prefer Using Abbreviated Form

When referencing within the same Script Set, the abbreviated form (i.e., omitting the Script Set ID and starting with __) should be used as much as possible.

This way, when the entire Script Set is cloned and the Script Set ID changes, the code in the newly cloned Script Set can still correctly reference the scripts within it.

2. Connector

Connector can be created in "Development / Connector". It is a tool provided by DataFlux Func to connect to external systems, and its ID is directly specified by the user during creation.

In fact, writing Python code in DataFlux Func is not much different from writing in original Python. Developers can completely ignore Connectors and connect to external systems on their own in the code.

However, for some external systems with connection pool concepts, Connectors have built-in connection pools, which can maintain connections during repeated function executions, avoiding repeatedly creating/closing connections with external systems.

Assume the user has configured a Connector with an ID of mysql. The code to obtain the operation object of this Connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different Connectors have different operation methods and parameters. For details, please refer to Script Development / Connector Object DFF.CONN

3. Environment Variables

Environment Variables can be created in "Development / Environment Variables". They are simple Key-Value configuration reading tools provided by DataFlux Func, and their IDs are directly specified by the user during creation.

Environment Variables are particularly suitable for scenarios where the same set of code runs in different environments.

For example, if the system accessed by the script distinguishes between test and production environments, you can set Environment Variables to switch between test and production environments without changing the code.

Assume the user has configured an Environment Variable with an ID of api_endpoint. The code to obtain the value of this Environment Variable is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Func API

Func API can be created in "Management / Func API". It is a common way to externally call functions in DataFlux Func. The calling process can be synchronous or asynchronous. When executed synchronously, the function can directly return the result to the caller after execution.

After creating a Func API for a function, it supports multiple different calling methods.

Func API supports GET and POST methods. Both methods support both "Simplified Form" and "Standard Form" for parameter passing.

In addition, the "Simplified" form of the POST method also supports file upload. The following is a list of features supported by various calling methods:

Calling Method Pass kwargs Parameters kwargs Parameter Types Pass options File Upload Submit Arbitrary Format Body
GET Simplified Form Supported Only Strings Not Supported Not Supported Not Supported
GET Standard Form Supported Data Types in JSON Supported Not Supported Not Supported
POST Simplified Form Supported Only Strings Not Supported Supported Supported
POST Standard Form Supported Data Types in JSON Supported Not Supported Not Supported

Different Passing Methods Result in Parameter Type Restrictions

For calling methods where only strings can be passed in kwargs, parameter type conversion needs to be performed in the function. In the Func API list, you can click "API Calling Example" to view specific calling methods.

Assume the following function exists:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

Assume the "Func API" ID created for this function is func-api-xxxxx, and the parameters passed are x=100 (integer), y="hello" (string).

Then, the various calling methods are as follows:

GET Simplified Form Parameter Passing

If the function parameters are relatively simple, you can use the GET simplified form to pass parameters, making the interface more intuitive.

Since parameters passed in the URL cannot distinguish between the string "100" and the integer 100, the function will receive all parameters as strings when called. The function needs to perform type conversion on the parameters.

Text Only
1
GET /api/v1/al/func-api-xxxxx/s?x=100&y=hello

For readability, the example is the content before URLEncode. Actual URL parameters need to be URLEncoded.

GET Standard Form Parameter Passing

In some cases, if you cannot send a POST request, you can also use the GET method to call the interface.

When passing parameters in the GET standard form, serialize the entire kwargs as JSON and pass it as a URL parameter. Since the parameters are actually sent in JSON format, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

For example, in this case, the x parameter received by the function is an integer, and no type conversion is needed.

Text Only
1
GET /api/v1/al/func-api-xxxxx?kwargs={"x":100,"y":"hello"}

For readability, the example is the content before URLEncode. Actual URL parameters need to be URLEncoded.

POST Simplified Form Parameter Passing

In some cases, if you cannot send an HTTP request with a JSON body, you can also pass parameters in a form similar to a Form, where each field name is the parameter name.

Since Form data submission cannot distinguish between the string "100" and the integer 100, the function will receive all parameters as strings when called. The function needs to perform type conversion on the parameters.

Text Only
1
2
3
4
POST /api/v1/al/func-api-xxxxx/s
Content-Type: x-www-form-urlencoded

x=100&y=hello

In addition, the POST simplified form also supports file upload (the parameter/field name must be files), and it needs to be processed using form-data/multipart.

The page HTML code example is as follows:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Func API address (if this page and DataFlux Func are not under the same domain, you need to write the full http://domain:prot/api/v1/al/func-api-xxxxx/s
        // Note: File upload must use the simplified form Func API
        var API_URL = '/api/v1/al/func-api-xxxxx/s';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, generate a FormData object and send it as the request body
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', API_URL);
            xhr.send(data);
        });
    </script>
</html>

POST Standard Form Parameter Passing

POST standard form parameter passing is the most common calling method. Since the parameters are sent in JSON format through the request body, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

For example, in this case, the x parameter received by the function is an integer, and no type conversion is needed.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/func-api-xxxxx
Content-Type: application/json

{
    "kwargs": {
        "x": 100,
        "y": "hello"
    }
}

5. Cron Job

Cron Job can be created in "Management / Cron Job". It is used to let DataFlux Func automatically call functions periodically.

After creating a Cron Job for a function, the function will be executed according to the specified Crontab expression without external calls.

Because of this, all parameters of the executed function must be satisfied, i.e.:

  1. The function does not require input parameters
  2. The function requires input parameters, but they are all optional parameters
  3. The function requires mandatory parameters, and specific values are configured for them in the Cron Job

Distinguish Execution Function of Function Runtime

If a function is configured with both "Cron Job" and other execution functions, and you want to distinguish processing in different execution functions, you can use the built-in variable _DFF_CRONTAB to distinguish:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRON_EXPR:
        # Only output logs during Cron Job
        print(f'x + y = {result}')

    return