Skip to content

Script Development / Concepts

In DataFlux Func, there are some concepts unique to DataFlux Func. This document will explain them.

1. Script Sets, Scripts, and Functions

Script sets, scripts, and functions can be created in "Development / Script Library". They are the core concepts of DataFlux Func. Their IDs are directly specified by the user during creation/coding.

  • A "Script Set" is a collection of several scripts. Its ID is directly specified by the user upon creation, and it can only contain scripts.
  • A "Script" is the Python script itself. It always belongs to a script set. Its ID is directly specified by the user upon creation.
  • A "Function" in DataFlux Func specifically refers to the top-level function decorated with the @DFF.API(...) decorator. It can be used as the entry point for calls by function APIs, scheduled tasks, etc.

A Script Set is not a folder

A script set is similar to a folder, but this "folder" is unrelated to folders in general Python coding.

When coding in DataFlux Func, you will frequently deal with the IDs of script sets, scripts, and functions, and these IDs are closely related.

Relationship Between Script Set, Script, and Function IDs

According to the hierarchical relationship of script set, script, and function, the ID of a lower-level concept always includes the ID of the higher-level concept.

Assuming there is a script set with ID demo, then all scripts belonging to this script set must start with demo__ (double underscore).

Further assuming there is a script with ID demo__test under this script set, which contains a function def hello(...). Then the ID of this function is demo__test.hello.

Example ID table:

Concept Example ID
Script Set demo
Script demo__test
Function demo__test.hello

Mutual Referencing in Code

In DataFlux Func scripts, it is allowed to reference another script to achieve code reuse.

Assuming there is a script demo__script_a containing a function func_a(). Then, to reference this function in script demo__script_b, you can use the following method:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement can also be used:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

You can also use the from ... import statement to import only the needed function:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between scripts belonging to the same script set, you can omit the script set ID and use the abbreviated form starting with __ (double underscore):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Use the Abbreviated Form Whenever Possible

When referencing within a script set, you should use the abbreviated form (i.e., omitting the script set ID and starting with __) as much as possible.

This way, when the entire script set is cloned and the script set ID changes, the code within the cloned new script set can still correctly reference scripts within its own script set.

2. Connectors

Connectors can be created in "Development / Connectors". They are tools provided by DataFlux Func to connect to external systems. Their IDs are directly specified by the user upon creation.

In fact, writing Python code in DataFlux Func is not much different from writing original Python. Developers can completely ignore connectors and connect to external systems on their own in the code.

However, for some external systems that have the concept of connection pools, connectors have built-in connection pools. They can maintain connections during repeated function executions, avoiding repeatedly creating/closing connections to external systems.

Assuming the user has configured a connector with ID mysql, the code to get the operation object of this connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different connectors have different operation methods and parameters. For details, please refer to Script Development / Connector Object DFF.CONN

3. Environment Variables

Environment variables can be created in "Development / Environment Variables". They are simple Key-Value configuration reading tools provided by DataFlux Func. Their IDs are directly specified by the user upon creation.

Environment variables are particularly suitable for scenarios where the same set of code runs in different environments.

For example, if the system the script needs to access distinguishes between test/production environments, you can set environment variables to switch between test/production environments without changing the code.

Assuming the user has configured an environment variable with ID api_endpoint, the code to get the value of this environment variable is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Function API

Function APIs can be created in "Management / Function APIs". They are a common way for external systems to call functions in DataFlux Func. The call process can be synchronous or asynchronous. When executed synchronously, the function can directly return the result to the caller after execution.

After creating a function API for a function, multiple different calling methods are supported.

Function APIs support both GET and POST methods. Both methods support parameter passing in both "Simplified Form" and "Standard Form".

Additionally, the "Simplified" form of the POST method also supports file upload. The following is a list of feature support for various calling methods:

Calling Method Passing kwargs Parameters kwargs Parameter Types Passing options File Upload Submitting Arbitrary Format Body
GET Simplified Form Supported Strings only Not supported Not supported Not supported
GET Standard Form Supported Data types in JSON Supported Not supported Not supported
POST Simplified Form Supported Strings only Not supported Supported Supported
POST Standard Form Supported Data types in JSON Supported Not supported Not supported

Different passing methods lead to restrictions on parameter types

For calling methods where parameters in kwargs can only be strings, type conversion of parameters needs to be performed within the function. In the function API list, you can click "API Call Example" to view specific calling methods.

Assuming the following function exists:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

Assuming the ID of the "Function API" created for this function is func-api-xxxxx, and the parameters passed are x=100 (integer), y="hello" (string).

Then, the various calling methods are as follows:

GET Simplified Form Parameter Passing

If the function parameters are relatively simple, you can use the GET simplified form to pass parameters, making the interface more intuitive.

Since parameters passed in the URL cannot distinguish between the string "100" and the integer 100, when the function is called, all parameters received are strings. The function needs to perform type conversion on the parameters itself.

Text Only
1
GET /api/v1/al/func-api-xxxxx/s?x=100&y=hello

For readability, the example shows content before URLEncode. Actual URL parameters need to be URLEncoded.

GET Standard Form Parameter Passing

In some cases, if a POST request cannot be sent, you can also use the GET method to call the interface.

When using GET standard form parameter passing, serialize the entire kwargs into JSON and pass it as a URL parameter. Since the parameters are actually sent in JSON format, their original types are preserved. The function does not need to perform type conversion on the parameters.

For example, in this case, the x parameter received by the function is an integer, no type conversion is needed.

Text Only
1
GET /api/v1/al/func-api-xxxxx?kwargs={"x":100,"y":"hello"}

For readability, the example shows content before URLEncode. Actual URL parameters need to be URLEncoded.

POST Simplified Form Parameter Passing

In some cases, if an HTTP request with a JSON body cannot be sent, you can pass parameters in a form similar to a Form form, where each field name is the parameter name.

Since Form form submission data cannot distinguish between the string "100" and the integer 100, when the function is called, all parameters received are strings. The function needs to perform type conversion on the parameters itself.

Text Only
1
2
3
4
POST /api/v1/al/func-api-xxxxx/s
Content-Type: x-www-form-urlencoded

x=100&y=hello

Additionally, POST simplified form parameter passing also supports file upload (the parameter/field name must be files), requiring processing using the form-data/multipart method.

Example HTML page code:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Function API address (if this page and DataFlux Func are not under the same domain name, write the full http://domain:port/api/v1/al/func-api-xxxxx/s
        // Note: Uploading files must use the simplified form function API.
        var API_URL = '/api/v1/al/func-api-xxxxx/s';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, create a FormData object and send it as the request body.
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', API_URL);
            xhr.send(data);
        });
    </script>
</html>

POST Standard Form Parameter Passing

POST standard form parameter passing is the most common calling method. Since parameters are sent via the request body in JSON format, their original types are preserved. The function does not need to perform type conversion on the parameters.

For example, in this case, the x parameter received by the function is an integer, no type conversion is needed.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/func-api-xxxxx
Content-Type: application/json

{
  "kwargs": {
    "x": 100,
    "y": "hello"
  }
}

5. Scheduled Tasks

Scheduled tasks can be created in "Management / Scheduled Tasks". They are used to make DataFlux Func automatically call functions periodically.

After creating a scheduled task for a function, the function will execute according to the specified Crontab expression without needing external calls.

Because of this, all parameters for the executed function must be satisfied, meaning:

  1. The function does not require input parameters.
  2. The function requires input parameters, but they are all optional parameters.
  3. The function requires mandatory parameters, and specific values are configured for them in the scheduled task.

Distinguishing the Execution Function to Which the Function Belongs at Runtime

If a function is configured with both "Scheduled Tasks" and other execution functions, and you want to distinguish processing between different execution functions, you can check the built-in variable _DFF_CRONTAB to differentiate:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRON_EXPR:
        # Output log only during scheduled tasks
        print(f'x + y = {result}')

    return