Skip to content

Script Development / Basic Concepts

In DataFlux Func, there are some concepts unique to DataFlux Func. This document will explain them.

1. Script Set, Script, and Function

Script Sets, Scripts, and Functions can be created in Development / Script Lib. They are core concepts of DataFlux Func. IDs are specified directly by the user when creating / writing code.

  • A Script Set is a collection of several scripts. Its ID is specified directly by the user at creation, and it can only contain scripts.
  • A Script is the Python script itself. It must belong to a certain Script Set, and its ID is specified directly by the user at creation.
  • A Function in DataFlux Func specifically refers to a top-level function decorated with the @DFF.API(...) decorator, and it can be used as an entry function called by Func API, Cron Job, etc.

Script Set is not a folder

A Script Set is similar to a folder, but this "folder" is not related to folders in general Python coding.

When coding in DataFlux Func, you will frequently deal with the IDs of Script Sets, Scripts, and Functions, and these IDs are closely related.

Relationship Between Script Set, Script, and Function IDs

According to the hierarchical relationship among Script Set, Script, and Function, the ID of a lower-level concept must contain the ID of the upper-level concept.

Suppose there is a Script Set with ID demo. Then all Scripts belonging to this Script Set must start with demo__ (double underscore).

Suppose there is a Script with ID demo__test under this Script Set, and it contains a function def hello(...). Then the ID of this function is demo__test.hello.

The ID example table is as follows:

Concept ID Example
Script Set demo
Script demo__test
Function demo__test.hello

Mutual References in Code

In DataFlux Func Scripts, you can reference another Script to achieve code reuse.

Suppose there is a Script demo__script_a that contains a function func_a(). Then, when referencing this function in Script demo__script_b, you can use the following methods:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement can also be used:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

You can also use the from ... import statement to import only the required functions:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between Scripts belonging to the same Script Set, the Script Set ID can be omitted and expressed in the abbreviated form that starts with __ (double underscore):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Use the Abbreviated Form Whenever Possible

References within a Script Set should use the abbreviated form whenever possible (that is, the form that omits the Script Set ID and starts with __).

In this way, after the entire Script Set is cloned and its ID changes, the code in the cloned new Script Set can still correctly reference Scripts within its own Script Set.

2. Connector

Connectors can be created in Development / Connector. They are tools provided by DataFlux Func for connecting to external systems. The ID is specified directly by the user at creation.

In fact, writing Python code in DataFlux Func is not much different from standard Python. Developers can completely ignore Connectors and connect to external systems on their own in code.

However, for some external systems with the concept of connection pooling, Connectors have a built-in connection pool, which can maintain connections during repeated Function runs, avoiding repeated creation / closure of connections to external systems.

Suppose the user has configured a Connector with ID mysql. The code to obtain the operation object of this Connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different Connectors have different operation methods and parameters. For details, refer to Script Development / Connector Objects with DFF.CONN

3. Environment Variables

Environment Variables can be created in Development / Environment Variables. They are simple Key-Value configuration reading tools provided by DataFlux Func. The ID is specified directly by the user at creation.

Environment Variables are especially suitable for scenarios where the same set of code runs in different environments.

For example, if the system that the Script needs to access has separate testing / production environments, you can set Environment Variables to switch between testing / production environments without changing the code.

Suppose the user has configured an Environment Variable with ID api_endpoint. The code to obtain the value of this Environment Variable is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Func API

Func API can be created in Management / Func API. It is a common way for external parties to call Functions in DataFlux Func. The call process can be synchronous or asynchronous. In synchronous execution, after the Function finishes execution, the result can be returned directly to the caller.

After creating a Func API for a Function, multiple different calling methods are supported.

Func API supports two methods: GET and POST. Parameter passing in both methods supports both the "simplified form" and the "standard form".

In addition, the "simplified" form of the POST method also supports file upload. The following is a list of supported features for various calling methods:

Calling method Pass kwargs parameter kwargs parameter type Pass options File upload Submit Body in any format
GET simplified form Supported String only Not supported Not supported Not supported
GET standard form Supported JSON data types Supported Not supported Not supported
POST simplified form Supported String only Not supported Supported Supported
POST standard form Supported JSON data types Supported Not supported Not supported

Different passing methods may impose restrictions on parameter types

For calling methods where parameters in kwargs can only be strings, type conversion must be performed on the parameters within the Function. In the Func API list, you can click "API Call Example" to view the specific calling method.

Suppose the following Function exists:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

Suppose the "Func API" ID created for this Function is func-api-xxxxx, and the passed parameters are x=100 (integer) and y="hello" (string).

Then, the various calling methods are as follows:

Passing Parameters via the GET Simplified Form

If the Function's parameters are relatively simple, you can use the GET simplified form to pass parameters, making the interface more intuitive.

Because when passing parameters in a URL, the string "100" cannot be distinguished from the integer 100, so when the Function is called, all received parameters are strings. The Function needs to perform type conversion on the parameters by itself.

Text Only
1
GET /api/v1/al/func-api-xxxxx/s?x=100&y=hello

For readability, the example shows the content before URL encoding; actual URL parameters need to be URL-encoded.

Passing Parameters via the GET Standard Form

In some cases, if you cannot send a POST request, you can also call the API using the GET method.

When passing parameters via the GET standard form, simply JSON-serialize the entire kwargs and pass it as a URL parameter. Since the parameters are still sent in JSON format, the original types of the parameters are preserved. The Function does not need to perform type conversion on the parameters again.

As in this example, the x parameter received by the Function is an integer, so no type conversion is needed.

Text Only
1
GET /api/v1/al/func-api-xxxxx?kwargs={"x":100,"y":"hello"}

For readability, the example shows the content before URL encoding; actual URL parameters need to be URL-encoded.

Passing Parameters via the POST Simplified Form

In some cases, if you cannot send an HTTP request with a JSON request body, you can pass parameters in a manner similar to a Form, where each field name is the parameter name.

When submitting data through a Form, the string "100" cannot be distinguished from the integer 100; therefore, when the Function is called, all received parameters are strings, and the Function needs to perform type conversion on the parameters by itself.

Text Only
1
2
3
4
POST /api/v1/al/func-api-xxxxx/s
Content-Type: x-www-form-urlencoded

x=100&y=hello

In addition, passing parameters via the POST simplified form also supports file upload (the parameter/field name must be files), and it needs to be processed using the form-data/multipart method.

An example of the page HTML code is as follows:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Func API address (If this page and DataFlux Func are not on the same domain, you need to write the full http://domain:port/api/v1/al/func-api-xxxxx/s)
        // Note: File uploads must use the simplified form of the Func API
        var API_URL = '/api/v1/al/func-api-xxxxx/s';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, create a FormData object and send it as the request body
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', API_URL);
            xhr.send(data);
        });
    </script>
</html>

Passing Parameters via the POST Standard Form

Passing parameters via the POST standard form is the most common calling method. Since parameters are sent through the request body in JSON format, their original types are preserved. The Function does not need to perform type conversion on the parameters.

As in this example, the x parameter received by the Function is an integer, so no type conversion is needed.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/func-api-xxxxx
Content-Type: application/json

{
  "kwargs": {
    "x": 100,
    "y": "hello"
  }
}

5. Cron Job

Cron Jobs can be created under "Management / Cron Job" to allow DataFlux Func to automatically call Functions on a regular basis.

After a Cron Job is created for a Function, the Function will execute periodically according to the specified Crontab expression, without requiring external invocation.

Because of this, all parameters of the executed Function must already be satisfied, that is:

  1. The Function does not require input parameters
  2. The Function needs input parameters, but all of them are optional
  3. The Function has required parameters, and specific values are configured for them in the Cron Job

Determining Which Execution Feature a Function Belongs to at Runtime

If a Function is configured with both a Cron Job and other execution features, and you want to handle them differently across these execution features, you can check the built-in variable _DFF_CRONTAB to distinguish:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRON_EXPR:
        # Only output logs when running as a Cron Job
        print(f'x + y = {result}')

    return