Skip to content

Script Development / Basic Concepts

In DataFlux Func, there are some concepts unique to DataFlux Func. This document will explain them.

1. Script Sets, Scripts, and Functions

Script sets, scripts, and functions can be created in "Development / Script Library". They are core concepts of DataFlux Func, and their IDs are specified directly by users during creation / coding.

  • A "Script Set" is a collection of several scripts. The ID is specified directly by the user at creation, and it can only contain scripts.
  • A "Script" is a Python script itself, which must belong to a script set. The ID is specified directly by the user at creation.
  • A "Function" in DataFlux Func specifically refers to the top-level function decorated with @DFF.API(...). It can be used as the entry function for Function APIs, cron jobs, etc.

Script Set is not a folder

A script set is similar to a folder, but this "folder" is unrelated to folders in general Python coding.

When coding in DataFlux Func, IDs of script sets, scripts, and functions are heavily involved, and there is a close relationship between these IDs.

Relationship Between Script Set, Script, and Function IDs

According to the hierarchical relationship of script sets, scripts, and functions, the ID of a lower-level concept always includes the ID of the upper-level concept.

Suppose there is a script set with ID demo. Then all scripts belonging to this script set must start with demo__ (double underscore).

Further suppose under this script set there is a script with ID demo__test, which contains a function def hello(...). Then the ID of this function is demo__test.hello.

Example ID table:

Concept ID Example
Script Set demo
Script demo__test
Function demo__test.hello

Mutual References in Coding

In DataFlux Func scripts, it is allowed to reference another script for code reuse.

Assume there is a script demo__script_a, which contains a function func_a(). Then, to reference this function in script demo__script_b, you can use the following methods:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement is also supported:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

Or use from ... import to import only the needed function:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between scripts belonging to the same script set, you can omit the script set ID and use the shorthand form starting with __ (double underscore):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Use Shorthand Form Whenever Possible

Inter-references within a script set should use the shorthand form as much as possible (i.e., omitting the script set ID and starting with __).

In this way, when the entire script set is cloned and the script set ID changes, the code in the new cloned script set can still correctly reference scripts within its own script set.

2. Connectors

Connectors can be created in "Development / Connectors". They are tools provided by DataFlux Func for connecting to external systems. The ID is specified directly by the user at creation.

In fact, writing Python code in DataFlux Func is not much different from original Python. Developers can completely ignore connectors and connect to external systems directly in their code.

However, for external systems that have a connection pool concept, connectors have built-in connection pools, which can maintain connections during repeated function runs, avoiding repeated creation / closure of connections to external systems.

Assuming a connector with ID mysql has been configured, the code to obtain the operation object of this connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different connectors have different operation methods and parameters. For details, refer to Script Development / Connector Object DFF.CONN

3. Environment Variables

Environment variables can be created in "Development / Environment Variables". They are simple Key-Value configuration reading tools provided by DataFlux Func. The ID is specified directly by the user at creation.

Environment variables are especially suitable for scenarios where the same code runs in different environments.

For example, if the system that the script needs to access distinguishes between test / production environments, you can set environment variables to switch between test / production environments without changing the code.

Assuming an environment variable with ID api_endpoint has been configured, the code to obtain its value is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Function API

Function APIs can be created in "Management / Function APIs". They are a common way for external systems to call functions in DataFlux Func. The call can be synchronous or asynchronous. When executed synchronously, the result can be returned directly to the caller after the function completes.

After creating a Function API for a function, multiple different calling methods are supported.

Function APIs support GET and POST methods. Both methods support "Simplified Form" and "Standard Form" for parameter passing.

In addition, the "Simplified Form" of POST also supports file upload. The following is a list of supported features for each calling method:

Calling Method Passing kwargs Parameters kwargs Parameter Type Passing options File Upload Submitting Arbitrary Format Body
GET Simplified Form Supported Strings only Not supported Not supported Not supported
GET Standard Form Supported Data types in JSON Supported Not supported Not supported
POST Simplified Form Supported Strings only Not supported Supported Supported
POST Standard Form Supported Data types in JSON Supported Not supported Not supported

Different passing methods may impose restrictions on parameter types

For calling methods where kwargs parameters can only be strings, type conversion of parameters is required within the function. In the Function API list, you can click "API Call Examples" to view specific calling methods.

Assume the following function exists:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

Assume the "Function API" created for this function has ID func-api-xxxxx, and the parameters passed are x=100 (integer), y="hello" (string).

Then the different calling methods are as follows:

GET Simplified Form Parameter Passing

If the function's parameters are simple, you can use the GET simplified form to pass parameters, making the interface more intuitive.

Since it is not possible to distinguish between the string "100" and the integer 100 when passing parameters in a URL, the function will receive all parameters as strings. The function needs to perform type conversion on the parameters itself.

Text Only
1
GET /api/v1/al/func-api-xxxxx/s?x=100&y=hello

For readability, the example shows content before URLEncode. Actual URL parameters need to be URLEncoded

GET Standard Form Parameter Passing

In some cases, if a POST request cannot be sent, the interface can also be called using GET.

When using GET standard form parameter passing, serialize the entire kwargs as JSON and pass it as a URL parameter. Since the parameters are actually sent in JSON format, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

In this example, the x parameter received by the function is an integer, no type conversion is needed.

Text Only
1
GET /api/v1/al/func-api-xxxxx?kwargs={"x":100,"y":"hello"}

For readability, the example shows content before URLEncode. Actual URL parameters need to be URLEncoded

POST Simplified Form Parameter Passing

In some cases, if an HTTP request with a JSON body cannot be sent, parameters can be passed in a way similar to a Form, where each field name is the parameter name.

Since Form submissions cannot distinguish between the string "100" and the integer 100, the function will receive all parameters as strings, and the function needs to perform type conversion on the parameters itself.

Text Only
1
2
3
4
POST /api/v1/al/func-api-xxxxx/s
Content-Type: x-www-form-urlencoded

x=100&y=hello

Additionally, the POST simplified form also supports file upload (parameter/field name must be files), which requires using form-data/multipart.

Example HTML page code:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Function API URL (if this page is not on the same domain as DataFlux Func, the full http://domain:port/api/v1/al/func-api-xxxxx/s needs to be written)
        // Note: File upload must use the simplified form Function API
        var API_URL = '/api/v1/al/func-api-xxxxx/s';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, generate a FormData object and send it as the request body
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', API_URL);
            xhr.send(data);
        });
    </script>
</html>

POST Standard Form Parameter Passing

POST standard form parameter passing is the most common calling method. Since the parameters are sent as JSON in the request body, the original types of the parameters are preserved. The function does not need to perform type conversion on the parameters.

In this example, the x parameter received by the function is an integer, no type conversion is needed.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/func-api-xxxxx
Content-Type: application/json

{
  "kwargs": {
    "x": 100,
    "y": "hello"
  }
}

5. Cron Job

Cron jobs can be created in "Management / Cron Jobs". They are used to automatically call functions periodically in DataFlux Func.

After creating a cron job for a function, the function will be executed periodically according to the specified Crontab expression, without requiring external invocation.

Therefore, all parameters of the executed function must be satisfied, i.e.:

  1. The function requires no input parameters
  2. The function requires input parameters, but they are all optional
  3. The function requires mandatory parameters, and specific values are configured for them in the cron job

Distinguishing the Execution Context of a Function

If a function is configured with both a "Cron Job" and other execution features, and you want to handle them differently based on the execution context, you can check the built-in variable _DFF_CRONTAB:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRON_EXPR:
        # Log only during cron job execution
        print(f'x + y = {result}')

    return