Skip to content

Script Development / Basic Concepts

In DataFlux Func, there exist some concepts unique to DataFlux Func. This document will explain them.

1. Script Sets, Scripts, and Functions

Script sets, scripts, and functions can be created in the "Development / Script Library". These are core concepts of DataFlux Func, and their IDs are directly specified by the user when creating/writing code.

  • A "script set" is a collection of several scripts. The ID is directly specified by the user upon creation and can only contain scripts.
  • A "script" refers to the Python script itself, which necessarily belongs to one script set. The ID is directly specified by the user upon creation.
  • A "function" in DataFlux Func specifically refers to the top-level function decorated with the @DFF.API(...) decorator, which can be called as an entry point by synchronous/asynchronous APIs, scheduled tasks, and other modules.

A script set is not a folder

A script set is similar to a folder, but this "folder" is unrelated to folders typically used in Python coding.

When coding in DataFlux Func, script sets, scripts, and function IDs are heavily involved, and these IDs have tight connections between them.

Relationships Between Script Set, Script, and Function IDs

According to the hierarchical relationship between script sets, scripts, and functions, the ID of a lower-level concept must include the ID of the higher-level concept.

Assuming there exists a script set with the ID demo, then all scripts belonging to this script set must start with demo__ (double underscores).

Furthermore, assuming this script set contains a script with the ID demo__test, which includes a function def hello(...), then the ID of this function would be demo__test.hello.

The following table shows examples of IDs:

Concept Example ID
Script Set demo
Script demo__test
Function demo__test.hello

Mutual References in Coding

In DataFlux Func scripts, you can reference another script to achieve code reuse.

Assume there exists a script demo__script_a, which contains a function func_a(). To reference this function in script demo__script_b, you can use the following method:

demo__script_a
1
2
def func_a():
    pass
demo__script_b
1
2
3
4
import demo__script_b

def test():
    return demo__script_b.func_a()

Python's as statement can also be used:

demo__script_b
1
2
3
4
import demo__script_b as b

def test():
    return b.func_a()

You can also use the from ... import statement to import only the required function:

demo__script_b
1
2
3
4
from demo__script_b import func_a

def test():
    return func_a()

For references between scripts within the same script set, the script set ID can be omitted, using the abbreviated form that starts with __ (double underscores):

demo__script_b
1
2
3
4
from __script_b import func_a

def test():
    return func_a()

Use Abbreviated Forms Whenever Possible

Mutual references within a script set should preferably use the abbreviated form (i.e., ignoring the script set ID and starting with __).

This way, if the entire script set is cloned and the script set ID changes, the code in the newly cloned script set can still correctly reference the scripts within the same script set.

2. Connectors

Connectors can be created in "Development / Connectors", and they are tools provided by DataFlux Func for connecting to external systems. The ID is directly specified by the user upon creation.

In reality, writing Python code in DataFlux Func is not much different from writing standard Python. Developers can completely ignore connectors and connect to external systems directly in their code.

However, for some external systems with connection pool concepts, connectors internally implement connection pooling, maintaining connections during repeated function executions to avoid repeatedly creating/closing connections to external systems.

Assume the user has already configured a connector with the ID mysql. The code to obtain the operation object for this connector is as follows:

Python
1
mysql = DFF.CONN('mysql')

Different connectors have different operation methods and parameters. For more details, please refer to Script Development / Connector Object DFF.CONN

3. Environment Variables

Environment variables can be created in "Development / Environment Variables", and they are simple Key-Value configuration retrieval tools provided by DataFlux Func. The ID is directly specified by the user upon creation.

Environment variables are especially suitable for scenarios where the same set of code runs in different environments.

If a script needs to access a system that distinguishes between testing and production environments, environment variables can be set to switch between testing and production environments without changing the code.

Assume the user has already configured an environment variable with the ID api_endpoint. The code to retrieve the value of this environment variable is as follows:

Python
1
api_endpoint = DFF.ENV('api_endpoint')

4. Synchronous API (Old Version: Authorized Link)

Synchronous APIs can be created in "Manage / Synchronous API". It is a common way for external systems to call functions in DataFlux Func. The calling process executes synchronously, and after the function execution completes, the result can be directly returned to the caller.

After creating a synchronous API for a function, it supports multiple different calling methods.

Synchronous APIs support GET and POST methods. Both methods support passing parameters in "simplified form" and "standard form."

Additionally, the "simplified" form of POST supports file uploads. Below is the feature support list for various calling methods:

Calling Method Passing kwargs Parameters kwargs Parameter Type Passing options File Upload Submit Arbitrary Format Body
Available since 1.6.9
GET Simplified Form Supported Strings Only Not Supported Not Supported Not Supported
GET Standard Form Supported JSON Data Types Supported Not Supported Not Supported
POST Simplified Form Supported Strings Only Not Supported Supported Supported
POST Standard Form Supported JSON Data Types Supported Not Supported Not Supported

Different passing methods may impose restrictions on parameter types

For calling methods where only strings can be passed as kwargs parameters, type conversion is necessary within the function. In the Synchronous API list, you can click "API Call Example" to view specific calling methods.

Assume there is the following function:

Python
1
2
3
@DFF.API('My Function')
def my_func(x, y):
    pass

The synchronous API ID created for this function is auln-xxxxx, with the parameters being x=100 (integer), y="hello" (string).

Thus, the various calling methods are as follows:

Passing Parameters via GET Simplified Form

If the function parameters are relatively simple, the GET simplified form can be used to pass parameters, making the interface more intuitive.

Since URL parameters cannot distinguish between the string "100" and the integer 100, the parameters received by the function when called will all be strings. The function needs to manually convert the parameter types.

Text Only
1
GET /api/v1/al/auln-xxxxx/simplified?x=100&y=hello

For ease of reading, the example shows content before URLEncode; actual URL parameters need to be URLEncoded

Passing Parameters via GET Standard Form

In some cases, if POST requests cannot be sent, the GET method can also be used to call the interface.

When passing parameters via the GET standard form, the entire kwargs is serialized into JSON and passed as a URL parameter. Since the parameters are actually sent in JSON format, their original types are retained, and the function does not need to perform type conversion.

In this example, the x parameter received by the function will be an integer, requiring no type conversion.

Text Only
1
GET /api/v1/al/auln-xxxxx?kwargs={"x":100,"y":"hello"}

For ease of reading, the example shows content before URLEncode; actual URL parameters need to be URLEncoded

Passing Parameters via POST Simplified Form

In some cases, if HTTP requests with JSON bodies cannot be sent, parameters can be passed in a form similar to a Form submission, where field names are parameter names.

Since Form submissions cannot distinguish between the string "100" and the integer 100, the parameters received by the function when called will all be strings, and the function needs to manually convert the parameter types.

Text Only
1
2
3
4
POST /api/v1/al/auln-xxxxx/simplified
Content-Type: x-www-form-urlencoded

x=100&y=hello

Additionally, the POST simplified form supports file uploads (parameter/field name must be files), which requires processing using form-data/multipart.

An HTML page code example is as follows:

HTML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<html>
    <body>
        <h1>File Upload</h1>
        <input id="file" type="file" name="files" required />
        <input id="submit" type="submit" value="Upload"/>
    </body>
    <script>
        // Synchronous API address (if this page and DataFlux Func are not on the same domain, write the full http://domain:port/api/v1/al/auln-xxxxx/simplified
        // Note: Uploading files must use the simplified form synchronous API
        var AUTH_LINK_URL = '/api/v1/al/auln-xxxxx/simplified';

        document.querySelector('#submit').addEventListener('click', function(event) {
            // After clicking the upload button, generate a FormData object and send it as the request body
            var data = new FormData();
            data.append('x', '100');
            data.append('y', 'hello');
            data.append('files', document.querySelector('#file').files[0]);

            var xhr = new XMLHttpRequest();
            xhr.open('POST', AUTH_LINK_URL);
            xhr.send(data);
        });
    </script>
</html>

Passing Parameters via POST Standard Form

The POST standard form is the most common calling method.

Since parameters are sent through the request body in JSON format, their original types are retained, and the function does not need to perform type conversion.

In this example, the x parameter received by the function will be an integer, requiring no type conversion.

Text Only
1
2
3
4
5
6
7
8
9
POST /api/v1/al/auln-xxxxx
Content-Type: application/json

{
    "kwargs": {
        "x": 100,
        "y": "hello"
    }
}

5. Asynchronous API (Old Version: Batch Processing)

Asynchronous APIs can be created in "Manage / Asynchronous API", providing another way for external systems to call functions in DataFlux Func.

The difference from "Synchronous API" lies in that after asynchronous APIs are called, they immediately return, executing processing asynchronously in the background without requiring the caller to wait for results. Apart from this, the calling method is the same as "Synchronous API".

Asynchronous APIs provide longer function execution times (not API response times), making them very suitable for performing time-consuming asynchronous processing.

6. Scheduled Tasks (Old Version: Automatic Trigger Configuration)

Scheduled tasks can be created in "Manage / Scheduled Tasks", allowing DataFlux Func to automatically call functions at regular intervals.

After creating a scheduled task for a function, the function will execute according to the specified Crontab expression without requiring external calls.

Because of this, all parameters for the executed function must be satisfied, meaning:

  1. The function does not require input parameters.
  2. The function requires input parameters, but they are optional.
  3. The function requires mandatory parameters, which are configured with specific values in the scheduled task.

Differentiating Function Execution Contexts

If a function is configured with both "Scheduled Task" and other execution functionalities and needs to handle different execution contexts separately, you can differentiate using the built-in variable _DFF_CRONTAB:

Python
1
2
3
4
5
6
7
8
9
@DFF.API('My Function')
def my_func(x, y):
    result = x + y

    if _DFF_CRONTAB:
        # Outputs logs only during scheduled tasks
        print(f'x + y = {result}')

    return

7. API Authentication

Added in version 1.3.2

Asynchronous APIs can be created in "Manage / Asynchronous API", providing another way for external systems to call functions in DataFlux Func.

For "Synchronous / Asynchronous APIs" generated as HTTP APIs, additional interface authentication can be added.

Currently supported interface authentications include:

Authentication Type Description
Fixed Field Verifies that the Header, Query, or Body of the request must contain a field with a specific value
HTTP Basic Standard HTTP Basic authentication (pops up a login box in browsers)
HTTP Digest Standard HTTP Digest authentication (pops up a login box in browsers)
Authentication Function Specifies a custom-written function as the authentication function

Users can add authentication configurations in "Manage / API Authentication" and then specify the added authentication configurations in "Synchronous / Asynchronous API Configurations".

If high security is required, HTTPS must be used to access the interface

Fixed Field Authentication

Fixed field authentication is the simplest authentication method, where the client and DataFlux Func agree to include a specific field and its value in a certain part of the request (Header, Query, or Body). This content is attached in every call to complete authentication.

Assume that each request must include the header x-auth-token="my-auth-token". You can complete authentication by calling it as follows:

Text Only
1
2
GET /api/v1/al/auln-xxxxx
x-auth-token: my-auth-token

When configuring multiple fixed field authentications, matching any one is considered successful

For fields in Query and Body used for authentication, the system automatically deletes them after authentication and does not pass them to the function

HTTP Basic / HTTP Digest

Browser-supported authentication methods.

Interfaces using these authentication methods will prompt the browser to display a username/password dialog when accessed directly via the browser address bar.

If you need to access programmatically, please refer to the following code:

Python
1
2
3
4
5
6
7
8
import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

# HTTP Basic authentication
resp = requests.get(url_1, auth=HTTPBasicAuth('user', 'password'))

# HTTP Digest authentication
resp = requests.get(url_2, auth=HTTPDigestAuth('user', 'password'))

Authentication Function

If the interface authentication method is complex or special (such as needing to integrate with business systems), you can choose to write your own function for authentication.

The authentication function must satisfy "having and only having one req parameter as the request" and return True or False to indicate success or failure of authentication.

The parameter req is a dict, with the following structure:

Field Name Type Description
method str Request method (uppercase)
Such as: "GET", "POST"
originalUrl str Original URL of the request. Includes the part after ?
Such as: /api/v1/al/auln-xxxxx?q=1
url str Request URL. Does not include the part after ?
Such as: /api/v1/al/auln-xxxxx
headers dict Request Header, field names are all lowercase
query dict Request Query, field names and field values are all strings
body dict Request Body
hostname str Hostname accessed by the request. Does not include the port number
Such as: example.com
ip str Client IP
Note: This field requires correct configuration of Nginx, Alibaba Cloud SLB, etc.
ips list List of client IP and all intermediate proxy server IP addresses
Note: This field requires correct configuration of Nginx, Alibaba Cloud SLB, etc.
ips[#] str Intermediate proxy server IP
xhr bool Whether it is an AJAX request
Example
1
2
3
@DFF.API('Authentication Function')
def my_auth_func(req):
    return req['headers']['x-auth-token'] == 'my-auth-token'