Skip to content

Script Development / Responding with Large Data DFF.RESP_LARGE_DATA

Added in version 1.3.0

When returning a large amount of content (MB level or higher), directly using the return method may significantly degrade performance due to internal system communication processing. In this case, you can use DFF.RESP_LARGE_DATA(...) to improve performance.

Parameter Type Required / Default Value Description
data str/dict/list Required Specifies the data to be returned
content_type str None Specifies the response body type, such as json, text, html, etc.

When using this method, ensure that the resource catalog configuration and mounting are correct, so all Web servers and work units can properly access the same shared directory.

Common usage examples are as follows:

Python
1
2
3
4
5
@DFF.API('Case 1')
def case_1():
    data = {} # Large data (MB level or higher)

    return DFF.RESP_LARGE_DATA(data)

Principle Explanation

The underlying layer of DataFlux Func is composed of a Web server and work units combined via Redis acting as a message queue. Data returned directly through return is serialized and sent into the message queue before being returned by the Web server to the caller.

Due to JSON serialization/deserialization, enqueue/dequeue operations on Redis, and internal network communications, performance will degrade when a single JSON data size becomes too large.

This function essentially performs the following operations at the underlying layer: 1. Saves the data to be returned as a file in the download directory of the resource catalog 2. Responds to the request as a "file download" (i.e., the aforementioned DFF.RESP_FILE) 3. The Web server directly reads the file saved in step 1 from the resource catalog and returns it to the client

By using this "detour" method, the processing within the system's internal communications is lightened to improve performance.

Performance Comparison

Below is a performance comparison when returning approximately 3.5MB of JSON data:

  • When returning JSON directly via return data, it takes 18 seconds
Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ time wget http://172.16.35.143:8089/api/v1/al/auln-Ljo3y8HMUl91
--2021-09-16 22:40:09--  http://172.16.35.143:8089/api/v1/al/auln-Ljo3y8HMUl91
Connecting to 172.16.35.143:8089... Connected.
HTTP request sent, awaiting response ... 200 OK
Length: 3363192 (3.2M) [application/json]
Saving to: “auln-Ljo3y8HMUl91”

auln-Ljo3y8HMUl91            100%[=============================================>]   3.21M  --.-KB/s  Time 0.06s

2021-09-16 22:40:27 (50.4 MB/s) - Saved “auln-Ljo3y8HMUl91” [3363192/3363192])

wget http://172.16.35.143:8089/api/v1/al/auln-Ljo3y8HMUl91  0.00s user 0.02s system 0% cpu 18.321 total
  • When returning JSON via return DFF.RESP_LARGE_DATA(data), it only takes less than 1 second
Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ time wget http://172.16.35.143:8089/api/v1/al/auln-HPrfGRKIhYET
--2021-09-16 22:40:50--  http://172.16.35.143:8089/api/v1/al/auln-HPrfGRKIhYET
Connecting to 172.16.35.143:8089... Connected.
HTTP request sent, awaiting response ... 200 OK
Length: 3687382 (3.5M) [application/json]
Saving to: “auln-HPrfGRKIhYET”

auln-HPrfGRKIhYET            100%[=============================================>]   3.52M  --.-KB/s  Time 0.02s

2021-09-16 22:40:50 (183 MB/s) - Saved “auln-HPrfGRKIhYET” [3687382/3687382])

wget http://172.16.35.143:8089/api/v1/al/auln-HPrfGRKIhYET  0.00s user 0.02s system 12% cpu 0.174 total