Increasing HTTP requests size using nRF7002

RE: Increasing HTTP requests size using nRF7002

Charlie — Wed, 24 Jan 2024 14:20:06 GMT

Hi Alink,

The cause of overflow is due to the following ram size limitation, I did a quick test with 16000 and it handles buffer[8192]. You can try more to optimize this value.

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE

Best regards,

Charlie

RE: Increasing HTTP requests size using nRF7002

alink — Wed, 24 Jan 2024 09:12:45 GMT

Hello Charlie,

Yes have been trying various configurations (from usage profiles) but still the limit is 2KB.

That's why if you can also update the buffer [4096][8192] on your side.

Reason is if we have larger buffer size, we can reduce the PUT requests.

Thank you.

RE: Increasing HTTP requests size using nRF7002

Charlie — Tue, 23 Jan 2024 08:53:09 GMT

Hi Alink,

It is no surprise there is a limitation that is decided by firmware setting, RAM limitation or HTTP server settings(which refuse to build HTTP connection with the client) as we discussed before.

I suggest you check them one by one if you want to extend this limitation for your application, especially the HTTP server side.

Best regards,

Charlie

RE: Increasing HTTP requests size using nRF7002

alink — Mon, 22 Jan 2024 01:44:19 GMT

Hello,

Yes, the code modification was correct, to 1976 buffer based on exercise. Indeed PUT request is successful for this.

But if it is greater than 2KB, say buffer is 4096, it will be error 128.

Please try on your side if buffer is 4KB or 8KB.

Thank you.

RE: Increasing HTTP requests size using nRF7002

Charlie — Fri, 19 Jan 2024 11:46:02 GMT

For #2 PUT, I still do not fully understand your attention.

I assume you are trying to fragment the 70KB file manually into 1976B to send through PUT requests. If this is correct, I made the following changes in the original exercise.

static int client_http_put(void)
{
...
    char buffer[1976] = {0};
...
    req.payload_len = sizeof(buffer);
...
LOG_INF("HTTP PUT request: %s,size:%d", buffer,req.payload_len);
...
}

Push button 1 can continuously send PUT requests to the server without limitation.

...
[00:00:18.677,001] <inf> Lesson5_Exercise1: Connected to server
[00:00:18.677,124] <inf> Lesson5_Exercise1: HTTP PUT request: 0,size:1976
[00:00:18.959,503] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:21.923,004] <inf> Lesson5_Exercise1: Connected to server
[00:00:21.923,126] <inf> Lesson5_Exercise1: HTTP PUT request: 1,size:1976
[00:00:22.236,328] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:24.265,258] <inf> Lesson5_Exercise1: Connected to server
[00:00:24.265,380] <inf> Lesson5_Exercise1: HTTP PUT request: 2,size:1976
[00:00:24.389,160] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:26.263,488] <inf> Lesson5_Exercise1: Connected to server
[00:00:26.263,610] <inf> Lesson5_Exercise1: HTTP PUT request: 3,size:1976
[00:00:26.379,333] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:28.804,107] <inf> Lesson5_Exercise1: Connected to server
[00:00:28.804,229] <inf> Lesson5_Exercise1: HTTP PUT request: 4,size:1976
[00:00:28.920,196] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:30.798,248] <inf> Lesson5_Exercise1: Connected to server
[00:00:30.798,370] <inf> Lesson5_Exercise1: HTTP PUT request: 5,size:1976
[00:00:30.919,982] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:32.581,878] <inf> Lesson5_Exercise1: Connected to server
[00:00:32.582,000] <inf> Lesson5_Exercise1: HTTP PUT request: 6,size:1976
[00:00:32.888,214] <inf> Lesson5_Exercise1: Response status: Accepted
[00:00:36.195,220] <inf> Lesson5_Exercise1: Connected to server
[00:00:36.195,343] <inf> Lesson5_Exercise1: HTTP PUT request: 7,size:1976
[00:00:36.306,335] <inf> Lesson5_Exercise1: Response status: Accepted
...

I wonder if there could be something wrong with your modification. Try to use a delayable work to trigger a put request instead of calling the client_http_put() function inside the callback.

Best regards,

Charlie

RE: Increasing HTTP requests size using nRF7002

alink — Thu, 18 Jan 2024 00:50:08 GMT

Hello,

Thank you for informative reply.

For #1 GET, this is mainly to test HTTP similar to 'download_client'. The limit of 2KB is on the 9160.

Understood on the limit on 8KB on web services and the socket can fragment the data automatically.

For #2 PUT, we use the wififund : https://github.com/NordicDeveloperAcademy/wifi-fund/blob/main/lesson5/wififund_less5_exer1_solution/src/main.c

Specifically the client_http_put() we increase the buffer payload to a 2048 character array buffer[1976], then modified response_cb() to

close(sock);

k_msleep(15);

count the request size sent up to 70KB (around 35 times)

ifsize > 70kb, call client_http_put()

You can try just by increasing buffer client_http_put() in req.payload

Below is the test on our side:

This is requested 3 times when request size is reached

Tried 1976 char array (72 bytes header) and it can send 3x,

increasing it further and we receive error 128

Thank you.

RE: Increasing HTTP requests size using nRF7002

Charlie — Wed, 17 Jan 2024 15:07:33 GMT

Hi Alink,

[quote user=""]

Our project requires to use the board for HTTP GET and PUT requests that will have size of:

GET: up to 60MB
PUT: 3 requests at around 70KB each

[/quote]

60MB HTTP GET request is not common even in the web browser world. web services - Maximum length of HTTP GET request - Stack Overflow. Maybe you should first consider if you use them correctly in your application according to GET and PUT definitions on HTTP - Wikipedia, POST seems to be the correct request type to upload a big file.

If you do need such kind of usage, I suggest you consider other solutions(PC, Raspberry Pi kind of devices) for your project. Linux host + nRF7002 could be a solution but it is still under development.

The MCU host+ nRF7002 solutions are targeted for lightweight IoT applications with very limited resources. nRF5340 only has 512KB RAM, which is far less resource than you need to send a GET request. It can provide limited computing power considering your data size.

[quote user=""]Based on sample and the lesson, seems that there is a limit of 2KB buffer size not only in 'download_client' but also when getting receive response buffer in 'https_client'[/quote]

This is a limitation of the nRF9160 modem, its secure socket buffer size is 2kB so the incoming TLS packet is no bigger than 2KB. You can choose Zephyr native TLS socket instead of the one supported by the modem. See the following ticket for reference. (+) MQTT 128kB packet size - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com)

nRF7002DK has no such limitation since it uses Zephyr TLS socket.

[quote user=""]1) In GET, I think a fixed 2KB buffer and size are passed to http_client_req(). Is there a way to set data larger than 2KB?[/quote]

This is not a limitation for the data sent out, since the socket can fragment the data automatically. If the device receives data from a server with TLS packet size bigger than 2KB, then it has problem.

[quote user=""]2) One PUT payload also has 2KB limit in HTTP, increasing it returns error -128. I don't think it is a size that can be sent by PUT due to the packet length limit.
　Is there a limit that can be specified with HTTP PUT? Or is there a upload counterpart for 'download_client'?[/quote]

Error -128 means the socket is not connected according to C:\NCS\toolchains\c57af46cb7\opt\zephyr-sdk\arm-zephyr-eabi\arm-zephyr-eabi\sys-include\sys\errno.h.

Could you share more details about your HTTP PUT request tests? I can try to repeat the issue to check the reason.

Best regards,

Charlie