Does modem TLS support fragmentation to allow HTTPS requests greater than 2K secure socket buffer limit?

[Anthony Ambuehl]

When I asked earlier, it was not possible to send HTTPS payloads greater than 2K due to the TLS secure socket buffer size limit.  I see in the most recent modem firmware release the size limit is still the same. 

It seems that this wouldn't be a problem if TLS fragmentation was supported and the modem negotiated a reduced TLS fragment size. 

Has TLS fragmentation with a negotiated size been investigated?  

Is it possible to send an HTTPS POST request with a payload greater than 2KB?

Is it possible to receive an HTTPS POST response with a payload greater than 2KB?

Is it possible to have TLS fragmentation working with the Native TLS approach?

Here are some related tickets that remain unresolved after many months:

https://devzone.nordicsemi.com/f/nordic-q-a/74251/nrf9160-serial-interface-with-https-support-for-large-post

https://devzone.nordicsemi.com/f/nordic-q-a/77351/problem-with-nrf-connect-sdk-http_client_req-function

[Albrecht Markus Schellenberger]

Hello Anthony,

please find enclosed the answers to your questions:

Anthony Ambuehl said:

In our system we use HTTP POST to upload and download data.  We need both uploads that are greater than 2k and downloads that are greater than 2k. 

For example for our DFU procedure, we send a POST where the body contains information about the device identity, hardware version and current firmware version and the server may respond with the data of a firmware image.  

Another example for our log upload procedure, we send a POST with meta-data about the device which itself may be almost 1K due in inefficient ASCII/JSON encoding, and the device logs.  Each post will be validated and then deposited into a cloud-based storage system.  From my understanding, HTTP POST is not idempotent so each successive request may generate a result on the server.

The above mentioned use cases are ok by HTTPC service in SLM.

Anthony Ambuehl said:

How does the Content-Range field work for a HTTP POST? Is the request only processed when the Content-Range is full received?     

The request is processed when SLM receives AT#XHTTPCREQ. If <payload_length> is greater than 0, SLM will send the HTTP request is sent in 2 step:

1. Send the Request-Line, Request Headers, and the CRLF after Request Header to the server. The <method>,<resource>,<header> in AT#XHTTPCREQ are placed in the request according to the HTTP request format.
2. Enter datamode and wait for incoming UART data. The external MCU then feeds SLM with payload data until <payload_length> bytes are sent.

So, datamode happens between #XHTTPCREQ: 1 and #XHTTPCREQ: 0

The length of data is defined in

Content-Length: 224

SLM HTTP client quits datamode when this number of data has been sent.

Datamode accept one-time transmit of 4096-256 bytes, which then send out the data to LTE by loop and multi-packets if needed.

If more than 4096-256 are send at once, SLM will drop the packet and stop receiving from UART.

Anthony Ambuehl said:

Does the server know when to abandon an incoming partial and incomplete POST request?

It all depends on the server implementation. Server may disconnect client if the request header, message payload are not in correct format. Server may also timeout client after some time.

Anthony Ambuehl said:

Does it require sending the request multiple times, each time with a different Range in the header?  

Single request or multiple request are both doable. In a stable LTE-M network, a single request with large payload can be sent to reduce some header overhead. In a NB-IoT network it would be better to send smaller message in multiple times.

Anthony Ambuehl said:

How would the http client sending the POST request know the size of the data that would be returned?

After sending whole request message (including payload), HTTP client keep doing blocking socket receive(recv) until the connection is closed by server. The received HTTP response fragment will be notified via URC #XHTTPCRSP. Therefore HTTP client does not know the size of the data, only server knows. When the server sent all the data, server will close the connection. 

Anthony Ambuehl said:

How does the client know if the data previously received becomes invalid and the procedure needs to be restarted from the beginning?

It is the responsibility of external MCU to parse the status code (EX: 200 OK, 405 Method Not Allowed) of HTTP response.

Anthony Ambuehl said:

Would the SLM be able to handle a streamed http resposne by  having the server send Transfer Encoding: chunked   ?

SLM will handle the chunked response. The chunk header will be stripped, only the payload will notified via URC #XHTTPCRSP.

I hope this will help you!

Regards,

Markus