Does modem TLS support fragmentation to allow HTTPS requests greater than 2K secure socket buffer limit?

[Anthony Ambuehl]

When I asked earlier, it was not possible to send HTTPS payloads greater than 2K due to the TLS secure socket buffer size limit.  I see in the most recent modem firmware release the size limit is still the same. 

It seems that this wouldn't be a problem if TLS fragmentation was supported and the modem negotiated a reduced TLS fragment size. 

Has TLS fragmentation with a negotiated size been investigated?  

Is it possible to send an HTTPS POST request with a payload greater than 2KB?

Is it possible to receive an HTTPS POST response with a payload greater than 2KB?

Is it possible to have TLS fragmentation working with the Native TLS approach?

Here are some related tickets that remain unresolved after many months:

https://devzone.nordicsemi.com/f/nordic-q-a/74251/nrf9160-serial-interface-with-https-support-for-large-post

https://devzone.nordicsemi.com/f/nordic-q-a/77351/problem-with-nrf-connect-sdk-http_client_req-function

[Albrecht Markus Schellenberger]

Hello Anthony,

Has TLS fragmentation with a negotiated size been investigated? 

Is it possible to have TLS fragmentation working with the Native TLS approach?

Unfortunately, it is not supported in any official branch of mbedTLS. 

Is it possible to send an HTTPS POST request with a payload greater than 2KB?

Is it possible to receive an HTTPS POST response with a payload greater than 2KB?

No, it is not possible. And just for clarification: since the header is included in the 2kB limit, the payload will be even smaller.

However, as I also mentioned in case 273122, you could try content-range as an alternative, if the server supports it. This http feature enables the possibility to split the payload into several <2kB packages.

My apologies that I can’t give you any more pleasant news than this, but I hope my answer will still help you!

Regards,

Markus

[Anthony Ambuehl]

Hi Markus,

  I have another question regarding the content-range method.

In our system we use HTTP POST to upload and download data.  We need both uploads that are greater than 2k and downloads that are greater than 2k. 

For example for our DFU procedure, we send a POST where the body contains information about the device identity, hardware version and current firmware version and the server may respond with the data of a firmware image.  

Another example for our log upload procedure, we send a POST with meta-data about the device which itself may be almost 1K due in inefficient ASCII/JSON encoding, and the device logs.  Each post will be validated and then deposited into a cloud-based storage system.  From my understanding, HTTP POST is not idempotent so each successive request may generate a result on the server.

How does the Content-Range field work for a HTTP POST? Is the request only processed when the Content-Range is full received?     

Does the server know when to abandon an incoming partial and incomplete POST request?

Does it require sending the request multiple times, each time with a different Range in the header?  

How would the http client sending the POST request know the size of the data that would be returned?

How does the client know if the data previously received becomes invalid and the procedure needs to be restarted from the beginning?

Would the SLM be able to handle a streamed http resposne by  having the server send Transfer Encoding: chunked   ?

[Albrecht Markus Schellenberger]

Hello Anthony,

Since I do not have that much insight information on content-range, I need to check this with our development team. I will come back to you as soon as possible.

Regards,

Markus

[Albrecht Markus Schellenberger]

Hello Anthony,

please find enclosed the answers to your questions:

Anthony Ambuehl said:

In our system we use HTTP POST to upload and download data.  We need both uploads that are greater than 2k and downloads that are greater than 2k. 

For example for our DFU procedure, we send a POST where the body contains information about the device identity, hardware version and current firmware version and the server may respond with the data of a firmware image.  

Another example for our log upload procedure, we send a POST with meta-data about the device which itself may be almost 1K due in inefficient ASCII/JSON encoding, and the device logs.  Each post will be validated and then deposited into a cloud-based storage system.  From my understanding, HTTP POST is not idempotent so each successive request may generate a result on the server.

The above mentioned use cases are ok by HTTPC service in SLM.

Anthony Ambuehl said:

How does the Content-Range field work for a HTTP POST? Is the request only processed when the Content-Range is full received?     

The request is processed when SLM receives AT#XHTTPCREQ. If <payload_length> is greater than 0, SLM will send the HTTP request is sent in 2 step:

1. Send the Request-Line, Request Headers, and the CRLF after Request Header to the server. The <method>,<resource>,<header> in AT#XHTTPCREQ are placed in the request according to the HTTP request format.
2. Enter datamode and wait for incoming UART data. The external MCU then feeds SLM with payload data until <payload_length> bytes are sent.

So, datamode happens between #XHTTPCREQ: 1 and #XHTTPCREQ: 0

The length of data is defined in

Content-Length: 224

SLM HTTP client quits datamode when this number of data has been sent.

Datamode accept one-time transmit of 4096-256 bytes, which then send out the data to LTE by loop and multi-packets if needed.

If more than 4096-256 are send at once, SLM will drop the packet and stop receiving from UART.

Anthony Ambuehl said:

Does the server know when to abandon an incoming partial and incomplete POST request?

It all depends on the server implementation. Server may disconnect client if the request header, message payload are not in correct format. Server may also timeout client after some time.

Anthony Ambuehl said:

Does it require sending the request multiple times, each time with a different Range in the header?  

Single request or multiple request are both doable. In a stable LTE-M network, a single request with large payload can be sent to reduce some header overhead. In a NB-IoT network it would be better to send smaller message in multiple times.

Anthony Ambuehl said:

How would the http client sending the POST request know the size of the data that would be returned?

After sending whole request message (including payload), HTTP client keep doing blocking socket receive(recv) until the connection is closed by server. The received HTTP response fragment will be notified via URC #XHTTPCRSP. Therefore HTTP client does not know the size of the data, only server knows. When the server sent all the data, server will close the connection. 

Anthony Ambuehl said:

How does the client know if the data previously received becomes invalid and the procedure needs to be restarted from the beginning?

It is the responsibility of external MCU to parse the status code (EX: 200 OK, 405 Method Not Allowed) of HTTP response.

Anthony Ambuehl said:

Would the SLM be able to handle a streamed http resposne by  having the server send Transfer Encoding: chunked   ?

SLM will handle the chunked response. The chunk header will be stripped, only the payload will notified via URC #XHTTPCRSP.

I hope this will help you!

Regards,

Markus