Problem with nRF-connect-SDK http_client_req() function

[royalbee]

Hi,


I open ssl socket to the server (without certificate verification),
and then use an http_client_req() call, to perform the actual http request (with additional optional headers).

That works fine for both http GET and POST requests, that have a short response from the server.

However, when the server's response is a bit larger (say 2.5KB not including response headers and cookie), I always get "Socket was closed by remote" error (-104).
I tried increasing recv_buf_len from 1KB to 8KB, but it didn't help.

When debugging http_client_req(), I see that the call to socket's recv() just returns 0 bytes, which is considered socket close by remote.
On the server logs I see that HTTP request, but no special problems appear on aws elb logs.

Performing the same request on my PC (using Postman for example), works just fine, so the problem is not on the server side (at least not with standard http/tcp settings)/

thanks in advance..

[royalbee]

Hi,

Can you be more specific what you mean by "smaller responses" ?
we have API and its response size depends on the data. and It can be larger than 2K.

How do you suggest to limit its size on the server side? you mean that the server will send the response in fragments that are smaller than 2K? (not sure how complex it is to implement something like that, since I never tried..)

Note that it is not possible for the modem (i.e. the http-client) to decide on the fragment size that the server sends (Unless the server supports the content-range header, which only few servers support, and it also complicates the implementation on the modem side).

When I asked the server to receive 1KB (Without content-range), which is smaller than 2KB, I got the same error.
It seems like the server tries send the fragment it wants to send in this case, and the modem side should divide it into smaller fragments (that fits into the its max receive-buffer size).
That division logic seems to work fine on the modem with none-secured sockets, but in TLS implementation there seems to be a bug.


In any case, that is a very serious limitation on your TLS implementation (and as you know, today most http communications are done using TLS..). 
So are you planning to fix this (serious) problem any time soon ?

[kobyatom]

Hello Markus.

I'm Ran's manager. Please allow me to add my inquiry.

I'm quite concerned, and would like to know if the nRF9160 can fit our needs.

As I understand, the modem doesn't support receiving data larger than 2KB over SSL sockets.

Is that so?

We will not be able to get our system to work with this unexpected limitation.

Is there a proven patch we can add to the modem FW to work around this limit?

Is there a date on your road map to remove this limitation altogether?

Thank you

   Koby Fruchtnis

   R&D Manager, Atomation

By the way, the limitation of 2KB secure socket buffer size was introduced starting from 1.1.0.
It did not exist prior to that version.

*** mfw_nrf9160_1.1.0
*********************
...

*** Limitations
***************
- TLS/DTLS
    ...
    - 2kB secure socket buffer size.

[Albrecht Markus Schellenberger]

Hello Koby,

thanks a lot for your feedback!

kobyatom said:

As I understand, the modem doesn't support receiving data larger than 2KB over SSL sockets.

Is that so?

 Yes, that is correct.

kobyatom said:

By the way, the limitation of 2KB secure socket buffer size was introduced starting from 1.1.0.
It did not exist prior to that version.

The limitation has always been there, but the release notes have been updated with this information during the release of modem firmware 1.1.0.

kobyatom said:

Is there a proven patch we can add to the modem FW to work around this limit?

No. Since this is a hardware limitation, there is no software patch available.

However, it is possible to offload the secure socket to the application core. The chapter Native TLS sockets of the Serial LTE modem application guide introduces this.

kobyatom said:

Is there a date on your road map to remove this limitation altogether?

We are continuously working on improving our products.

Regarding Rans question:

royalbee said:

Can you be more specific what you mean by "smaller responses" ?

With TLS, the modem has to receive the whole package to be able to decode it. Thus, the nRF9160 will not be able to receive a TLS package larger than 2kB. However, HTTP has a feature called content-range, where you can tell the server which part of the data you would like to receive in order to split the content into several TLS packages. But the server has to support this feature otherwise this won’t work.

You can have a look at how this is implemented in our download_client library.

I hope my answers could clarify the situation.

Best regards,

Markus

[kobyatom]

Thank you for your reply Markus.

Some more questions:

Looking at the paragraph on Native TLS Sockets, it states that HTTP Client is not supported.

Does that mean that HTTPS GET and POST not be supported if I use TLS Sockets?

Regarding "We are continuously working on improving our products.", and since it will require a HW change to support TLS fragments > 2KB, is there a known plan to update the nRF9160 HW in that respect?

Regarding "content-range", will it also work when sending data > 2KB?

Kind regards,

Koby

[Albrecht Markus Schellenberger]

Hello Koby,

kobyatom said:

Does that mean that HTTPS GET and POST not be supported if I use TLS Sockets?

The Serial LTE modem application does not come with native TLS support for FTP and HTTP. But it is possible to add it to this or any other application.

As a reference, you can have a look at this example. The commit adds native TLS support to TCP proxy command within the Serial LTE modem application. Hence, similar changes could be made for HTTP commands to add native TLS support or the host could run the HTTP stack and the client can manually send HTTP via the TCP commands.

kobyatom said:

Regarding "content-range", will it also work when sending data > 2KB?

 Yes, this should be possible as well.

kobyatom said:

Regarding "We are continuously working on improving our products.", and since it will require a HW change to support TLS fragments > 2KB, is there a known plan to update the nRF9160 HW in that respect?

I’m afraid I can’t give you any more information than I already gave you. You should discuss this with our local sales department. If desired, I can provide you with more details.

Regards,

Markus