Problem with nRF-connect-SDK http_client_req() function

[royalbee]

Hi,


I open ssl socket to the server (without certificate verification),
and then use an http_client_req() call, to perform the actual http request (with additional optional headers).

That works fine for both http GET and POST requests, that have a short response from the server.

However, when the server's response is a bit larger (say 2.5KB not including response headers and cookie), I always get "Socket was closed by remote" error (-104).
I tried increasing recv_buf_len from 1KB to 8KB, but it didn't help.

When debugging http_client_req(), I see that the call to socket's recv() just returns 0 bytes, which is considered socket close by remote.
On the server logs I see that HTTP request, but no special problems appear on aws elb logs.

Performing the same request on my PC (using Postman for example), works just fine, so the problem is not on the server side (at least not with standard http/tcp settings)/

thanks in advance..

[Albrecht Markus Schellenberger]

Thanks a lot for your feedback!

royalbee said:

Maybe you can try to check this problem on your side as well? just connect to any server that supports ssl and returns response that is longer than 2.5K (I actually think that 2K will also be enough).

Our modem does not supported TLS fragments larger than 2kB. You have to send smaller responses in this case, or download the data bit for bit. Could you try this and check if that makes any differences?

Regards,

Markus

[royalbee]

Hi,

Can you be more specific what you mean by "smaller responses" ?
we have API and its response size depends on the data. and It can be larger than 2K.

How do you suggest to limit its size on the server side? you mean that the server will send the response in fragments that are smaller than 2K? (not sure how complex it is to implement something like that, since I never tried..)

Note that it is not possible for the modem (i.e. the http-client) to decide on the fragment size that the server sends (Unless the server supports the content-range header, which only few servers support, and it also complicates the implementation on the modem side).

When I asked the server to receive 1KB (Without content-range), which is smaller than 2KB, I got the same error.
It seems like the server tries send the fragment it wants to send in this case, and the modem side should divide it into smaller fragments (that fits into the its max receive-buffer size).
That division logic seems to work fine on the modem with none-secured sockets, but in TLS implementation there seems to be a bug.


In any case, that is a very serious limitation on your TLS implementation (and as you know, today most http communications are done using TLS..). 
So are you planning to fix this (serious) problem any time soon ?

[kobyatom]

Hello Markus.

I'm Ran's manager. Please allow me to add my inquiry.

I'm quite concerned, and would like to know if the nRF9160 can fit our needs.

As I understand, the modem doesn't support receiving data larger than 2KB over SSL sockets.

Is that so?

We will not be able to get our system to work with this unexpected limitation.

Is there a proven patch we can add to the modem FW to work around this limit?

Is there a date on your road map to remove this limitation altogether?

Thank you

   Koby Fruchtnis

   R&D Manager, Atomation

By the way, the limitation of 2KB secure socket buffer size was introduced starting from 1.1.0.
It did not exist prior to that version.

*** mfw_nrf9160_1.1.0
*********************
...

*** Limitations
***************
- TLS/DTLS
    ...
    - 2kB secure socket buffer size.