nrf9160 serial interface with HTTPS support for large POST

[Anthony Ambuehl]

I'm evaluating feasibility and developing a Proof-of-Concept for a client using the NRF9160 + NRF52840.   This is a second generation product where the client wants to evaluate the possibility of integrating the connectivity solution of the NRF9160 with a code-base that was developed several years ago on NRF52832.  The existing implementation communicates over BLE to another device which integrates a LTE/Cat-M1 modem.  That modem included support for HTTPS and a filesystem for storing the large data objects.

For the new system, the interface between the NRF52840 and the NRF9160 will be UART / serial.  

The requirements include: 

* securely upload data to a cloud service, existing interface sends large data blocks up to hundreds of kilobytes via HTTPS

* download data objects such as DFU images from an HTTPS service.. these might be several hundred kilobytes as well

* ability to update all components in the system with images sourced from the cloud  (e.g. must be able to update the non-secure application in nrf9160 and the application running in nrf52840)

* all updates must be able to complete atomically without user intervention... in other words all images must be downloaded, authenticated/verified and only then installed.

Currently in the design there is a flash memory connected to the nrf52840, but not the nrf9160.

In the nRFConnect SDK 1.5.1, I see that there are some relevant examples

* https_client - installs a certificate and performs a small https query

* at_client - pass through of AT commands from serial to internal modem.

It seems from https_client example, the management of data sent to the http server is primarily handled by the end-user application code.  If I understand it correctly, we would create the HTTP header and body in application code and then send it to a socket connected to the port. 

I have some initial concerns, that I would  like some help to get oriented:

1) What would be required from AT_Client example interface to initiate and send an HTTP POST request ?   It appears using AT client interface that all of the socket code, tls, keys etc would need to be included into the nrf52840 since they seem to be built on top of the modem AT commands.

2) Is there any sort of timeout on socket used in https request?  For example if the body is several hundred kilobytes and the transfer rate is limited... will the socket close?

3) Is there an https_client library code provided or recommended third party library ? 

3a) I noticed in zephyr\inlcude\net\http_client.h which was not used in https_client example.  Is it possible to Zephyr's http_client code with the TLS secured socket demonstrated in https_client?

4) Which solution would require less work:

    option a) using existing AT_client in nrf9160  (seems hard since zephyr socket / tls code would need to be shifted into non-zephyr code base)

    option b) extending AT_client protocol to provide an command for controlling http client    (is it possible to extend AT_client protocol?)

    option c) creating a custom serial protocol into code based on https client?  (must create robust serial protocol for https)

5) Is there another approach I should consider besides AT_client vs https_client?

6) Is there a protocol or example for updating nrf9160 application via serial interface?

7) Is it recommended to attach a flash chip to nrf9160 for DFU?  For example can the socket rate limit to match the data rate of the serial/uart connection?

8) How can I determine the flash and ram size of the http client example?  ( I want to know how much space is left for buffering and for additional feature development)

[Anthony Ambuehl]

Another question..  will it be possible to send/receive https payload greater than the modem 2K TLS size limit?  

According to this https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/applications/serial_lte_modem/doc/slm_description.html#native-tls-sockets

HTTP client does not support native TLS, and according to 

https://devzone.nordicsemi.com/f/nordic-q-a/73647/mqtt-128kb-packet-size

the modem internal TLS only support 2K TLS fragments.

[Didrik Rokhaug]

Hi,

 

1) What would be required from AT_Client example interface to initiate and send an HTTP POST request ?   It appears using AT client interface that all of the socket code, tls, keys etc would need to be included into the nrf52840 since they seem to be built on top of the modem AT commands.

 The AT client simply forwards AT commands to the modem. To extend it, you would have to first intercept the UARt traffic, see if it is one of your commands or a "normal" AT command, and then handle the command appropriately.

The modem itself also doesn't provide a AT command interface for sending data, only a socket interface (to the application running on the application core).

If you are using the TLS stack in the modem (the normal solution for a "pure" nRF9160 application), the TLS keys must be written to the modem. It is recommended to do this during production with AT commands rather than embedding them in the application FW.

 

2) Is there any sort of timeout on socket used in https request?  For example if the body is several hundred kilobytes and the transfer rate is limited... will the socket close?

 There is a send timeout, which can be adjusted by the application. No, the socket will not close. You will get an ETIMEDOUT error if the socket is not able to allocate memory for the data in the send queue (or ENOMEM if the data doesn't fit).

The modem will keep trying to send the data until the TLS/TCP connection is closed, or the devices gets disconnected from the network.

 

3) Is there an https_client library code provided or recommended third party library ? 

 No recommended third party libraries. Zephyr has a http_client library (give it a TLS socket and it will do HTTPS), but see the answer to 3a.

 

3a) I noticed in zephyr\inlcude\net\http_client.h which was not used in https_client example.  Is it possible to Zephyr's http_client code with the TLS secured socket demonstrated in https_client?

 It does look compatible with the nRF9160. I'll have to ask our developers about why it isn't used. I'll let you know when I get an answer.

 

4) Which solution would require less work:

 a isn't really an option as there are no AT commands in the modem for doing data transfers.

b is possible, though not that different from c. You will have to specify the data transfer commands yourself, and you need to recognize your custom commands compared to the modem's AT commands.

 

5) Is there another approach I should consider besides AT_client vs https_client?

 Yes: Serial LTE Modem

Which already has support for HTTP

 

6) Is there a protocol or example for updating nrf9160 application via serial interface?

 You can use smp_svr to let the application receive the DFU image, or you can use MCUBoot's serial recovery mode (see here to see what commands are supported in serial recovery mode) to let the bootloader receive the image.

On the host side, you can look at mcumgr or central_dfu_smp. The sample uses BLE, and only supports the ECHO comand, but it shouldn't be too hard to replace BLE with UART and add the extra commands.

 

7) Is it recommended to attach a flash chip to nrf9160 for DFU?

 This would save flash on the nRF9160 as the bootloader's secandary partition (where the DFU image is stored) can be moved to the external flash. It will also enable you to do full modem firmware updates if the flash is big enough.

If the flash is shared between the two MCUs, it will also give you a third option for how to do DFU: The host MCU downloads the DFU image, and stores it in the shared flash directly. Then, it triggers a reset on the nRF9160. As the bootloader has been configured to use the external flash for upgrade, it will see the new image, and do the upgrade.

 

For example can the socket rate limit to match the data rate of the serial/uart connection?

 I am not sure what you are asking here. The modem will try to send the data as fast as possible, and the host should not try to send data much faster than the modem can send it (or you need an application in between that buffers the data).

 

8) How can I determine the flash and ram size of the http client example?  ( I want to know how much space is left for buffering and for additional feature development)

 When building your project, you will get a report about flash and RAM usage.

E.g. (from an unrelated simple project, the SLM will use more):

Memory region         Used Size  Region Size  %age Used
           FLASH:       95628 B       992 KB      9.41%
            SRAM:       20648 B     162584 B     12.70%
        IDT_LIST:         136 B         2 KB      6.64%

If you want more detailed information, you can build the ram_report and rom_report targets. When using West, you can specify the build target with the -t flag.

 

Anthony Ambuehl said:

Another question..  will it be possible to send/receive https payload greater than the modem 2K TLS size limit?

 In the TLS stack in the modem? No.

But, as I've explained in the ticket you link to, you can also run the TLS stack on the application core. That will let you configure the buffer size yourself.

 

Anthony Ambuehl said:

HTTP client does not support native TLS,

 I don't know why the HTTP client commands doesn't support native TLS. I'll see if I can find someone who knows and who knows how much work it would be to implement, and come back to you.

Best regards,

Didrik

[Didrik Rokhaug]

Regarding Zephyr's http_client library:

The http_client was introduced after or around the same time that the download_client. It is also more complicated than what the download_client needs, and would have increased the flash size of the download_client unnecessarily.

But, there doesn't seem to be any reasons for why an application which needs the extra features the http_client shouldn't use the http_client.

[Didrik Rokhaug]

I've also talked to one of the developers of the SLM about native TLS support in the HTTP commands

Native TLS was originally added to the SLM to add support for features not supported by the modem, such as TLS server. For the HTTP commands, the modem's feature set is enough in most cases. Adding native TLS to the HTTP commands should not take much work, it just hasn't been requested yet, and we have prioritized other things.

This commit adds native TLS support to the TCP proxy client, and should give you a good indication of what must be done to the HTTP client: https://github.com/lats1980/fw-nrfconnect-nrf/commit/0ec5173a599b18aae16745ebd3b59fd7a0abcd18

Another option is of course to use the TCP or TCP proxy commands in the SLM with native TLS, and implement the HTTP layer on the host MCU.

[Anthony Ambuehl]

Thanks Didrik, this information is very helfpul!  

I'm now planning to use Serial_LTE_Modem (SLM), as you suggested.  If we find its difficult to work within the 2K limit for the uploads, I will explore implementing Native TLS support into http_client.

Native TLS would still face a limit based on the available memory in the nrf9160.. it would not be possible to handle the maximum size DFU images for the NRF52840 even with Native TLS, therefore implementing the download using Content-Range seems mandatory.

I have a few more questions

Does SLM pass through the credential management AT commands (CMNG) to the modem?  I assume that it must since the Link LTE montior can provision credentials, but please confirm my understanding.

Does Native TLS use the same certificates that are stored in the modem?

Is there a reduction in security caused by using Native TLS and reading the certificates into the nonsecure application space ?

[Didrik Rokhaug]

Anthony Ambuehl said:

Does SLM pass through the credential management AT commands (CMNG) to the modem? 

 When not using native TLS, yes.

 

Anthony Ambuehl said:

Does Native TLS use the same certificates that are stored in the modem?

 No. When using native TLS, the certificates are stored in the application core.

It is not possible to read certificates out of the modem (only CA root certificates), so the application core is not able to use the certificates provisioned to the modem.

 

Anthony Ambuehl said:

Is there a reduction in security caused by using Native TLS and reading the certificates into the nonsecure application space ?

 There shouldn't be a reduction in security by using th enative TLS stack in itself. As you are able to tailor the stack more to your needs, you might even be able to improve the security.

However, as the certificates will have to be stored in the application core, they will be easier to get to. That will reduce the security. But, given that the application logic will reside in the host MCU, that risk might be acceptible.