MQTT support streaming large payload

Hi,

nRF9160 examples in NCS, the MQTT publish payload now with a pre allocated buffer,  is it possible to support streaming payload to save memory usage for large payload publish.

Q1: does current NCS release (modem firmware 1.2.3, or 1.3.0) support publish >20KBytes payload in 1 single message?

Q2: is it possible to publish >20K bytes with data streaming to socket instead with pre allocated buffer in RAM?

  • Q1: does current NCS release (modem firmware 1.2.3, or 1.3.0) support publish >20KBytes payload in 1 single message?

     Are you using TLS?

    The nRF9160 only supports TLS fragments of ~2kB. So if you are using TLS, the answer is no.

    If you are using UDP, I believe you are limited by the network MTU size. You can check the MTU size with the +CGCONTRDP AT command.

    If you are using TCP, I would argue that the question isn't really valid. It would still techincally take more than one packet, but the other end will see the data as a stream of bytes regardless of how many TCP packets are actually sent. It should therefore not matter if the nRF9160 sends it as one big buffer, or a series of smaller ones.

     

    Q2: is it possible to publish >20K bytes with data streaming to socket instead with pre allocated buffer in RAM?

     Again, if you are using TLS or UDP, then no; but if you are using TCP, then yes. That is what the protocol was designed for.

     

    the MQTT publish payload now with a pre allocated buffer,  is it possible to support streaming payload to save memory usage for large payload publish.

     MQTT is based on TCP, so in theory, it is supported. However, the MQTT client that is implemented in NCS only supports pre-allocated buffers.

    It is technically possible to first send the MQTT header, followed by the payload split up in multiple calls to send(). The broker should then see it as just one big MQTT packet. However, this is not supported by the current API of the MQTT client in NCS.

    Best regards,

    Didrik

  • Thanks,

    One more questions on the large data upload options.

    Do you know any limitations with HTTP or HTTPS to stream large payloads (over 20K) to server with current NCS libraries/examples.

  • Looking at Zephyr's HTTP client library, it seems it is able to stream large payloads by using a callback function to request data.

    You can see the API in action in Zephyr's http_client sample https://developer.nordicsemi.com/nRF_Connect_SDK/doc/1.5.0/zephyr/samples/net/sockets/http_client/README.html

    Note that the sample has not been tested on the nRF9160, and might need some modifications to run on the nRF9160.

Related