nRF9160 issue in receiving MQTT publish messages larger than 2048 bytes over TLS

[Ravikiran]

Hi,

My Setup details:

nRF9160 DK Board: 0.8.3
nRF Connect SDK tag: 1.3.0
OS: Windows 10 Pro
LTE Network: NB-IoT
Modem Firmware: mfw_nrf9160_1.2.0
Cloud Platform: AWS IoT Core
CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN: 5120

Use case:

As part of device provisioning process we need to receive a MQTT message from AWS which contains certificate details, this message can't be split as we don't have control over it. Size of such messages are around 3KB.

Issue:

Whenever the incoming PUBLISH message size from AWS is more than 2048 bytes, MQTT connection is dropped. After some debugging the function "mqtt_read_and_parse_fixed_header" in MQTT library (mqtt_rx.c source code) returns -ENOTCONN.

In the release notes of modem firmware, a limitation is mentioned about TLS feature: "2kB secure socket buffer size."
However after experimenting with incoming message sizes I noticed publish messages up to 2235 bytes are received successfully. Few bytes over it and the connection drops. In general with TCP protocol large packets can be received with small buffers with appropriate MTU/Window size. Is this issue linked to the above TLS limitation or something else?

Reproducing the issue is simple, use any cloud example to connect to AWS over TLS, subscribe to any topic and from AWS send 2KB+ sized messages.

Thanks,
Ravikiran

[Øyvind]

Hello Ravikiran, 

Receiving a packet of 3kB is not supported over TLS as you have found, and this is due to what you found: 

In the release notes of modem firmware, a limitation is mentioned about TLS feature: "2kB secure socket buffer size."

You will need to find a way to downsize the MQTT message. What is this MQTT message with certificates? Do you mean the TLS handshake?

 Kind regards,
Øyvind

[Ravikiran]

Hi Øyvind,

AWS IoT Core has a service called Fleet Provisioning which can be used to automate device provisioning in field, that is, creating a THING and certificate-key pair for the device in AWS IoT. In this flow a device sends a CreateKeysAndCertificate MQTT message to AWS and in response it receives Certificate pem and Private key in one message. This method is simplest, requires least intervention during product deployment and highly scalable but results in large MQTT message size. There are alternative methods we can explore like CreateCertificateFromCsr or something custom but it will take more steps.

On the modem firmware limitations is it possible to trade off between number of open TLS sockets and socket buffer size. Is there some configuration to reduce the simultaneous socket connections to two and increase buffer size per socket?

Regards,
Ravikiran

[Dominic]

Just commenting on this because I ran into the same issue. The lambda which intercepts the CreateKeysAndCertificate response ("$aws/certificates/create/json/accepted") must then republish the certificate and key separately to two new topics.

There is some trickery going on in the AWS IoT backend because the policy document states that a device can subscribe to any subtopic of "$aws/certificates/create/*" but you will find that subscribing to a subtopic other than /accepted and /rejected causes the client to be disconnected from the broker. Also, the lambda cannot publish to a reserved topic denoted by the '$' prefix

The solution is to create two new topics, eg: "fleetprovision/certificate" and "fleetprovision/key" which the lambda can publish to. The final step is updating the fleet certificate policy document so that the device can also subscribe and receive messages from these topics using its fleet provisioning certificate.

[Dominic]

Well unfortunately I have hit a dead end as the AWS IoT broker only sends the CreateKeysAndCertificate response to the client on same same connection as the originating request, meaning it can't be intercepted by the rules engine nor any other clients.

A potential hack is to use an EC2 instance as a middle-man which can both submit a certificate request and receive a response, then encode that into a smaller payload and pass it onto the nrf9160 but this likely has security implications.

[Markus Tacker (he/him)]

Hei Dominic,

for this specific message it is a good security measure on AWS that it is only sent to the requesting client. 

However you can implement this using a lambda (or EC2 instance) which sends the new certificate to the device: the device would e.g. send a blank MQTT message to /<deviceId>/newcert/request and a lambda listening to this message creates a new certificate and sends it via/<deviceId>/newcert/created and the device can retrieve it there (again split up in multiple messages because of the size).

This would be as secure resp. insecure as using the AWS IoT MQTT topics directly (it is still sent over the wire).

[Dominic]

Thanks Markus, this is exactly how I ended up implementing it

[Dominic]

Following up because I finally got this to work, here are some tips to save the next person some time:

1. The provision-template request needs to be published and received by the same client which made the certificateRequest (EC2 instance or lambda depending on your setup). Trying to do so from the nrf9160 results in a 'InvalidCertificateId' error.

2. Using Zephyr's json parser library, newline characters are parsed as literal '\' and 'n' characters. I wrote a function which uses strtok to replace these characters with actual 0x0A newline chars.

static int copy_certificate(char *dest, char *src, int max_len)
{
    int copy_len = 0;
    char backslash[] = "\\";
    char newline = '\n';

    char* tok = strtok(src, backslash);

    while(true){
        copy_len += strlen(tok);
        if(copy_len >= max_len){
            LOG_ERR("Certificate dest not large enough");
            return -ENOMEM;
        }
        strcat(dest, tok);

        tok = strtok(NULL, backslash);
        if(tok == NULL) break;

        *tok = newline;
    }
    return 0;
}

3. modem_key_mgmt_delete and modem_key_mgmt_write only work when the LTE modem is off however they will still return a zero response code when the modem is on/connected, even though modem_key_mgmt_exists confirms that the certificates are not written to the modem.

So overall a slightly painful process of trial and error but saves a step at the factory and you don't need to expose the modem's UART to write the certificates.