nRF9160 issue in receiving MQTT publish messages larger than 2048 bytes over TLS

[Ravikiran]

Hi,

My Setup details:

nRF9160 DK Board: 0.8.3
nRF Connect SDK tag: 1.3.0
OS: Windows 10 Pro
LTE Network: NB-IoT
Modem Firmware: mfw_nrf9160_1.2.0
Cloud Platform: AWS IoT Core
CONFIG_AWS_IOT_MQTT_PAYLOAD_BUFFER_LEN: 5120

Use case:

As part of device provisioning process we need to receive a MQTT message from AWS which contains certificate details, this message can't be split as we don't have control over it. Size of such messages are around 3KB.

Issue:

Whenever the incoming PUBLISH message size from AWS is more than 2048 bytes, MQTT connection is dropped. After some debugging the function "mqtt_read_and_parse_fixed_header" in MQTT library (mqtt_rx.c source code) returns -ENOTCONN.

In the release notes of modem firmware, a limitation is mentioned about TLS feature: "2kB secure socket buffer size."
However after experimenting with incoming message sizes I noticed publish messages up to 2235 bytes are received successfully. Few bytes over it and the connection drops. In general with TCP protocol large packets can be received with small buffers with appropriate MTU/Window size. Is this issue linked to the above TLS limitation or something else?

Reproducing the issue is simple, use any cloud example to connect to AWS over TLS, subscribe to any topic and from AWS send 2KB+ sized messages.

Thanks,
Ravikiran

[Øyvind]

Hello Ravikiran, 

Receiving a packet of 3kB is not supported over TLS as you have found, and this is due to what you found: 

In the release notes of modem firmware, a limitation is mentioned about TLS feature: "2kB secure socket buffer size."

You will need to find a way to downsize the MQTT message. What is this MQTT message with certificates? Do you mean the TLS handshake?

 Kind regards,
Øyvind

[Ravikiran]

Hi Øyvind,

AWS IoT Core has a service called Fleet Provisioning which can be used to automate device provisioning in field, that is, creating a THING and certificate-key pair for the device in AWS IoT. In this flow a device sends a CreateKeysAndCertificate MQTT message to AWS and in response it receives Certificate pem and Private key in one message. This method is simplest, requires least intervention during product deployment and highly scalable but results in large MQTT message size. There are alternative methods we can explore like CreateCertificateFromCsr or something custom but it will take more steps.

On the modem firmware limitations is it possible to trade off between number of open TLS sockets and socket buffer size. Is there some configuration to reduce the simultaneous socket connections to two and increase buffer size per socket?

Regards,
Ravikiran

[Markus Tacker (he/him)]

Hei Dominic,

for this specific message it is a good security measure on AWS that it is only sent to the requesting client. 

However you can implement this using a lambda (or EC2 instance) which sends the new certificate to the device: the device would e.g. send a blank MQTT message to /<deviceId>/newcert/request and a lambda listening to this message creates a new certificate and sends it via/<deviceId>/newcert/created and the device can retrieve it there (again split up in multiple messages because of the size).

This would be as secure resp. insecure as using the AWS IoT MQTT topics directly (it is still sent over the wire).

[Dominic]

Thanks Markus, this is exactly how I ended up implementing it

[Dominic]

Following up because I finally got this to work, here are some tips to save the next person some time:

1. The provision-template request needs to be published and received by the same client which made the certificateRequest (EC2 instance or lambda depending on your setup). Trying to do so from the nrf9160 results in a 'InvalidCertificateId' error.

2. Using Zephyr's json parser library, newline characters are parsed as literal '\' and 'n' characters. I wrote a function which uses strtok to replace these characters with actual 0x0A newline chars.

static int copy_certificate(char *dest, char *src, int max_len)
{
    int copy_len = 0;
    char backslash[] = "\\";
    char newline = '\n';

    char* tok = strtok(src, backslash);

    while(true){
        copy_len += strlen(tok);
        if(copy_len >= max_len){
            LOG_ERR("Certificate dest not large enough");
            return -ENOMEM;
        }
        strcat(dest, tok);

        tok = strtok(NULL, backslash);
        if(tok == NULL) break;

        *tok = newline;
    }
    return 0;
}

3. modem_key_mgmt_delete and modem_key_mgmt_write only work when the LTE modem is off however they will still return a zero response code when the modem is on/connected, even though modem_key_mgmt_exists confirms that the certificates are not written to the modem.

So overall a slightly painful process of trial and error but saves a step at the factory and you don't need to expose the modem's UART to write the certificates.

[Rob Nuttall]

Were you able to implement this using just lambda's or did you require a EC2 instance. Obviously as you mentioned before you can not publish to a reserved topic in a lambda, so how did you request a cert to be created from inside the lambda?

[Dominic]

Hi Rob, a workaround for publishing and subscribing to reserved topics is to use a standalone MQTT client such as paho-mqtt inside the lambda function.

The rough steps are:

1. Device publishes to user-defined certificate request topic (Make sure your fleet provisioning certificate only allows publishing to this topic)

2. IoT core invokes lambda which initializes a MQTT client and connects to the broker.

3. MQTT client running inside lambda publishes to `$aws/certificates/create/json` and waits for a response on `$aws/certificates/create/json/accepted`

4. (Optional) MQTT client publishes to `$aws/provisioning-templates/<Template Name>/provision/json` in order to provision device. Wait for message on `$aws/provisioning-templates/<Template Name>/provision/json/accepted` to confirm device is in fact genuine. (You can define a provision-check lambda to confirm if the serial code exists in your DB or similar)

5. MQTT client publishes certificate and key in separate messages to device and lambda execution finishes.

The biggest security vulnerability is that anyone with access to a fleet provisioning certificate could subscribe to the endpoint where the certificate is published in step 5, so it is crucial that you configure the fleet provisioning certificate's policy to only allow subscribing to `certificateResponse/<serial_code>` or similar and not `certificateResponse/#`. This way someone would have to guess the genuine device's serial code in order to subscribe to the certificate response.

Drop here!

[Rob Nuttall]

Thanks for your reply Dominic. Using a standalone MQTT client makes sense. I ended up getting this working and also implemented the whitelist of devices in the provisioning hook. I'm still unsure if this is secure enough for the application the device is to be used in however.

I am going to stick with this for testing and development but I am contemplating implementing JITR and creating our own Certificate Authority instead. This also allows having shorter certificate expiry dates as when AWS issues a certificate I have not found a way of reducing it from expiring in 2050. You could implement your own method to store the date the certificate was issued and revoke it after a certain time period has expired. But this seems more work than it needs to be