Memfault certificate challenges on nRF9160

[Dutchiots]

Dear support,

We have challenges with the Memfault implementation on the nRF9160, we also have questions regarding Memfault on the nRF52833. We are experiencing unknown setbacks with the SSL certificates and authentication process; this is related to Memfault not running correctly on the latest NCS (version (2.4.2 which uses memfault-firmware-sdk release 0.43.3). Can you please have a look at this? 

The nRF9160 modem firmware version we are using is mfw_nrf9160_1.3.4 and the hardware version is nRF9160 SICA B1A.

At the end of our working hours today(19-10-2023) we were able to perform FOTA over the MemFault server over LTE with few adjustments to the code in the memfault-sdk-library.

In the function starting MemFault FOTA over LTE the TLS certificate for HTTPS connection to MemFault FOTA server is chosen between 4 different  root Certificate Authorities (CA). One by one certificate candidates are verified to match the MemFault FOTA certificate by trying to start FOTA download using the NCS FOTA download library. If the NCS FOTA download start function returns zero the certificate used to trigger the start is deemed to be valid. 

However the problem is that the NCS FOTA download start function doesn't seem to check the validity of the certificate so the first certificate tried is always deemed the right one.
This proved to be the cause of the problem because the first certificate tried is Amazon Root CA 1 which is not valid to obtain connection to the MemFault FOTA server. Looking at the HTTPS certificate MemFault FOTA server on the browser it seemed to be DigiCert Global Root G2 which is the last on the list of certificate to be tried for FOTA connection.
After changing the memfault-sdk-library code to first use certificate DigiCert Global Root G2 to start the FOTA connection the FOTA connection was ssuccessfullyestablished. After that we were able to perform a successful FOTA.

However during the FOTA download the connection needed to be reconnected multiple times because of an error (EMSGSIZE) indicating the TLS fragment sent by the MemFault FOTA server is larger than what the modem is able to store in its internal buffers. After a bit of research it seems that modem can only support TLS fragments up to 2 kB in size and not larger.
The solution proposed is to decrease the maximum FOTA server TLS fragment size to work around this constraint.
There doesn't seem to be a way to configure the FOTA server on our own so we would need your help with this.
Do you know of any other possible solutions to this problem?

We are currently using nRF Connect SDK release 2.4.2 which uses memfault-firmware-sdk release 0.43.3.
Looking at the releases of the memfault-firmware-sdk from release 1.3.0 the code for checking the certificate validity hasn't changed, however the first certificate to be tried for FOTA server connection is set to be DigiCert Global Root G2 for which the FOTA connection would work.
Although this is not an optimal solution it should work for our use case.

The plan is to test MemFault FOTA tomorrow with the latest memfault-firmware-sdk release 1.3.5 and see how it behaves.
Can you confirm it is okay to use the latest memfault-firmware-sdk release even though the NCS we are using recommends to use the release 0.43.3?

The best option would be if you could supply us with MemFault FOTA over LTE sample code so we can tailor our application code to it and we are sure everything is configured correctly. Is there an existing MemFault FOTA over LTE sample code for NCS?

Our project also includes the nRF52833 MCU with BLE support on which we would also need to implement FOTA.
If I understand it correctly MemFault also supports FOTA over BLE, but there seems to be lacking documentation on this.
Can you confirm this and supply us with the working FOTA over BLE sample if it exists.
If there is no sample, can you recommend NCS sample and change in configuration to achieve this.

Thank you!

[Raoul]

Hi,

I apologise for taking longer than usual! For the past few days we've been short on people who might handle this sort of case. I'll try to find someone to answer you as soon as possible.

Best regards,

Raoul

[Raoul]

Hi again,

I have now heard back from a developer working on our Memfault integration.

I'll try to address your issues point by point:

However the problem is that the NCS FOTA download start function doesn't seem to check the validity of the certificate so the first certificate tried is always deemed the right one.
This proved to be the cause of the problem because the first certificate tried is Amazon Root CA 1 which is not valid to obtain connection to the MemFault FOTA server. Looking at the HTTPS certificate MemFault FOTA server on the browser it seemed to be DigiCert Global Root G2 which is the last on the list of certificate to be tried for FOTA connection.
After changing the memfault-sdk-library code to first use certificate DigiCert Global Root G2 to start the FOTA connection the FOTA connection was ssuccessfullyestablished. After that we were able to perform a successful FOTA.

After testing for themselves, the developer says:

I think we need more details from the customer on what exactly is going wrong here, and preferably a modem trace. When testing, the TLS connection is successful with AWS Root CA 1.

The TLS stack always checks the validity of the server certificate. The connection would fail if an unrecognized root CA was used.

So if you are able to reproduce the issue, a modem trace would be great.

Regarding:

However during the FOTA download the connection needed to be reconnected multiple times because of an error (EMSGSIZE) indicating the TLS fragment sent by the MemFault FOTA server is larger than what the modem is able to store in its internal buffers. After a bit of research it seems that modem can only support TLS fragments up to 2 kB in size and not larger.
The solution proposed is to decrease the maximum FOTA server TLS fragment size to work around this constraint.
There doesn't seem to be a way to configure the FOTA server on our own so we would need your help with this.
Do you know of any other possible solutions to this problem?

The dev says:

The only way to increase the TLS fragment size is to run the TLS stack on the application core. That's possible, but takes up a lot of flash and RAM, and certificates will have to be stored and handled in the application code. I would not recommend that if they can get away with the method described below:

If they're using Memfault server for downloads, I don't think they can configure it in anyway. I'm assuming that the download is done over HTTP, though. Then they can use range requests to limit the size of the TLS fragments. It's not controlling the TLS fragment size directly, but rather indirectly by limiting the HTTP content length in each response. The downside is that it doesn't take into account the HTTP header size, which is unknown, so careful tuning of the requested content size is required. Note that this feature is supported already in the download client and can be set using Kconfig options. 

 

The plan is to test MemFault FOTA tomorrow with the latest memfault-firmware-sdk release 1.3.5 and see how it behaves.
Can you confirm it is okay to use the latest memfault-firmware-sdk release even though the NCS we are using recommends to use the release 0.43.3?

According to the dev, you may cherry-pick the commits in: https://github.com/nrfconnect/sdk-nrf/pull/12254. These should be fine with NCS v2.4.2.

Can you confirm this and supply us with the working FOTA over BLE sample if it exists.

No, we don't have any code showing how to use Memfault's own FOTA solution.

One important thing to note is that FOTA using Memfault is not a feature that Nordic officially supports. It's a feature in Memfault's SDK that they have implemented. Nordic has their own FOTA solution, using nRF Cloud. Still, if many customers have issues with integrating Memfault FOTA, I think our developers might have interest in knowing, and taking a closer look.

But I do recommend checking if Memfault provides some support channel for their FOTA solution.

Best regards,

Raoul