Loaded malformed certificates and now bsdlib_init returns error on every boot

[jbrzozoski]

We have possibly discovered a serious loophole that can brick a nRF9160 and will require JTAG to recover it.

We were loading a full set of certificates (CA, private, and cert), but due to a bug in our code accidentally provided a single-byte buffer containing just 0x04 to nrf_inbuilt_key_write for each of them.

On the next reboot, and every subsequent reboot, bsdlib_init now returns -1.

There is no apparent way to recover from this in software, since you need to be able to get past bsdlib_init before the application can request that the modem remove the bad certificates.

Leaving the modem turned on for several minutes before rebooting hasn't seemed to resolve anything.

We haven't tried to reproduce this on another nRF9160 yet.

I'm about to try recovering by reloading modem firmware.  If that doesn't work, I also have a special purpose modem firmware that Nordic provided me a while back to recover from bad certificates, and I expect that will do the trick if nothing else does.  I will update here with my results.

Once successful, I may try to reproduce this and see if the same failure occurs again.

[jbrzozoski]

So, the error is less imposing than expected, but still frustrating.

The modem still seems to be functional even though bsd_init() returns an error on boot.  If I ignore that result, all other functionality seems okay as long as I don't try to use the bad credentials.  I can even use good credentials in security slots and use TLS connections.

I don't seem able to erase the bad credentials with nrf_inbuilt_key_delete, as it returns a result code of 5 (NRF_EIO), AT%CMNG is still able to do the job.

And once I erase the bad certificates I am able to reboot and bsd_init will succeed.

I think there needs to be some clarifications around which situation bsd_init can return failures and what functionality may or may not be usable at that point.  I also think the modem firmware might want to self-correct by deleting malformed certificates so that a subsequent reboot will not show the failure, or perhaps succeed at boot time but just refuse to use the bad certificate.

[Håkon Alseth]

Hi,

 

First of all; thank you very much for evaluating the issue and reporting this back to us with this level of detail.

I agree with you, this is frustrating, especially when the error is reported, but voided by the startup routine.  I'll report this internally and have the bsdlib team evaluate what can be done in this scenario.

 

Kind regards,

Håkon