Loaded malformed certificates and now bsdlib_init returns error on every boot

[jbrzozoski]

We have possibly discovered a serious loophole that can brick a nRF9160 and will require JTAG to recover it.

We were loading a full set of certificates (CA, private, and cert), but due to a bug in our code accidentally provided a single-byte buffer containing just 0x04 to nrf_inbuilt_key_write for each of them.

On the next reboot, and every subsequent reboot, bsdlib_init now returns -1.

There is no apparent way to recover from this in software, since you need to be able to get past bsdlib_init before the application can request that the modem remove the bad certificates.

Leaving the modem turned on for several minutes before rebooting hasn't seemed to resolve anything.

We haven't tried to reproduce this on another nRF9160 yet.

I'm about to try recovering by reloading modem firmware.  If that doesn't work, I also have a special purpose modem firmware that Nordic provided me a while back to recover from bad certificates, and I expect that will do the trick if nothing else does.  I will update here with my results.

Once successful, I may try to reproduce this and see if the same failure occurs again.

[jbrzozoski]

I can't recover my modem.

I was able to reload mfwnrf9160101.zip just fine, but that didn't resolve the issue.

I then tried to load the special-built mfw_nrf9160_corrupt_fix.zip provided to me a while back, but that failed to load. I'm guessing that failed since it looks like a downgrade.

Can I please get a corrupt_fix that will load on top of 1.0.1?

Oh yeah, the initial issue was while running modem firmware v1.0.1.

[jbrzozoski]

For some reason I don't understand yet, the at_client sample was able to get past bsd initialization okay, and I was able to remove the bad certificate store using AT%CMNG commands, and then reload my application and it is working again.

Device un-bricked!

I'm not sure what's different about how at_client calls bsd_init, and this still required JTAG on the board to recover.

I'm continuing to investigate...

[jbrzozoski]

Yeah, I can definitely re-create this by loading the malformed buffer.  (I thought I couldn't briefly, but I had made a typo in my debug code.)

[jbrzozoski]

at_client boots since it uses CONFIG_BSD_LIBRARY_SYS_INIT=y, and that ignores the result code from bsd_init:

	err = bsd_init();

	if (IS_ENABLED(CONFIG_BSD_LIBRARY_SYS_INIT)) {
		/* bsd_init() returns values from a different namespace
		 * than Zephyr's. Make sure to return something in Zephyr's
		 * namespace, in this case 0, when called during SYS_INIT.
		 * Non-zero values in SYS_INIT are currently ignored.
		 */
		return 0;
	}

	return err;

So ... it appears the modem is actually running "well enough" even when bsd_init returns an error.

[jbrzozoski]

So, the error is less imposing than expected, but still frustrating.

The modem still seems to be functional even though bsd_init() returns an error on boot.  If I ignore that result, all other functionality seems okay as long as I don't try to use the bad credentials.  I can even use good credentials in security slots and use TLS connections.

I don't seem able to erase the bad credentials with nrf_inbuilt_key_delete, as it returns a result code of 5 (NRF_EIO), AT%CMNG is still able to do the job.

And once I erase the bad certificates I am able to reboot and bsd_init will succeed.

I think there needs to be some clarifications around which situation bsd_init can return failures and what functionality may or may not be usable at that point.  I also think the modem firmware might want to self-correct by deleting malformed certificates so that a subsequent reboot will not show the failure, or perhaps succeed at boot time but just refuse to use the bad certificate.