Loaded malformed certificates and now bsdlib_init returns error on every boot

[jbrzozoski]

We have possibly discovered a serious loophole that can brick a nRF9160 and will require JTAG to recover it.

We were loading a full set of certificates (CA, private, and cert), but due to a bug in our code accidentally provided a single-byte buffer containing just 0x04 to nrf_inbuilt_key_write for each of them.

On the next reboot, and every subsequent reboot, bsdlib_init now returns -1.

There is no apparent way to recover from this in software, since you need to be able to get past bsdlib_init before the application can request that the modem remove the bad certificates.

Leaving the modem turned on for several minutes before rebooting hasn't seemed to resolve anything.

We haven't tried to reproduce this on another nRF9160 yet.

I'm about to try recovering by reloading modem firmware.  If that doesn't work, I also have a special purpose modem firmware that Nordic provided me a while back to recover from bad certificates, and I expect that will do the trick if nothing else does.  I will update here with my results.

Once successful, I may try to reproduce this and see if the same failure occurs again.

[jbrzozoski]

For some reason I don't understand yet, the at_client sample was able to get past bsd initialization okay, and I was able to remove the bad certificate store using AT%CMNG commands, and then reload my application and it is working again.

Device un-bricked!

I'm not sure what's different about how at_client calls bsd_init, and this still required JTAG on the board to recover.

I'm continuing to investigate...

[jbrzozoski]

at_client boots since it uses CONFIG_BSD_LIBRARY_SYS_INIT=y, and that ignores the result code from bsd_init:

	err = bsd_init();

	if (IS_ENABLED(CONFIG_BSD_LIBRARY_SYS_INIT)) {
		/* bsd_init() returns values from a different namespace
		 * than Zephyr's. Make sure to return something in Zephyr's
		 * namespace, in this case 0, when called during SYS_INIT.
		 * Non-zero values in SYS_INIT are currently ignored.
		 */
		return 0;
	}

	return err;

So ... it appears the modem is actually running "well enough" even when bsd_init returns an error.