Updatable bootloader for nrf9160

[gregers]

Hi, I'm confused how to do fota of mcuboot on the nrf9160.

The source code is public on github in this branch: https://github.com/ExploratoryEngineering/nrf9160-telenor/tree/immutable-bootloader

fota sample is here: https://github.com/ExploratoryEngineering/nrf9160-telenor/tree/immutable-bootloader/samples/fota

The sample use lwm2m to download a new firmware image from our (Telenor's) IoT gateway. It's just standard LwM2M, and uses dfu_target to flash the new firmware. I have enabled CONFIG_SECURE_BOOT to use the immutable bootloader and use CONFIG_FW_INFO_FIRMWARE_VERSION to bump mcuboot's version.

I've tried to read the source code to understand what's going on, but I'm struggling to understand a few pieces. According to the mcuboot design docs, I get the impression that image swaps only happen between the image slots. However the dfu_target doesn't look at the image header before flashing the image to the secondary slot for the application instead of s1. I might be missing some magic configuration or call to change this behaviour, but I can't find it. By commenting out the reboot I can verify that the image is actually written to the secondary slot:

$ nrfjprog -f nrf91 --memrd 0x94400 --n 28
0x00094400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00094410: 00009080 00000002 00015200            |.........R..|

The secondary slot starts at 0x94000 and by looking at the fw_info data we can see that this is version 2 and the start of the image is 0x15200, which is where s1 is located.

Double check fw_info for s0 and s1:

$ nrfjprog -f nrf91 --memrd 0x8400 --n 28
0x00008400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00008410: 00009080 00000001 00008200            |............|

$ nrfjprog -f nrf91 --memrd 0x15400 --n 28
0x00015400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00015410: 00009080 00000001 00015200            |.........R..|

When calling dfu_target_done, it writes BOOT_SWAP_TYPE_TEST to the FLASH_AREA_IMAGE_SECONDARY swap field.

On the next reboot the immutable bootloader choose s0, since s1 is not changed in flash yet. Then (old) mcuboot does it's magic and boots the application. Now if I look at the flash, s1 has been updated with the image flashed to the secondary slot:

$ nrfjprog -f nrf91 --memrd 0x8400 --n 28
0x00008400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00008410: 00009080 00000001 00008200            |............|

$ nrfjprog -f nrf91 --memrd 0x15400 --n 28
0x00015400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00015410: 00009080 00000002 00015200            |.........R..|

In the fota code in our application we would like to see what version and slot the bootloader is using. We used fw_info to check this, but even though s0 was used during boot fw_info will report what's on flash after booting, so it will look like s1 was used since it's valid and has a higher version.

By manually rebooting, the immutable bootloader now sees version 2 in s1 and boots from 0x15200. However I don't know how we'll be able to detect this state from code and trigger the second reboot.

Sorry for the long explanation. Here are my questions:

• Are we doing something wrong since the dfu_target stores the image in the wrong slot? (or is it not wrong)
• mcuboot only has one public key embedded to verify the signature of the application. If we want to change this key, I assume we have to update both mcuboot and the application in flash before rebooting. How can we do this if both are stored in the secondary slot?
• Is it possible to do a test run of mcuboot before persisting it?
• The immutable bootloader have a list of public keys stored in the OTP area for verifying the mcuboot signature. Will we brick the device if we upload a new version of mcuboot signed with a new key and mcuboot or the application fails the test? Looks like all previous public keys are invalidated before the new images are marked OK. Is this intentional?
• I've read that images can have dependencies between them, so when one fails mcuboot will roll back the others. Is this used automatically for mcuboot and the application, or does it not make sense to use for the bootloader?
• I have a few suggestions for improving the documentation. What is your preferred method of giving feedback on the documentation? Do you have a public github for the docs hosted on https://developer.nordicsemi.com?

[shibshab]

"I have enabled CONFIG_SECURE_BOOT to use the immutable bootloader and use CONFIG_FW_INFO_FIRMWARE_VERSION to bump mcuboot's version."

(looks like you did this correct, but I'm explaining here for any future readers)
Note that this must be done for the mcuboot image.
E.g. by running `ninja mcuboot_menuconfig` by adding it to `ncs/bootloader/mcuboot/boot/zephyr/prj.conf`

"However the dfu_target doesn't look at the image header before flashing the image to the secondary slot for the application instead of s1"

This is a bit tricky to explain, but I will do my best.
MCUBoot supports multiple images, where each image has a secondary and primary slot.
When an update is approved, it is swapped from its secondary slot to its primary slot.
We made a modification to MCUBoot where two images can _share_ the secondary slot.
When MCUBoot iterates over all of its images (two in your case) it will look at the vector table of the image in the secondary slot to ensure that the correct primary slot is used.
Hence, both application and mcuboot updates will be stored in the same secondary slot.
For this reason (single secondary slot) there is no support for simultaneous updates of the application and mcuboot, hence there is no dependency management in this architecture.

The overall flow is now:
- Download mcubootv2 into secondary slot
- mcubootv1 checks if image0 (application) has an update in the secondary slot
- there is an update, but is it for image0 (check address of vector table against address of image0 primary slot)
- NO
- mcubootv1 check if image1 (mcuboot) has an update in the secondary slot (same slot as above)
- there is an update, AND it is for image1 (vector table is within range of image1 primary slot)
- YES
- Swap image from secondary slot to image1 (mcuboot) primary slot (S0 or S1).

Note that the mcuboot candidate generated for running from S0 has S1 as image1 primary while the mcuboot candidate running from S1 has S0 as image1 primary slot.

"In the fota code in our application we would like to see what version and slot the bootloader is using. We used fw_info to check this, but even though s0 was used during boot fw_info will report what's on flash after booting, so it will look like s1 was used since it's valid and has a higher version"

I see your point, this would be fixed if mcuboot did a reset after performing the update of mcuboot, do you agree?

Now to your actual questions:

"Are we doing something wrong since the dfu_target stores the image in the wrong slot? (or is it not wrong)"

Its not wrong. As explained above the two images seen by mcuboot (mcuboot and application) share the secondary slot.

"mcuboot only has one public key embedded to verify the signature of the application. If we want to change this key, I assume we have to update both mcuboot and the application in flash before rebooting. How can we do this if both are stored in the secondary slot?"

You can not do this with the current solution (single secondary bank again).
I would propose that you post a issue in the upstream MCUBoot project to support the same key revocation scheme that we use in NCS, or check why its not supported).
As a hack, you can update to a version of mcuboot which does not verify the application on startup, then the new key will only be used in DFU scenarios.

"Is it possible to do a test run of mcuboot before persisting it?"

This is not trivial, since the mcuboot instance being reverted is the one currently running (if that makes sense) if we want to use the mcuboot test run mechanism.
Instead this is left for B0, and if a signature check of s0 or s1 fails it will be marked invalid, and the alternate slot will be used.

"The immutable bootloader have a list of public keys stored in the OTP area for verifying the mcuboot signature. Will we brick the device if we upload a new version of mcuboot signed with a new key and mcuboot or the application fails the test? Looks like all previous public keys are invalidated before the new images are marked OK. Is this intentional?"

Yes, this is the key revocation scheme. If you have 5 keys [0-4] and you loose key 0 you can sign with key 1 to invalidate key 0 etc.

"I've read that images can have dependencies between them, so when one fails mcuboot will roll back the others. Is this used automatically for mcuboot and the application, or does it not make sense to use for the bootloader?"

Again, because of the single secondary slot this is not supported.

"I have a few suggestions for improving the documentation. What is your preferred method of giving feedback on the documentation? Do you have a public github for the docs hosted on ">developer.nordicsemi.com

You are very welcome, I would propose PRs directly against github.com/.../fw-nrfconnect-nrf

[gregers]

Thank you for the thorough answer shibshab!

shibshab said:

We made a modification to MCUBoot where two images can _share_ the secondary slot.

I don't understand why this choice was made rather than storing the image in the slot for the corresponding image in the first place. I would rather have that fixed than filing an issue upstream. Is this a temporary solution, or do you have a plan to fix this?

shibshab said:

I see your point, this would be fixed if mcuboot did a reset after performing the update of mcuboot, do you agree?

It's a hack that could work to update mcuboot (without changing app signing key) until the issue above is solved.

shibshab said:

Again, because of the single secondary slot this is not supported.

It would be really nice if the images were flashed to the corresponding slot, so we could update both at once and be able to roll back when one of them fails. Also without invalidating the old public key.

[shibshab]

Our assumptions is that the bootloader will be changed 100 times less than then application. The reasons would be 1 - security flaw discovered, or 2 - its no longer able to update the application properly (several reasons why this might happen). Hence we did not see a big loss of functionality when removing the support for simultaneous update of mcuboot and application. The cost, however, of supporting simultaneous update is severe for most applications in terms of flash space used. 

IMO it would be a small change to get the behavior you describe, just remove the NCS patch in loader.c (diff the ncs fork from the upstream) which enables sharing of the secondary slot, and modify dfu_target to check the vector table (like we do in the ncs version of loader.c) when choosing which secondary slot to store the image to.

[gregers]

shibshab said:

The reasons would be 1 - security flaw discovered, or 2 - its no longer able to update the application properly (several reasons why this might happen).

You would also need to update the bootloader if the signing key gets leaked. Of course you should secure the private key properly. But with the lack of best practices on the subject, I'd say it's quite likely some of your customers will experience this.

shibshab said:

Hence we did not see a big loss of functionality when removing the support for simultaneous update of mcuboot and application.

Except that it's impossible to change the application signing key without disabling the signature check while updating.

shibshab said:

The cost, however, of supporting simultaneous update is severe for most applications in terms of flash space used.

I don't understand why. The vector table is in the beginning of the image, so it should be possible to detect the correct address very early in the download process. Right? I don't have much experience with this, so I haven't really thought this through, but would like to understand. Is the unused bootloader slot write protected, or can the application write to the slot that is not in use?

Thank you for the tip about loader.c. I got the diff, and will test it out.

[shibshab]

So if you loose your signing key, you can (as I might have mentioned above) update mcuboot with a new version which has a new key, but does not perform validation on boot-up, only on DFU. This way you should be able to only accept new updates signed with the new key.

Yes, the unused bootloader slot is write protected. There are several reasons for this, the most important one being hardware limitations on how small blocks of flash can be write protected. Since the minimum size is 32kB we squeeze 2 mcubootinstances inside (s0 and S1) and use 32kB for mcuboot instead of 64kB.