This post is older than 2 years and might not be relevant anymore
More Info: Consider searching for newer posts

Updatable bootloader for nrf9160

Hi, I'm confused how to do fota of mcuboot on the nrf9160.

The source code is public on github in this branch: https://github.com/ExploratoryEngineering/nrf9160-telenor/tree/immutable-bootloader

fota sample is here: https://github.com/ExploratoryEngineering/nrf9160-telenor/tree/immutable-bootloader/samples/fota

The sample use lwm2m to download a new firmware image from our (Telenor's) IoT gateway. It's just standard LwM2M, and uses dfu_target to flash the new firmware. I have enabled CONFIG_SECURE_BOOT to use the immutable bootloader and use CONFIG_FW_INFO_FIRMWARE_VERSION to bump mcuboot's version.

I've tried to read the source code to understand what's going on, but I'm struggling to understand a few pieces. According to the mcuboot design docs, I get the impression that image swaps only happen between the image slots. However the dfu_target doesn't look at the image header before flashing the image to the secondary slot for the application instead of s1. I might be missing some magic configuration or call to change this behaviour, but I can't find it. By commenting out the reboot I can verify that the image is actually written to the secondary slot:

$ nrfjprog -f nrf91 --memrd 0x94400 --n 28
0x00094400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00094410: 00009080 00000002 00015200            |.........R..|

The secondary slot starts at 0x94000 and by looking at the fw_info data we can see that this is version 2 and the start of the image is 0x15200, which is where s1 is located.

Double check fw_info for s0 and s1:

$ nrfjprog -f nrf91 --memrd 0x8400 --n 28
0x00008400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00008410: 00009080 00000001 00008200            |............|

$ nrfjprog -f nrf91 --memrd 0x15400 --n 28
0x00015400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00015410: 00009080 00000001 00015200            |.........R..|

When calling dfu_target_done, it writes BOOT_SWAP_TYPE_TEST to the FLASH_AREA_IMAGE_SECONDARY swap field.

On the next reboot the immutable bootloader choose s0, since s1 is not changed in flash yet. Then (old) mcuboot does it's magic and boots the application. Now if I look at the flash, s1 has been updated with the image flashed to the secondary slot:

$ nrfjprog -f nrf91 --memrd 0x8400 --n 28
0x00008400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00008410: 00009080 00000001 00008200            |............|

$ nrfjprog -f nrf91 --memrd 0x15400 --n 28
0x00015400: 281EE6DE 8FCEBB4C 00005B02 0000003C   |...(L....[..<...|
0x00015410: 00009080 00000002 00015200            |.........R..|

In the fota code in our application we would like to see what version and slot the bootloader is using. We used fw_info to check this, but even though s0 was used during boot fw_info will report what's on flash after booting, so it will look like s1 was used since it's valid and has a higher version.

By manually rebooting, the immutable bootloader now sees version 2 in s1 and boots from 0x15200. However I don't know how we'll be able to detect this state from code and trigger the second reboot.

Sorry for the long explanation. Here are my questions:

  • Are we doing something wrong since the dfu_target stores the image in the wrong slot? (or is it not wrong)
  • mcuboot only has one public key embedded to verify the signature of the application. If we want to change this key, I assume we have to update both mcuboot and the application in flash before rebooting. How can we do this if both are stored in the secondary slot?
  • Is it possible to do a test run of mcuboot before persisting it?
  • The immutable bootloader have a list of public keys stored in the OTP area for verifying the mcuboot signature. Will we brick the device if we upload a new version of mcuboot signed with a new key and mcuboot or the application fails the test? Looks like all previous public keys are invalidated before the new images are marked OK. Is this intentional?
  • I've read that images can have dependencies between them, so when one fails mcuboot will roll back the others. Is this used automatically for mcuboot and the application, or does it not make sense to use for the bootloader?
  • I have a few suggestions for improving the documentation. What is your preferred method of giving feedback on the documentation? Do you have a public github for the docs hosted on https://developer.nordicsemi.com?
Parents
  • Thank you for the thorough answer shibshab!

    We made a modification to MCUBoot where two images can _share_ the secondary slot.

    I don't understand why this choice was made rather than storing the image in the slot for the corresponding image in the first place. I would rather have that fixed than filing an issue upstream. Is this a temporary solution, or do you have a plan to fix this?

    I see your point, this would be fixed if mcuboot did a reset after performing the update of mcuboot, do you agree?

    It's a hack that could work to update mcuboot (without changing app signing key) until the issue above is solved.

    Again, because of the single secondary slot this is not supported.

    It would be really nice if the images were flashed to the corresponding slot, so we could update both at once and be able to roll back when one of them fails. Also without invalidating the old public key.

  • Our assumptions is that the bootloader will be changed 100 times less than then application. The reasons would be 1 - security flaw discovered, or 2 - its no longer able to update the application properly (several reasons why this might happen). Hence we did not see a big loss of functionality when removing the support for simultaneous update of mcuboot and application. The cost, however, of supporting simultaneous update is severe for most applications in terms of flash space used. 

    IMO it would be a small change to get the behavior you describe, just remove the NCS patch in loader.c (diff the ncs fork from the upstream) which enables sharing of the secondary slot, and modify dfu_target to check the vector table (like we do in the ncs version of loader.c) when choosing which secondary slot to store the image to.

  • The reasons would be 1 - security flaw discovered, or 2 - its no longer able to update the application properly (several reasons why this might happen).

    You would also need to update the bootloader if the signing key gets leaked. Of course you should secure the private key properly. But with the lack of best practices on the subject, I'd say it's quite likely some of your customers will experience this.

    Hence we did not see a big loss of functionality when removing the support for simultaneous update of mcuboot and application.

    Except that it's impossible to change the application signing key without disabling the signature check while updating.

    The cost, however, of supporting simultaneous update is severe for most applications in terms of flash space used.

    I don't understand why. The vector table is in the beginning of the image, so it should be possible to detect the correct address very early in the download process. Right? I don't have much experience with this, so I haven't really thought this through, but would like to understand. Is the unused bootloader slot write protected, or can the application write to the slot that is not in use?

    Thank you for the tip about loader.c. I got the diff, and will test it out.

  • So if you loose your signing key, you can (as I might have mentioned above) update mcuboot with a new version which has a new key, but does not perform validation on boot-up, only on DFU. This way you should be able to only accept new updates signed with the new key.

    Yes, the unused bootloader slot is write protected. There are several reasons for this, the most important one being hardware limitations on how small blocks of flash can be write protected. Since the minimum size is 32kB we squeeze 2 mcubootinstances inside (s0 and S1) and use 32kB for mcuboot instead of 64kB.

  • I've attempted disabling boot validation by adding CONFIG_BOOT_VALIDATE_SLOT0=n to the mcuboot config and re-compile mcuboot and application with a new signature file for the application. However mcuboot refuse to swap the new mcuboot image into s1 with the error: «Image in the secondary slot is not valid!». This is strange, since I have only changed the signature file that is used for signing the application. So mcuboot should be signed with the same key. Not entirely sure I'm doing this correctly though, since I had to specify the signature file with CONFIG_BOOT_SIGNATURE_KEY_FILE for both the application and mcuboot. I had to do that to get mcuboot to validate the application with the key that was used in the application and not the default key in mcuboot - since the cmake code strips all previous CONFIG_* when adding a new target.

    Did I configure it incorrectly? Please see the github repo for how I've specified signature files in the config.

    I also looked into the flash access control that you mentioned. Assuming the 32 regions of 32 kB start at address 0 and increase by 0x8000 for each region, the mcuboot s0 starts at 0x8000 - which makes sense. However it's not correct that both mcuboot slots fit into one of these regions. The size of one slot it 0xC200, or roughly 50 kB. So you could almost squeeze both slots into three regions, but s1 starts at 0x15000 and goes 4608 bytes into the 5th region that starts at 0x20000. So in practice they use 4 of the 32 kB regions.

    Looking at the mcuboot fork, it seems like the flash protection was added after the last release (sdk 1.2), and can be disabled from config. The patch to loader.c is part of sdk 1.2, so the reason for patching loader.c to write the mcuboot image to the applications secondary slot couldn't have been because s0 and s1 was write protected. Could it be another reason why you decided to patch loader.c?

    To me it seems strange to disable the «chain of trust» to be able to flash protect the mcuboot slots. If you reverted the patch to loader.c (or at least have it configurable) so the image would be written to the slot it belongs, we could rely on «chain of trust» during boot instead - and updating both mcuboot and the application simultaneously would be possible.

Reply
  • I've attempted disabling boot validation by adding CONFIG_BOOT_VALIDATE_SLOT0=n to the mcuboot config and re-compile mcuboot and application with a new signature file for the application. However mcuboot refuse to swap the new mcuboot image into s1 with the error: «Image in the secondary slot is not valid!». This is strange, since I have only changed the signature file that is used for signing the application. So mcuboot should be signed with the same key. Not entirely sure I'm doing this correctly though, since I had to specify the signature file with CONFIG_BOOT_SIGNATURE_KEY_FILE for both the application and mcuboot. I had to do that to get mcuboot to validate the application with the key that was used in the application and not the default key in mcuboot - since the cmake code strips all previous CONFIG_* when adding a new target.

    Did I configure it incorrectly? Please see the github repo for how I've specified signature files in the config.

    I also looked into the flash access control that you mentioned. Assuming the 32 regions of 32 kB start at address 0 and increase by 0x8000 for each region, the mcuboot s0 starts at 0x8000 - which makes sense. However it's not correct that both mcuboot slots fit into one of these regions. The size of one slot it 0xC200, or roughly 50 kB. So you could almost squeeze both slots into three regions, but s1 starts at 0x15000 and goes 4608 bytes into the 5th region that starts at 0x20000. So in practice they use 4 of the 32 kB regions.

    Looking at the mcuboot fork, it seems like the flash protection was added after the last release (sdk 1.2), and can be disabled from config. The patch to loader.c is part of sdk 1.2, so the reason for patching loader.c to write the mcuboot image to the applications secondary slot couldn't have been because s0 and s1 was write protected. Could it be another reason why you decided to patch loader.c?

    To me it seems strange to disable the «chain of trust» to be able to flash protect the mcuboot slots. If you reverted the patch to loader.c (or at least have it configurable) so the image would be written to the slot it belongs, we could rely on «chain of trust» during boot instead - and updating both mcuboot and the application simultaneously would be possible.

Children
  • You should sign the mcuboot update with the OLD key, but it should contain the NEW key inside itself, was this what you did? To do this I think you have to manually invoke the imgtool.py scripts. See the build.ninja file in your build directory for examples of signing commands.

    The sizes in the default configurations are not 16kB each, you are correct. To achieve this you need to optimize the mcuboot image, and re-use the crypto from B0 instead of having mcuboot using its own.

    There is an open PR fixing the order in main.c of our fork, since mcuboot updates were practically broken since the flash was locked before any swaps happened.
    https://github.com/NordicPlayground/fw-nrfconnect-mcuboot/pull/97

    When working correctly the chain of trust is not broken.
    What is important is to protect mcuboot from tampering by the application, once this is in place you can more safely disably on-boot verification.

    I agree that a scheme of updating both mcuboot and application simultaneously would be beneficial (especially since external flash would be in place for most 91 devices AFAIK), but this is not something we have planned for now.

  • You should sign the mcuboot update with the OLD key, but it should contain the NEW key inside itself, was this what you did?

    That's what I tried. We use a different key for mcuboot and the application, and I've only changed CONFIG_BOOT_SIGNATURE_KEY_FILE when testing. mcuboot is signed using CONFIG_SB_SIGNING_KEY_FILE, which hasn't been modified. So mcuboot should be signed using the same key as before, whereas the public key for the application should be in the fresh build of mcuboot.

    To do this I think you have to manually invoke the imgtool.py scripts. See the build.ninja file in your build directory for examples of signing commands.

    Really? Isn't it the immutable bootloaders responsibility to validate the signature of mcuboot using the public keys stored in the OTP area? As far as I can tell from studying the cmake files, the provision hex generated for the OTP area extracts the public key from SB_SIGNING_KEY_FILE and includes SB_PUBLIC_KEY_FILES. As far as I can tell, mcuboot is signed using SB_SIGNING_KEY_FILE. But I don't understand how mcuboot is supposed to validate the signature of mcuboot in the dfu process when it doesn't have the public key.

    I thought CONFIG_SB_SIGNING_KEY_FILE was the file used to sign mcuboot when using the immutable bootloader. Haven't changed that file, only CONFIG_BOOT_SIGNATURE_KEY_FILE.

    The sizes in the default configurations are not 16kB each, you are correct. To achieve this you need to optimize the mcuboot image, and re-use the crypto from B0 instead of having mcuboot using its own.

    Is this just a matter of config options? Would be nice with a sample demonstrating more of this. Documentation is really scarce when trying to do this properly.

    There is an open PR fixing the order in main.c of our fork, since mcuboot updates were practically broken since the flash was locked before any swaps happened.

    Sorry for being difficult, but you're avoiding answering my question. I don't see why I need protecting the mcuboot flash area when the immutable bootloader is validating the signature. What I'm curious about is why you decided to patch mcuboot, so the update candidate is written to the secondary slot instead of the address it belongs.

    The reason for asking is that I would like to know if it's worth it for us to revert the patch and implement multi image dfu ourselves. If there are other problems than the flash protection, we'd like to know before starting on this Slight smile

  • Note that mcuboot is signed twice, first by CONFIG_SB_SIGNING_KEY_FILE then by CONFIG_BOOT_SIGNATURE_KEY_FILE. This because mcuboot needs to verify the update of itself using its own key before swapping mcubootv2 into s0/s1, then when b0 boots in will validate mcubootv2 using CONFIG_SB_SIGNING_KEY_FILE. So the procedure is:
    - App downloads mcubootv2 into secondary slot
    - Reboot
    - mcuboot verifies update in secondary slot using CONFIG_BOOT_SIGNATURE_KEY_FILE
    - mcuboot performs swap operation into S0/S1
    - ... reboot
    - b0 performs verification using CONFIG_SB_SIGNING_KEY_FILE

    Is this just a matter of config options? Would be nice with a sample demonstrating more of this. Documentation is really scarce when trying to do this properly.


    Yes. I agree.

    Sorry for being difficult

    You are not being difficult :D This page is now the most thorough documentation of the procedure of mcuboot upgrades :)


    I don't see why I need protecting the mcuboot flash area when the immutable bootloader is validating the signature. What I'm curious about is why you decided to patch mcuboot, so the update candidate is written to the secondary slot instead of the address it belongs.
    The reason for asking is that I would like to know if it's worth it for us to revert the patch and implement multi image dfu ourselves. If there are other problems than the flash protection, we'd like to know before starting on this



    I see your point, and I agree. I have added the feature you request into our backlog. You could create a PR for it and maybe it will be even faster Slight smile

  • Note that mcuboot is signed twice, first by CONFIG_SB_SIGNING_KEY_FILE then by CONFIG_BOOT_SIGNATURE_KEY_FILE. This because mcuboot needs to verify the update of itself using its own key before swapping mcubootv2 into s0/s1, then when b0 boots in will validate mcubootv2 using CONFIG_SB_SIGNING_KEY_FILE.

    So to update mcuboot with a new public key for application signature, you'd have to build with the new key. Then look in the build.ninja file in the build folder. Locate the command that outputs signed_by_mcuboot_and_b0_s1_image_update.bin and replace the key with the old key:

    cd /Users/gregers/devel/nrf9160-telenor/build/modules/mcuboot && /usr/local/bin/python3 /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../scripts/imgtool.py sign --key /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../../../../samples/fota/cert.pem --header-size 0x200 --align 4 --version 0.0.0+0 --slot-size 0x6c000 --pad-header /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_b0_s1_image.hex /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_signed.hex && /usr/local/opt/gcc-arm-none-eabi/bin/arm-none-eabi-objcopy --input-target=ihex --output-target=binary /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_b0_s1_image.hex /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_to_sign.bin && /usr/local/bin/python3 /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../scripts/imgtool.py sign --key /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../../../../samples/fota/cert.pem --header-size 0x200 --align 4 --version 0.0.0+0 --slot-size 0x6c000 --pad-header /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_to_sign.bin /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_update.bin && /usr/local/bin/python3 /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../scripts/imgtool.py sign --key /Users/gregers/devel/nrf9160-telenor/deps/bootloader/mcuboot/zephyr/../../../../samples/fota/cert.pem --header-size 0x200 --align 4 --version 0.0.0+0 --slot-size 0x6c000 --pad-header --pad /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_b0_s1_image.hex /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_test_update.hex && /usr/local/opt/gcc-arm-none-eabi/bin/arm-none-eabi-objcopy --input-target=ihex --output-target=ihex --change-address 520192 /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_test_update.hex /Users/gregers/devel/nrf9160-telenor/build/zephyr/signed_by_mcuboot_and_b0_s1_image_moved_test_update.hex

    Why does mcuboot need to validate the signature of the new mcuboot image before storing it in the slot? Isn't this the job of the immutable bootloader to validate the image with one of the keys in the OTP area? If the image is not signed by any of the keys, it will get invalidated. Right?

    Using the application key to sign mcuboot and making mcuboot validate itself seems like a weird solution to me. It makes it much harder to understand, and makes the update process very fault prone. A process that might end up bricking the device. IoT has already been criticised for it's lack of security. One of the selling points of nrf9160 is that it has ARM Cortex M33 and many good solutions to make secure product. But it doesn't help much if the security features are so hard to use that it scares developers away from using it.

    Please let me explain how I imagine to solve this, and let me know if there are any problems with the solution.

    1. We revert your patch to mcuboot loader.c to write the image to the address it belongs (by looking at the headers)
      1. This solves being able to update the bootloader and the application before rebooting the device, since the images are stored in the extra slot belonging to the image.
      2. It also skips the problematic (imho) signature verification of mcuboot using the application key, or will it still check using the application key? Any way to skip this check and only rely on boot validation?
      3. On the first boot the immutable bootloader will validate the new mcuboot image using the same public key as before from the OTP area, and boot from the new mcuboot address.
      4. The new mcuboot (with a new public key for the application signature) will validate the new application image and do a test swap.
      5. Hopefully the new image is able to boot and confirm. If not the device is bricked until someone has physical access. So it's probably best to not update any bootloader nor application logic at the same time as changing the signature for the application.
  • There is no technical reason for the double signing, and ideally it mcubootv2 should only be signed by B0. The double signing was done to save time when developing this feature.
    (I won't go into the details, but the building blocks available at this today (non secure services among others) make it much easier to come up with a similar scheme without double signing). Hence there is good motivation for improving the mcuboot update procedure by having the app download mcubootv2 directly into S0/S1 and then only having 1 level of signing.

    Why does mcuboot need to validate the signature of the new mcuboot image before storing it in the slot? Isn't this the job of the immutable bootloader to validate the image with one of the keys in the OTP area? If the image is not signed by any of the keys, it will get invalidated. Right?

    It doesn't. It is. The key which does not result in a successful validating will get invalidated, not the image. If the next key in the list of keys validates the image, this is now used, and so on.

    Using the application key to sign mcuboot and making mcuboot validate itself seems like a weird solution to me. It makes it much harder to understand, and makes the update process very fault prone.

    Double signing is weird, and it makes it harder to understand the process, I agree.

    a) Note that you will not have any dependency tracking or synchronization of the two images, because this is handled by mcuboot (B0 does not support this concept at all). So you must verify that you don't end up in the situation where one of the two images (mcuboot and app) has been updates but not the other, such that the device is bricked.

    b) If the app writes mcubootv2 straight into s0/s1 only b0's key will be used to perform validation. There might be a pre-validate function exposed from b0 which you can use for this to verify mcubootv2 already at the application level, using b0's key.

    c) Correct, as long as the version number (CONFIG_FIRMWARE_VERSION_NUMBER or something) is higher.

    d) Yes, if there is an update in mcuboots secondary slot.

    e) This scheme is too risky for production IMO, it would be easier to add the same key-revocation scheme to mcuboot that is used in b0 today. This way you only need to sign the app with key#2 to get key#1 revoked, instead of updating mcuboot itself. Also, if key revocation is the only reason to have an upgradeable bootloader you no longer need b0 at all.

Related