DFU by app using MTP

Question

Hi my application will be based on nRF52820 and has only USB connection (no BLE). I'm developping with nRF5_SDK_17.0.2_d674dde and a nRF52840 devkit. The end user should be able to update the firmware without any tools. Thus I implemented Media Transfer Protocol to copy the new image to the device. First I used nrfutil pkg generate ... to create and sign the new application and I wrote a script to unzip the result and combine bootloader data and firmware image to single file which can be copied to the device using MTP. On the device I receive the combined file and I first check the bootloader data After I've written the firmware to bank 1 the check with nrf_dfu_validation_prevalidate() returns NRF_DFU_RES_CODE_SUCCESS The I took a copy from nRF5_SDK_17.0.2_d674dde/examples/dfu/secure_bootloader/pca10056_uart/ and removed DFU completely (I just want the activation part, but no transport at all) Now I think I have to write valid bootloader settings and reboot. After many days of reading examples, bootloader code and trial/error I think I miserably fail because of some conceptual misunderstanding or wrong expectation. I try to prepare bootloader settings following the procedures I find in the examples. settings_forbidden_parts_copy_from_backup() detects unacceptable changes compared to the backup settings in MBR parameter page OK - I did not do any DFU before, the backup can not contain reasonable data yet - this makes sense. 
 Lets "invent" reasonable bootloader settings reflecting the current app (very nasty hack) at startup and write them to the backup too. 
 nrfutil says about my fake... Bootloader DFU Settings: * File: fakeBootloaderSettings.hex * Family: NRF52840 * CRC: 0xF36F5B3B * Settings Version: 0x00000001 (1) * App Version: 0x00000001 (1) * Bootloader Version: 0x00000002 (2) * Bank Layout: 0x00000000 * Current Bank: 0x00000000 * Application Size: 0x0007AFFC (503804 bytes) * Application CRC: 0xEC72A580 * Bank0 Bank Code: 0x00000001 Looking at the flash memory I see my dummy settings, but the newly created settings are never written to flash. I tried nrf_dfu_validation_activation_prepare() nrf_dfu_validation_post_data_execute() and other variants when I follow the execution in Segger Studio and it comes to reboot I have the strong feeling to get started in the app again. I think I never hit the bootloader. So I checked the UICR 0x10001014 and 0x10001018 and I think they point to the correct location I use nrf_pwr_mgmt_shutdown(NRF_PWR_MGMT_SHUTDOWN_RESET); I have 3 problems: a) how to correctly prepare bootloader settings in the app? b) how to shutdown/reboot the system correctly? c) how can I know if Segger reboots into the bootloader or the app? I'd appreciate a comment how solve problem a) because here I'm definitely stuck 
 best regards Peter

peterz · Answer

Hi again
finally SOLVED (pfff...)
the last problem was a misunderstanding on my side. As debugging bootloader mechanics with Segger Studio tends to be complicated (is it possible at all?) I used J-Link RTT Viewer to see bootloader debug output.
But it seems to me that a restart causes RTT Viewer to lose connection (reset might clear RTT buffers?).
This led me to the wrong conclusion that bootloader is not started by MBR.
I added 20 seconds wait in the bootloader code - enough time to stop/restart JRTT Viewer and see bootloader messages until the new firmware image is installed and takes control.

What other developer should know when they are implementing application based firmware update:

a) after a reset the MBR takes control, calls the bootloader which in turn calls whatever sits at 0x00001000 (softdevice or application)

b) <SDK>/examples/peripheral/blinky/pca10056/mbr/armgcc/blinky_gcc_nrf52.ld is your friend. Make sure you understand
FLASH (rx) : ORIGIN = 0x1000, LENGTH = 0xff000
RAM (rwx) : ORIGIN = 0x20000008, LENGTH = 0x3fff8

c) VERY PAINFUL to find: for my application I used <SDK>/examples/peripheral/usbd_hid_composite
   For most of the development I used Segger Studio. But for deployment I used armgcc.
   In Segger Studio the code worked and at the beginning the GCC version worked as well.
   You don't want to read about the journey it took to find the following:
   the GCC Makefile of the example says
       # Optimization flags
       OPT = -O3 -g3
   while the Segger project says
       gcc_optimization_level="Optimize For Size"
   Setting OPT = -Os in Makefile did the trick.
   Why/where/what gets broken by -O3 -g3 I don't know and I will not find out

d) the bootloader concept covers successfully many complex questions.
   But the idea that an application takes care of firmware transport is not really supported.
   Read the source files in <SDK>/components/libraries/bootloader
   Wrong expectations lead to lots of wasted hours.
   Beware: if the bootloader has no clue yet about the installed application nrf_dfu_bank1_start_addr() will return 0x00001000

e) depending on what you want to achieve (how close/far your use case is to the standard DFU procedure) it will take 1 hour - or 1 month.
REFUSE a deadline for the implementation - it takes what it takes

f) I could not open the COM port with nrfutil for USB-Serial DFU.
   get a precompiled binary from github.com/.../
   (I agree: downloading a precompiled EXE for python based application is weird. But I spent 5 hours
   on Windows 8.1, Windows 10 and a Debian Linux trying python 2.7, 3.6, 3.8 - nothing worked)

g) note: the debug versions of examples/dfu/secure_bootloader/... not only add debug output but also size because a lot more code is linked. As a consequence the bootloader base address moves away from 0x000F8000

h) the function nrf_bootloader_app_start_final() used within the bootloader to launch the application tries to nrf_bootloader_flash_protect() which can cause the bootloader to reboot, creating an infinite loop without ever calling the application. I don't understand this (something to do with interrupts which are disabled just before the function?). This problem does not arise with the application /examples/dfu/secure_dfu_test_images/uart/nrf52840/blinky_mbr.hex (only with my own app - I have not the faintest idea why)