DFU by app using MTP

RE: DFU by app using MTP

peterz — Sun, 29 Aug 2021 12:31:47 GMT

Hi again
finally SOLVED (pfff...)
the last problem was a misunderstanding on my side. As debugging bootloader mechanics with Segger Studio tends to be complicated (is it possible at all?) I used J-Link RTT Viewer to see bootloader debug output.
But it seems to me that a restart causes RTT Viewer to lose connection (reset might clear RTT buffers?).
This led me to the wrong conclusion that bootloader is not started by MBR.
I added 20 seconds wait in the bootloader code - enough time to stop/restart JRTT Viewer and see bootloader messages until the new firmware image is installed and takes control.

What other developer should know when they are implementing application based firmware update:

a) after a reset the MBR takes control, calls the bootloader which in turn calls whatever sits at 0x00001000 (softdevice or application)

b) <SDK>/examples/peripheral/blinky/pca10056/mbr/armgcc/blinky_gcc_nrf52.ld is your friend. Make sure you understand
FLASH (rx) : ORIGIN = 0x1000, LENGTH = 0xff000
RAM (rwx) : ORIGIN = 0x20000008, LENGTH = 0x3fff8

c) VERY PAINFUL to find: for my application I used <SDK>/examples/peripheral/usbd_hid_composite
   For most of the development I used Segger Studio. But for deployment I used armgcc.
   In Segger Studio the code worked and at the beginning the GCC version worked as well.
   You don't want to read about the journey it took to find the following:
   the GCC Makefile of the example says
       # Optimization flags
       OPT = -O3 -g3
   while the Segger project says
       gcc_optimization_level="Optimize For Size"
   Setting OPT = -Os in Makefile did the trick.
   Why/where/what gets broken by -O3 -g3 I don't know and I will not find out

d) the bootloader concept covers successfully many complex questions.
   But the idea that an application takes care of firmware transport is not really supported.
   Read the source files in <SDK>/components/libraries/bootloader
   Wrong expectations lead to lots of wasted hours.
   Beware: if the bootloader has no clue yet about the installed application nrf_dfu_bank1_start_addr() will return 0x00001000

e) depending on what you want to achieve (how close/far your use case is to the standard DFU procedure) it will take 1 hour - or 1 month.
REFUSE a deadline for the implementation - it takes what it takes

f) I could not open the COM port with nrfutil for USB-Serial DFU.
   get a precompiled binary from github.com/.../
   (I agree: downloading a precompiled EXE for python based application is weird. But I spent 5 hours
   on Windows 8.1, Windows 10 and a Debian Linux trying python 2.7, 3.6, 3.8 - nothing worked)

g) note: the debug versions of examples/dfu/secure_bootloader/... not only add debug output but also size because a lot more code is linked. As a consequence the bootloader base address moves away from 0x000F8000

h) the function nrf_bootloader_app_start_final() used within the bootloader to launch the application tries to nrf_bootloader_flash_protect() which can cause the bootloader to reboot, creating an infinite loop without ever calling the application. I don't understand this (something to do with interrupts which are disabled just before the function?). This problem does not arise with the application /examples/dfu/secure_dfu_test_images/uart/nrf52840/blinky_mbr.hex (only with my own app - I have not the faintest idea why)

RE: DFU by app using MTP

peterz — Fri, 27 Aug 2021 09:56:02 GMT

Hi Sigurd

the HEX file of the bootloader ends with
:020000041000EA
:0410140000800F0049
:0410180000E00F00E5
:04000003F00082B5D2
which writes UICR registers

double checking:
nrfjprog --memrd 0x10001014 0x04 0x01
0x10001014: 000F8000 |....|

nrfjprog --memrd 0x10001018 0x04 0x01
0x10001018: 000FE000 |....|

and most important: the bootloader gets started as long as there is no blinky app.
IMHO this proves the MBR knows the starting address of the bootloader.

Adding the blinky app with nrfjprog (without touching anything else! UICR is NOT modified, just checked again) alters the situation: MBR ignores the bootloader and starts blinky.

IMHO this is at least unexpected. You never encounter this situation for the whole concept of the bootloader is based on the idea, that the first application is loaded by the bootloader (not the app). So setting the UICR vector is not enough. The bootloader does something more - something that I do not do. But what???

RE: DFU by app using MTP

Sigurd — Fri, 27 Aug 2021 09:35:22 GMT

Hi,

We have some MBR documentation here: https://infocenter.nordicsemi.com/topic/sds_s140/SDS/s1xx/mbr_bootloader/bootloader.html

[quote user="zankl"]To find out why MBR prefers starting the app instead of the bootloader[/quote]

From the docs:

The BOOTLOADERADDR address can be provided either at the MBR_BOOTLOADER_ADDR flash memory location, which is defined in nrf_mbr.h, or in the UICR.NRFFW[0] register.

So if this is not done, MBR will not start the bootloader.

RE: DFU by app using MTP

peterz — Thu, 26 Aug 2021 20:58:36 GMT

Hi again
as I can bear 3-4 hours of frustration per day I make very little progress
I boiled it all down to a simple question (see below):

nrfjprog -f nrf52 --halt
nrfjprog -f nrf52 --eraseall
nrfjprog -f nrf52 --program <myOwnBootloader.hex>
nrfjprog -f nrf52 --program nRF5_SDK_17.0.2_d674dde/components/softdevice/mbr/hex/mbr_nrf52_2.4.1_mbr.hex
nrfjprog -f nrf52 --reset

My bootloader gets started and produces empty bootloader settings at 0x000FF000 and an identical backup at 0x000FE000
This is essentially unmodified code and expected behaviour (taken from an example, just removed the transport).

Then I add the blinky MBR example image

nrfjprog -f nrf52 --program nRF5_SDK_17.0.2_d674dde/examples/dfu/secure_dfu_test_images/uart/nrf52840/blinky_mbr.hex
nrfjprog -f nrf52 --reset

result: MBR does NOT call bootloader anymore but starts directly blinky
But if the bootloader is not called anymore it has no chance to ever install the new image I prepare within my app.
To find out why MBR prefers starting the app instead of the bootloader I compared bootloader settings produced by examples with settings I produced myself. I found only one candidate which could explain what I see, the structure nrf_dfu_settings_t member
boot_validation_t boot_validation_bootloader

That would make sense: MBR could check validity of the bootloader.
So I tried to reproduce the values I found in the examples - I failed.
Then I tried to guess (and failed again):

mySettings.boot_validation_bootloader.type = VALIDATE_CRC;
crc = crc32_compute((uint8_t*)0x000F8000, 0x000FE000 - 0x000F8000, NULL);
*(uint32_t *)&(mySettings.boot_validation_bootloader.bytes[0]) = crc;

Now I'm out of ideas and there's no MBR source to check:
MBR calls the bootloader as long as there's no application.
But why does MBR not call the bootloader as soon as there's the blinky app?

Maybe you could copy/paste the few lines of MBR code that show how/hwy the bootloader is ruled out?

Thank you

RE: DFU by app using MTP

Sigurd — Mon, 23 Aug 2021 10:23:45 GMT

Hi,

[quote user="zankl"]Now I guess my path to succes should be:[/quote]

Correct.

[quote user="zankl"]interesting and not yet understood: 8 bytes of RAM are skipped - why?[/quote]

The first 8 bytes are used by the MBR for interrupt forwarding.

RE: DFU by app using MTP

peterz — Sun, 22 Aug 2021 13:37:23 GMT

Hi
after further investigation I think the missing piece in my picture is MBR. Its source code is not available so I have to guess.
The RESET vector points to the code in MBR. If the MBR code detects a bootloader it passes control to this bootloader. Otherwise it calls an application at 0x1000 (aka softdevice).

Helpful: the blinky example comes in 2 flavours - blank and MBR. The difference between the linker files is

$ diff blank/armgcc/blinky_gcc_nrf52.ld mbr/armgcc/blinky_gcc_nrf52.ld
<   FLASH (rx) : ORIGIN = 0x0, LENGTH = 0x100000
<   RAM (rwx) :  ORIGIN = 0x20000000, LENGTH = 0x40000
---
>   FLASH (rx) : ORIGIN = 0x1000, LENGTH = 0xff000
>   RAM (rwx) :  ORIGIN = 0x20000008, LENGTH = 0x3fff8

The MBR version puts blinky in the position of the softdevice (interesting and not yet understood: 8 bytes of RAM are skipped - why?)

Now I guess my path to succes should be:
a) move my application code to 0x1000 (reserving 8 bytes of RAM) to put it in place of the softdevice
b) add MBR

best regards, Peter

RE: DFU by app using MTP

peterz — Thu, 19 Aug 2021 13:35:21 GMT

after checking my DFU for days I find that everything would work - except booting. The application prepares bank1, the bootloader settings and reboots. But the CPU does definitely NOT execute the bootloader but restarts the application instead.

On the other hand: all is in place and I step through the bootloader in Segger Studio. It perfectly recognizes the new image, copies bank1 to bank0, updates bootloader settings and MBR (settings backup), restarts the bootloader, detects there^s nothing left to do...
        case ACTIVATION_NONE:
            nrf_bootloader_app_start();
            break;
and nrf_bootloader_app_start() ends up somewhere in Nirwana but not in the app.

Yes, UICR settings are correct and checked (snippet from bootloader hex):
:02 0000 04 1000 EA
:04 1014 00 00800F00 49
:04 1018 00 00E00F00 E5

I have the bootloader and a valid app in place and do a power up:
the app starts - no trace from the bootloader.

Now I compiled the blinky example to check what pointer is set at VTOR address 0x00000000
The hex file says in the first line
:10 0000 00 00000420 B5020000 DD020000 DF020000 55
does this really mean VTOR Reset vector points to 0x20040000, the very first non-existing address in RAM? I do definitely not understand what's going on here :-(
Does this address intentionally provoke a hard fault which is then handled?
Or is VTOR not at address 0x00000000 as I expect? What vector is used at power up?

I need some pat on the shoulder (a hint, a pointer to docs, whatever):
how the heck does BOOT work???

best regards, Peter

PS - EDIT:
gcc_startup_nrf52840.S reveals
0x00000000 __StackTop
0x00000004 Reset_Handler
(so I was wrong to expect the RESET vector at address 0x00000000)

RE: DFU by app using MTP

peterz — Mon, 09 Aug 2021 08:59:22 GMT

> Is it correct that you have set "#define NRF_DFU_IN_APP 1" in the bootloader?
no this was wrong - thanks for the hint!

and also wrong in my fake bootloader settings for the very first DFU - I should have added:

    // copied from boot_validation_crc(nrf_dfu_settings_t const * p_settings) in nrf_dfu_settings.c
    dummySettings.boot_validation_crc = crc32_compute((const uint8_t *)&(dummySettings.boot_validation_softdevice),
                          (3 * sizeof(boot_validation_t)), NULL);

Providing debug output from the bootloader requires adding a lot of library code which leads to much bigger bootloader - it won't fit anymore to 0xF8000 - 0xFDFFF
It isn't working yet but I feel like making progress again - I'll provide feedback - hopefully I'll tell you that it works.

thanks a lot
Peter

RE: DFU by app using MTP

Jørgen Holmefjord — Thu, 05 Aug 2021 14:17:10 GMT

Hi,

Is it correct that you have set "#define NRF_DFU_IN_APP 1" in the bootloader? This should normally be set in the "Background DFU application" to prevent it from writing the parts that bootloader will handle. If you set this in the bootloader, it may not work properly.

Would you be able to provide minimal examples/instructions that can be used to reproduce/debug this issue?

Can you enable and provide logs (preferably debug level) from the bootloader?

Best regards,
Jørgen

RE: DFU by app using MTP

peterz — Thu, 29 Jul 2021 14:02:39 GMT

Hi Jørgen

yes, my image is just a proof of concept and small enough to fit (<100k)
the 'fake bootloader settings' are calculated to eat up the first half of the available flash and to place bank 1 in the second half. Then I calculate CRC of whatever is currently in the first half (bank0).
I'm aware of the 52820 having just a quarter of the flash and I want to make it work with the DevKit before I move to my own hardware.

[quote userid="14926" url="~/f/nordic-q-a/77902/dfu-by-app-using-mtp/322492#322492"]The initial bootloader settings should either be generated by the bootloader when you perform the first DFU, or be generated by nrfutil and merged with the initial flashed application.[/quote]

I do not have a "first DFU" and integrating MTP as transport into the bootloader would a nightmare and a waste of flash. Generating settings with nrfutil is both uncomfortable while developing and an (unwanted) additional step for production. IMHO it's quite simple to create these initial settings for the very first DFU on the fly.

I like to separate my settings in app_config.h and for the bootloader it contains the following:

#define NRF_BL_DFU_ALLOW_UPDATE_FROM_APP 1
#define NRF_DFU_REQUIRE_SIGNED_APP_UPDATE 1
#define NRF_DFU_APP_DOWNGRADE_PREVENTION 0
#define NRF_DFU_EXTERNAL_APP_VERSIONING 0
#define NRF_DFU_FORCE_DUAL_BANK_APP_UPDATES 1
#define NRF_DFU_SINGLE_BANK_APP_UPDATES 0

#define NRF_BL_DFU_ENTER_METHOD_BUTTON 0
#define NRF_BL_DFU_ENTER_METHOD_BUTTONLESS 0

#define NRF_DFU_HW_VERSION 52


// <i> Enable this to disable writing to areas of the settings that are protected
// <i> by the bootlader. If this is not enabled in the app, certain settings write
// <i> operations will cause HardFaults or will be ignored. Enabling this option
// <i> also causes postvalidation to be disabled since this is meant to be done
// <i> in the bootloader. NRF_BL_DFU_ALLOW_UPDATE_FROM_APP must be enabled in the bootloader.
#define NRF_DFU_IN_APP 1


// NRF_DFU_APP_DATA_AREA_SIZE - The size (in bytes) of the flash area reserved for application data. 
// This area is found at the end of the application area, next to the start of
// the bootloader. This area will not be erased by the bootloader during a
// firmware upgrade. The size must be a multiple of the flash page size.
#define NRF_DFU_APP_DATA_AREA_SIZE 12288


// <i> External apps are apps that will not be activated. They can 
// <i> e.g. be apps to be sent to a third party. External app updates 
// <i> are verified upon reception, but will remain in bank 1, and 
// <i> will never be booted. An external app will be overwritten if 
// <i> a new DFU procedure is performed. Note: This functionality is 
// <i> experimental and not yet used in any examples.
#define NRF_DFU_SUPPORTS_EXTERNAL_APP 0



#define NRF_LOG_ENABLED 0
#define NRF_LOG_BACKEND_RTT 0
#define NRF_LOG_DEFERRED 0
#define NRF_LOG_DEFAULT_LEVEL 4

#define NRF_LOG_BACKEND_RTT_ENABLED 1
#define NRF_LOG_BACKEND_RTT_TEMP_BUFFER_SIZE 64
#define SEGGER_RTT_CONFIG_BUFFER_SIZE_UP 4096
#define SEGGER_RTT_CONFIG_MAX_NUM_UP_BUFFERS 2
#define SEGGER_RTT_CONFIG_BUFFER_SIZE_DOWN 16
#define SEGGER_RTT_CONFIG_MAX_NUM_DOWN_BUFFERS 2
#define SEGGER_RTT_CONFIG_DEFAULT_MODE 0

thank you very much for your support :-))

Peter

RE: DFU by app using MTP

Jørgen Holmefjord — Thu, 29 Jul 2021 13:42:56 GMT

Hi,

Are you using an image that will fit in the flash of the device?

The "Fake Bootloader settings" you posted shows an application size of 0x0007AFFC (503804 bytes), while the nRF52820 only have 256 kB of flash available.

The initial bootloader settings should either be generated by the bootloader when you perform the first DFU, or be generated by nrfutil and merged with the initial flashed application.

It sounds like what you are trying to achieve is very similar to how the Background DFU solution in previous SDKs work. Have you configured the bootloader to allow updates from the application (NRF_BL_DFU_ALLOW_UPDATE_FROM_APP)? I assume this is where you try to write the bootloader settings from?

Best regards,
Jørgen