Custom transport layer for dfu in app and dual bank settings issue

[MassimoF]

I'm currently using a HostController with pregiven communication protocol for transfering data to the NRF52840 (over uart). The NRF is flashed with the SD140 and I'm using the SDK V16.0. The communication protocol of the host is already implemented in the NRF app. As I have to update only the application I want to implement the transport layer for dfu by myself using the dual bank update procedure. I figured out the borders of bank1 and succesfully transfere and store the new application data in bank1 (crosschecked the flash by nrfjprog -readcode). I'm receiving the most important settings (size, app_crc,...) for the new application by the host (they are generated by nrfutils, then extracted ...) which then are written and stored in the settings for bank1 (see code below). But for storing the settings I had to modify the nrf_dfu_settings.c (static bool settings_forbidden_parts_equal_to_backup(uint8_t * p_compare_addr)) to return always true like in this thread. RTT Log tells me that storing the settings worked then.

For the bootloader I took an example and reduced it by the transport layer.

BL main code:

Lauching the bootloader after receiving and storing the new application also works.

But then the bootloader does not detect the new app in bank1 and hangs in dfu mode (see RTT output).

Is it a problem in the settings storage or a non properly working bootloader?

Bootloader RTT output:

[00000000] <info> app: Inside main

[00000000] <debug> app: In nrf_bootloader_init

[00000000] <debug> nrf_dfu_settings: Calling nrf_dfu_settings_init()...

[00000000] <debug> nrf_dfu_flash: Initializing nrf_fstorage_nvmc backend.

[00000000] <debug> nrf_dfu_settings: Using settings page.

[00000000] <debug> nrf_dfu_settings: Copying forbidden parts from backup page.

[00000000] <debug> nrf_dfu_settings: Writing settings...

[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FF000

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FF000, len=1 pages), queue usage: 0

[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FF000, pending 0

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FF000, src=0x20003AEC, len=896 bytes), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FF000, pending 0

[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.

[00000000] <debug> nrf_dfu_settings: Writing settings...

[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FE000

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FE000, len=1 pages), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FE000, pending 0

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FE000, src=0x20003E6C, len=896 bytes), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FE000, pending 0

[00000000] <info> app: Postvalidating update after reset.

[00000000] <debug> nrf_dfu_settings: Writing settings...

[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FF000

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FF000, len=1 pages), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FF000, pending 0

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FF000, src=0x20003AEC, len=896 bytes), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FF000, pending 0

[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.

[00000000] <debug> nrf_dfu_settings: Writing settings...

[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FE000

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FE000, len=1 pages), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FE000, pending 0

[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FE000, src=0x20003E6C, len=896 bytes), queue usage: 1

[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FE000, pending 0

[00000000] <debug> app: Enter nrf_bootloader_fw_activate

[00000000] <info> app: No firmware to activate.

[00000000] <debug> app: App is valid

[00000000] <debug> app: DFU mode requested via GPREGRET.

[00000000] <info> nrf_bootloader_wdt: WDT is not enabled

[00000000] <debug> app: in weak nrf_dfu_init_user

[00000000] <debug> app: timer_stop (0x2000005C)

[00000000] <debug> app: timer_activate (0x2000005C)

[00000000] <info> app: Entering DFU mode.

[00000000] <debug> app: Initializing transports (found: 0)

[00000000] <debug> nrf_dfu_flash: Initializing nrf_fstorage_nvmc backend.

[00000000] <debug> app: Enter main loop

[03932160] <debug> app: timer_stop (0x2000005C)

[03932160] <debug> app: timer_fire (0x2000005C)

[03932161] <info> app: Inactivity timeout.

[03932161] <debug> app: Resetting bootloader.

[03932161] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.

[03932164] <debug> nrf_dfu_settings: Destination settings are identical

[Hung Bui]

Hi Massimo, 

Have you invalide Bank 0 and valid Bank 1 before you switch to bootloader to swap the image ?

We do that in postvalidate_app() function. 

We have an example of doing background DFU on UART which is quite similar to what you are trying to do. It's for SDK v15.3 but should be a good source of reference for your application I think. Please have a look here. The example named nrf_sdk_15.3.0_background_dfu_3.zip

[MassimoF]

Hello Hung,

thank you for your answer. I followed the suggestion of the mentioned thread and instead of invalidating banks by myself, I called nrf_dfu_validation_post_data_execute(bank1_start, app_size);.

Then I had to enabled the ram buffering (int sdk_config.h) and replaced the cc310_backend_hash.c with Vidar's one of the dfu example.

But now I get a non matching hash:

[1;32m<debug> nrf_dfu_validation: Hash verification. start address: 0x55000, size: 0x29AB4
[1;33m<warning> nrf_dfu_validation: Hash verification failed.
[1;32m<debug> nrf_dfu_validation: Expected FW hash:
[1;32m<debug> nrf_dfu_validation:  08 23 00 00 08 19 00 00|.#......
[1;32m<debug> nrf_dfu_validation:  08 0F 00 00 08 05 00 00|........
[1;32m<debug> nrf_dfu_validation:  07 FB 00 00 07 F1 00 00|........
[1;32m<debug> nrf_dfu_validation:  07 E7 00 00 07 DD 00 00|........
[1;32m<debug> nrf_dfu_validation: Actual FW hash:
[1;32m<debug> nrf_dfu_validation:  71 E0 6D D1 E1 12 E9 72|q.m....
[1;32m<debug> nrf_dfu_validation:  ED 40 43 D1 80 6C A6 3E|[email protected].>
[1;32m<debug> nrf_dfu_validation:  3A 12 CB 30 19 8F 94 FD|:..0....
[1;32m<debug> nrf_dfu_validation:  C6 2A 6A 81 F8 5F 7D B1|.*j.._}.

Where is the "expected FW" hash stored? As I just decode and store the incomming hex file (app only from 0x27000) data in bank1 I do no previous hash calculation as reference for this step.

[Hung Bui]

I assume you meant the .dat file ? 
As far as I know there is no overhead or filter. But you may need to double check.

I would suggest the following test. You can modify the original bootloader so that it pause at the swapping phase. So that you know what exactly expected in the bootloader setting in order for the bootloader to swap the image. 

[MassimoF]

In the first step I don't need any app signature and. Is it possible to trigger the image swap in the bootloader without the init_command and validation inside the app?

Now I'm trying to store the settingspage with information about bank1:

memset(&s_dfu_settings.bank_0, 0, sizeof(nrf_dfu_bank_t));
    s_dfu_settings.write_offset = 0;
    s_dfu_settings.bank_layout = NRF_DFU_BANK_LAYOUT_DUAL;
    s_dfu_settings.bank_current = NRF_DFU_CURRENT_BANK_1;
    s_dfu_settings.bank_1.image_size = m_settings_decoding.appsize; // received by host (gen. by nrfutil)
    s_dfu_settings.bank_1.image_crc = m_settings_decoding.app_crc;  // received by host (gen. by nrfutil)
    s_dfu_settings.bank_1.bank_code = NRF_DFU_BANK_VALID_APP;
    s_dfu_settings.bank_0.bank_code = NRF_DFU_BANK_INVALID;
    s_dfu_settings.bank_current = NRF_DFU_CURRENT_BANK_1;

...

err_code = nrf_dfu_settings_write(firmwareDfu_flashOp_cb);

The return value tells me that storing was succesfull. Is there a way to check this by reloading the settings from flash?

After the settings storage i call:

 err_code = sd_power_gpregret_clr(0, 0xFFFFFFFF);
 APP_ERROR_CHECK(err_code);
 nrf_pwr_mgmt_shutdown(NRF_PWR_MGMT_SHUTDOWN_GOTO_DFU);

When entering the Bootloader the settings seem to be not recovered. I added a breakpoint nrf_bootloader_fw_activate() to check the settings but they seem not present:

The Bootloader log shows the wildly copying settings pages but for what?

The Bootloader Logging output was:

[0m[00000000] <info> app: Inside main[0m
[0m[00000000] <debug> app: In nrf_bootloader_init[0m
[0m[00000000] <debug> nrf_dfu_settings: Calling nrf_dfu_settings_init()...[0m
[0m[00000000] <debug> nrf_dfu_flash: Initializing nrf_fstorage_nvmc backend.[0m
[0m[00000000] <debug> nrf_dfu_settings: Using settings page.[0m
[0m[00000000] <debug> nrf_dfu_settings: Copying forbidden parts from backup page.[0m
[0m[00000000] <debug> nrf_dfu_settings: Writing settings...[0m
[0m[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FF000[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FF000, len=1 pages), queue usage: 0[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FF000, pending 0[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FF000, src=0x20003AEC, len=896 bytes), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FF000, pending 0[0m
[0m[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.[0m
[0m[00000000] <debug> nrf_dfu_settings: Writing settings...[0m
[0m[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FE000[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FE000, len=1 pages), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FE000, pending 0[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FE000, src=0x20003E6C, len=896 bytes), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FE000, pending 0[0m
[0m[00000000] <info> app: Postvalidating update after reset.[0m
[0m[00000000] <debug> nrf_dfu_settings: Writing settings...[0m
[0m[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FF000[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FF000, len=1 pages), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FF000, pending 0[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FF000, src=0x20003AEC, len=896 bytes), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FF000, pending 0[0m
[0m[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.[0m
[0m[00000000] <debug> nrf_dfu_settings: Writing settings...[0m
[0m[00000000] <debug> nrf_dfu_settings: Erasing old settings at: 0x000FE000[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_erase(addr=0x0x000FE000, len=1 pages), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash erase success: addr=0x000FE000, pending 0[0m
[0m[00000000] <debug> nrf_dfu_flash: nrf_fstorage_write(addr=0x000FE000, src=0x20003E6C, len=896 bytes), queue usage: 1[0m
[0m[00000000] <debug> nrf_dfu_flash: Flash write success: addr=0x000FE000, pending 0[0m
[0m[00000000] <debug> app: Enter nrf_bootloader_fw_activate[0m
[0m[00000000] <info> app: No firmware to activate.[0m
[0m[00000000] <debug> app: App is valid[0m
[1;33m[00000000] <warning> nrf_dfu_settings: No additional data erased[0m
[0m[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.[0m
[0m[00000000] <debug> nrf_dfu_settings: Destination settings are identical to source, write not needed. Skipping.[0m
[0m[00000000] <debug> app: Running nrf_bootloader_app_start with address: 0x00001000[0m
[0m[00000000] <debug> app: Disabling interrupts. NVIC->ICER[0]: 0x0[0m
[0m[00000000] <info> app: Inside main[0m
[0m[00000000] <debug> app: In nrf_bootloader_init[0m
[0m[00000000] <debug> nrf_dfu_settings: Calling nrf_dfu_settings_init()...[0m
[0m[00000000] <debug> nrf_dfu_flash: Initializing nrf_fstorage_nvmc backend.[0m
[0m[00000000] <debug> nrf_dfu_settings: Using settings page.[0m
[0m[00000000] <debug> nrf_dfu_settings: Copying forbidden parts from backup page.[0m
[0m[00000000] <debug> nrf_dfu_settings: Destination settings are identical to source, write not needed. Skipping.[0m
[0m[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.[0m
[0m[00000000] <debug> nrf_dfu_settings: Destination settings are identical to source, write not needed. Skipping.[0m
[0m[00000000] <debug> app: Enter nrf_bootloader_fw_activate[0m
[0m[00000000] <info> app: No firmware to activate.[0m
[0m[00000000] <debug> app: App is valid[0m
[1;33m[00000000] <warning> nrf_dfu_settings: No additional data erased[0m
[0m[00000000] <info> nrf_dfu_settings: Backing up settings page to address 0xFE000.[0m
[0m[00000000] <debug> nrf_dfu_settings: Destination settings are identical to source, write not needed. Skipping.[0m
[0m[00000000] <debug> app: Running nrf_bootloader_app_start with address: 0x00001000[0m
[0m[00000000] <debug> app: Disabling interrupts. NVIC->ICER[0]: 0x0[0m

[Hung Bui]

Hi Massimo,

I haven't got time to dig into the code to find a solution for you. But could you check the example ? It's made for SDK v15.3 but it shouldn't too difficult to port to v16. 

I think the bootloader setting flash re-write in the log has something to do with the fact that the bootloader setting back up doesn't match with the current bootloader setting.  This will cause the bootloader to copy the bootloader setting backup back to the bootloader setting. 
Also, I'm not sure if you aware of but every time you store bootloader setting you need to calculate the .crc of it. Please have a look at nrf_dfu_settings_write_and_backup() function.

[MassimoF]

Hello Hung,

the example is not what I want to achieve. The transport layer is custom and not comparable with the example. I think the only problem is the storage of the settings.

If I check the flash memory at 0xFF000 after calling nrf_dfu_settings_write() and compare it with the s_dfu_settings struct it does not match. Variables like bank_code, bank_current or current_bank do not match. nrf_defu-settings_write_and_backup() call only nrf_dfu_settings_write as I'm in the App and calling the backup is not allowed.

Is it possible to swap the image without init command (no signature...), just by a correct settings page with app_crc, settings_crc, app_size?

[Hung Bui]

Hi Massimo,

Since you are doing custom transport for DFU, and you are modifying the bootloader anyway, it's all up to you  to modify the bootloader to do what you need.

I don't see any problem modifying the bootloader to allow you to write to bootloader setting backup, or to change the bootloader to ignore hash check, and so on. 
The bootloader is just like a normal application code, the only difference is that it runs before the application.

In addition, even though the transport layer in the example is different from what you are planning to do, the last part to save to bootloader setting and switch to bootloader to swap image should be the same and  you can use that as the reference for your solution.