OTA Thread DFU: Merge SDK example with custom application

[AS_Kalpa]

Hi all!

I'm able to fully test the SDK example https://infocenter.nordicsemi.com/index.jsp?topic=%2Fsdk_tz_v3.2.0%2Fthread_example_dfu.html and it works pretty well.

Now, the next step is the integration of this OTA DFU over Thread example in my custom application (that already exploits Thread and MQTT-SN protocol).

I haven't found a tutorial that explains step-by-step this kind of operation. 

Basically I need some help to integrate the DFU client with the user application. I think the key point is the IoT SDK CoAP library that implement a DFU algorithm that should run concurrently to the user application, but I'm a bit confused about merging process.

Thanks in advance,

Alessio

[AS_Kalpa]

I think I need some clarifications about flashing different FW components.

In order to have the OTA DFU example working, I have to flash:

• mbr_nrf52_2.4.1_mbr.hex (MBR)

• nrf52840_xxaa_mbr.hex (BOOTLOADER)

• dfu_client.hex (EXAMPLE HEX)

In order to have my application working, I have to flash:

• s140_nrf52_7.0.1_softdevice.hex (SOFTDEVICE)

• my_app.hex (APPLICATION HEX)

At the moment, I have merged the two application codes and, let's say, I have the  

my_app_DFU.hex

I have tried to flash all the components:

1. mbr_nrf52_2.4.1_mbr.hex (MBR)

2. nrf52840_xxaa_mbr (BOOTLOADER)

3. s140_nrf52_7.0.1_softdevice.hex (SOFTDEVICE)

4. my_app_DFU.hex (APPLICATION+DFU HEX)

I received back this when I flash the SD

and this when I flash the application

Moreover, the dongle is stucked with the green led lit up.

Maybe I am supposed to do some mergehex operations?

Thanks,

Alessio 

 

[Jørgen Holmefjord]

Hi,

You do not need to flash both the MBR and Softdevice HEX-files. The MBR is merged into the softdevice, and you will find that if you do a compare, the first page (0x0 - 0x1000) will be identical.

The Bootloader needs information about the application in order to verify that it is a valid application. This means that you either needs to generate the settings using nrfutil, or create a DFU packet of the application and "flash" it using the DFU process. 

I recommend that you follow the instructions in the Testing section of the Thread Secure OTA DFU Example, especially steps 3.c-d.

Best regards,
Jørgen 

[AS_Kalpa]

UPDATE

I found that I didn't have this macro on my custom app

while It is present in the DFU example.

I added it and now the FW crashes when I start the DFU process.

The blue mark indicates where the crash happens.

I think that macro should be in my app, so the problem is that the app crashes.

Thanks in advance,

Alessio

[AS_Kalpa]

UPDATE

I think I found the bug finally.

My problem is that when the board starts the update, it starts erasing flash memory where there is the Softdevice so It crashes.

I had a try with a version of my application that doesn't exploit the SD and the DFU starts normally.

I think something must be changed in the DFU example code to adapt it with a custom application that includes the softdevice.

I don't wanna update also the SD, but just the application.

Do you have any ideas which are the required steps? 

I can attach the debug during the crash

Seems like there is a problem with 0x000FF000 memory slot, where there is the bootloader_settings_page

IIs it normal that the DFU application tries to erase settings memory space?

Moreover I don't know if it is normal this data when I generate the settings with

$ nrfutil settings generate --family NRF52840 --application SAMBA_Node.hex --application-version 1 --bootloader-version 1 --bl-settings-version 2 settings.hex

I think i'm using a bootloader compiled without the Softdevice options set...

[AS_Kalpa]

UPDATE

Also if I use the NOSD version of my application I have the crash.

The DFU process stucks at 33% 

and my client starts rebooting continuosly.

This is the log of the client

As with the SD version, it tries to erase and write at 0xFF000 where there is the bootloader_setting_page and the FW crashes.

I think the new data written in 0XFF000 are wrong in some way and then the process can't go ahead.

Anyone has a tip for this? I'm out of ideas.

[Jørgen Holmefjord]

Are you using the same sdk_config.h configurations in your application as the Thread DFU client example? If you could post your full project, it would be easier to debug the application and help you find the root cause of the issue. If you do not want to share in public, I can convert the case to a private one.

[AS_Kalpa]

Hi Jorgen,

unfortunatly at the moment I can't share the project with third parties, I need to check the policy with my company.

I compared the Thread DFU client sdk_config (that I copied-pasted in my project) with the one of my project. I found these main differences (other small differences are about LOG but I think they are useless)

This is what I have in the Thread DFU client sdk_config:

// <e> MEM_MANAGER_ENABLED - mem_manager - Dynamic memory allocator
//==========================================================
#ifndef MEM_MANAGER_ENABLED
#define MEM_MANAGER_ENABLED 1
#endif
// <o> MEMORY_MANAGER_SMALL_BLOCK_COUNT - Size of each memory blocks identified as 'small' block.  <0-255> 

#ifndef MEMORY_MANAGER_SMALL_BLOCK_COUNT
#define MEMORY_MANAGER_SMALL_BLOCK_COUNT 8
#endif

// <o> MEMORY_MANAGER_SMALL_BLOCK_SIZE -  Size of each memory blocks identified as 'small' block. 
// <i>  Size of each memory blocks identified as 'small' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_SMALL_BLOCK_SIZE
#define MEMORY_MANAGER_SMALL_BLOCK_SIZE 128
#endif

// <o> MEMORY_MANAGER_MEDIUM_BLOCK_COUNT - Size of each memory blocks identified as 'medium' block.  <0-255> 

#ifndef MEMORY_MANAGER_MEDIUM_BLOCK_COUNT
#define MEMORY_MANAGER_MEDIUM_BLOCK_COUNT 4
#endif

// <o> MEMORY_MANAGER_MEDIUM_BLOCK_SIZE -  Size of each memory blocks identified as 'medium' block. 
// <i>  Size of each memory blocks identified as 'medium' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_MEDIUM_BLOCK_SIZE
#define MEMORY_MANAGER_MEDIUM_BLOCK_SIZE 256
#endif

// <o> MEMORY_MANAGER_LARGE_BLOCK_COUNT - Size of each memory blocks identified as 'large' block.  <0-255> 

#ifndef MEMORY_MANAGER_LARGE_BLOCK_COUNT
#define MEMORY_MANAGER_LARGE_BLOCK_COUNT 2
#endif

// <o> MEMORY_MANAGER_LARGE_BLOCK_SIZE -  Size of each memory blocks identified as 'large' block. 
// <i>  Size of each memory blocks identified as 'large' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_LARGE_BLOCK_SIZE
#define MEMORY_MANAGER_LARGE_BLOCK_SIZE 1024
#endif

This is what I have in my application:

// <e> MEM_MANAGER_ENABLED - mem_manager - Dynamic memory allocator
//==========================================================
#ifndef MEM_MANAGER_ENABLED
#define MEM_MANAGER_ENABLED 1
#endif
// <o> MEMORY_MANAGER_SMALL_BLOCK_COUNT - Size of each memory blocks identified as 'small' block.  <0-255> 


#ifndef MEMORY_MANAGER_SMALL_BLOCK_COUNT
#define MEMORY_MANAGER_SMALL_BLOCK_COUNT 64
#endif

// <o> MEMORY_MANAGER_SMALL_BLOCK_SIZE -  Size of each memory blocks identified as 'small' block. 
// <i>  Size of each memory blocks identified as 'small' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_SMALL_BLOCK_SIZE
#define MEMORY_MANAGER_SMALL_BLOCK_SIZE 64
#endif

// <o> MEMORY_MANAGER_MEDIUM_BLOCK_COUNT - Size of each memory blocks identified as 'medium' block.  <0-255> 


#ifndef MEMORY_MANAGER_MEDIUM_BLOCK_COUNT
#define MEMORY_MANAGER_MEDIUM_BLOCK_COUNT 16
#endif

// <o> MEMORY_MANAGER_MEDIUM_BLOCK_SIZE -  Size of each memory blocks identified as 'medium' block. 
// <i>  Size of each memory blocks identified as 'medium' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_MEDIUM_BLOCK_SIZE
#define MEMORY_MANAGER_MEDIUM_BLOCK_SIZE 256
#endif

// <o> MEMORY_MANAGER_LARGE_BLOCK_COUNT - Size of each memory blocks identified as 'large' block.  <0-255> 


#ifndef MEMORY_MANAGER_LARGE_BLOCK_COUNT
#define MEMORY_MANAGER_LARGE_BLOCK_COUNT 4
#endif

// <o> MEMORY_MANAGER_LARGE_BLOCK_SIZE -  Size of each memory blocks identified as 'large' block. 
// <i>  Size of each memory blocks identified as 'large' block. Memory block are recommended to be word-sized.

#ifndef MEMORY_MANAGER_LARGE_BLOCK_SIZE
#define MEMORY_MANAGER_LARGE_BLOCK_SIZE 512
#endif

Do you think this could have an impact?

Another question, I thought that the DFU process (at least the single bank approach) was carried out by bootloader (so the application is stopped). So what's the meaning of this macro? 

// <q> NRF_DFU_IN_APP  - Specifies that this code is in the app, not the bootloader, so some settings are off-limits.
 

// <i> Enable this to disable writing to areas of the settings that are protected
// <i> by the bootlader. If this is not enabled in the app, certain settings write
// <i> operations will cause HardFaults or will be ignored. Enabling this option
// <i> also causes postvalidation to be disabled since this is meant to be done
// <i> in the bootloader. NRF_BL_DFU_ALLOW_UPDATE_FROM_APP must be enabled in the bootloader.

#ifndef NRF_DFU_IN_APP
#define NRF_DFU_IN_APP 1
#endif

And why it is called "background" DFU if the application is stopped anyway?

If the application is carrying out the DFU process is pretty easy to understand that I can't use the single bank approach (since che app can't erase itself if it is running). I need some clarifications about this.

Thanks in advance,

Alessio

[Jørgen Holmefjord]

Hi,

AS_Kalpa said:

unfortunatly at the moment I can't share the project with third parties, I need to check the policy with my company.

Ok, it would help a lot if your could be allowed to share the project. Everything is treated confidentially by Nordic, and other users will not have access to the content of this ticket if I convert it to private.

AS_Kalpa said:

Do you think this could have an impact?

It could be related, check out the suggestions in this post.

AS_Kalpa said:

Another question, I thought that the DFU process (at least the single bank approach) was carried out by bootloader (so the application is stopped). So what's the meaning of this macro? 

The DFU libraries contains code for both Bootloader and application parts of the process. This config is used to determine if the build is an application or the bootloader.

AS_Kalpa said:

And why it is called "background" DFU if the application is stopped anyway?

Thread secure DFU only supports dual-bank DFU. In order to support single-bank DFU, the transport layer (Thread protocol in this case) must be supported in the bootloader, if not you risk "bricking" the device if the DFU process fails. The OpenThread libraries are too large to fit in the bootloader, and are not a separate module (like for instace our BLE softdevice, which is stored in a separate location in flash, fully detached from the application. So for Thread DFU, the application is not stopped until the entire new application is received. When the application is received, the application will reset and bootloader will handle swapping of application images.

AS_Kalpa said:

If the application is carrying out the DFU process is pretty easy to understand that I can't use the single bank approach (since che app can't erase itself if it is running). I need some clarifications about this.

Yes, the application is responsible for receiving the DFU image. Bootloader is responsible for swapping images.

Best regards,
Jørgen

[AS_Kalpa]

Hi Jorgen,

Jørgen Holmefjord said:

Thread secure DFU only supports dual-bank DFU. In order to support single-bank DFU, the transport layer (Thread protocol in this case) must be supported in the bootloader, if not you risk "bricking" the device if the DFU process fails. The OpenThread libraries are too large to fit in the bootloader, and are not a separate module (like for instace our BLE softdevice, which is stored in a separate location in flash, fully detached from the application. So for Thread DFU, the application is not stopped until the entire new application is received. When the application is received, the application will reset and bootloader will handle swapping of application images.

So are you saying that I can't perform a single bank DFU with Thread protocol?

How can I manage the issue that in the flash I can't storage the current FW and the DFU image (lack of memory space)?

I can attach some debug log about my error

Thanks,

Alessio

EDIT:

I tried with a DFU image that fits my memory space with the NRF_DFU_FORCE_DUAL_BANK_APP_UPDATES macro set and I have the crash problem anyway. So probably I'm facing at the moment a different problem than the lack of memory space. 

[AS_Kalpa]

UPDATE

I want to summarize the main issues I'm facing at the moment so people can help with a clear vision of my scenario

• More theoretical issue: can I implement single-bank approach exploiting Thread DFU example? I don't have enough space in my flash to store both old FW and new DFU image so dual-bank approach is unfeasible. Any workaround to this? 
• Let's pretend that I can implement a dual bank approach (I'm using a dummy DFU image so it can fit in my flash and I can isolate the issue explained above). I had some tries with two version of my application: an older one that exploits BLE and Thread protocol in a "switched mode" and a more recent version where I implemented the multiprotocol approach (BLE and Thread together).
  The DFU starts correctly with the first version (the older one). It works (reset and run the new application) only if I put --sd-req 0x00 when I generate the package, if I put --sd-req 0xCA (that is the SD I'm using in my application) the DFU starts and ends, but the new application is not run.
  
  
  
  Instead, the DFU doesn't start at all with the second version (the more recent one with the multiprotocol implementation)
  
   
  
  The main difference between the two versions is how I exploit the SD, so I'm pretty confident that my issue is someway related to SD.
  Moreover, as you can see from debug logs, the cache_adress changes between the two versions: in the switched case it is 0xC2000 (that is the first free flash page after my current flashed fw), in the multiprotocol case it is 0x27000 (that is the start of my current flashed fw). I think 0xC2000 should be the correct adress in a dual-bank scenario (let me know if I'm wrong), but I can't get why in the multiprotocol version the starting adress is 0x27000 (I think the solution crashes because the application tries to overwrite itself someway). 

Thanks in advance,

Alessio

[Jørgen Holmefjord]

Hi,

I believe that this post sums up your questions and options regarding single bank DFU.

Do you support updates over BLE in the bootloader in any of the versions, or only over Thread? The sd_req option may not be needed it this is not supported, see Updates without a SoftDevice.

I think the issue in the second application is that the "Bank 0 size" field is set to 0. This indicates that there is no code in bank 0, and the new application is placed directly after the softdevice (which has a size of 0x27000). You mention before tha the Bootloader settings page (0xFF000) was erased, can you try to read out this page to see if it is blank or has incorrect size field for bank 0?

>nrfjprog --memrd 0xFF000 --w 32 --n 0x1000
0x000FF000: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF   |................|
0x000FF010: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF   |................|
0x000FF020: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF   |................|

If the settings-page is valid, you may need to debug your application to see where the fields are cleared in s_dfu_settings.

Best regards,
Jørgen