nrf52832 mesh bricks after OTA update

[Mr Sunshine]

Hello,

for clarification I use the nrf5_SDK_for_Mesh_v2.1.1.

After an OTA update (with use of the nRF toolbox app) the nrf52832 seem to be in a state of constant reboots. After some digging I see that the error happens during the mesh initialization in the "flash_manager_add()" function located at "flash_manager.c". In my case the when updating through OTA the "flash_area_is_valid()" will turn false. In addition the following is written in the function "If the area build fails, it's because we can't fit all the metadata in the action queue. Consider increasing its size."

What can I do to fix this issue? How can I increase this action queue size?  Am I looking in the right direction or am I just fixing a symptom instead of the cause?

[Edvin]

Hello,

 

After an OTA update (with use of the nRF toolbox app) the nrf52832

 This means that you are using a BLE bootloader, and not the mesh bootloader, right?

I don't remember what combination, but I remember that there was one Mesh SDK that used a version of the nRF5 "normal BLE" SDK where this combination was not compatible with one another. The reason was that the nRF5 SDK used a new method of storing the bootloader address, which the Mesh SDK didn't check for.

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

BR,

Edvin

[Mr Sunshine]

hello Edvin,

Edvin said:

This means that you are using a BLE bootloader, and not the mesh bootloader, right?

Yes it uses BLE for updates. The system can be set in a separate DFU mode when using.

Edvin said:

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

I don't really understand what you are saying. Where is this "flash_manager_defrag.c" is this a separate file? 

I have generated the bootloader settings file if you are interested in that, but I cannot see this BOOTLOADERADDR() you are mentioning in this file. below the contents of the bootloader settings file:

:020000040007F3
:10E00000445500E70100000000000000000000008F
:10E01000000000000000000020F40100C4627BB397
:10E0200001000000000000000000000000000000EF
:10E0300000000000000000000000000000000000E0
:10E0400000000000000000000000000000000000D0
:0CE05000000000000000000000000000C4
:10F00000445500E70100000000000000000000007F
:10F01000000000000000000020F40100C4627BB387
:10F0200001000000000000000000000000000000DF
:10F0300000000000000000000000000000000000D0
:10F0400000000000000000000000000000000000C0
:0CF05000000000000000000000000000B4
:00000001FF

I hope this helps.

[Edvin]

Sorry. I was a bit short in my previous reply.

I meant the call to  BOOTLOADERADDR() in your mesh application project. it is found in flash_manager_defrag.c.

This is a macro that checks whether or not there is a bootloader present in flash.

In your case, you know that there indeed is a bootloader in the flash, but if this check returns false, then the mesh application will try to use the flash in the bootloader area to store the network data, while it actually should have tried to use the flash below the bootloader instead.

So after performing the OTA DFU, can you please try to set a breakpoint on line 591 in flash_manager_defrag.c:

bool flash_manager_defrag_init(void)
{
#ifdef FLASH_MANAGER_RECOVERY_PAGE
    mp_recovery_area = (flash_manager_recovery_area_t *) FLASH_MANAGER_RECOVERY_PAGE;
#else
    flash_manager_recovery_area_t * p_flash_end;
    if (BOOTLOADERADDR() != BLANK_FLASH_WORD &&     //<-- line 591
        BOOTLOADERADDR() != 0)
    {
        ...

and do some debugging to see what p_flash_end ends up being after these checks.

BOOTLOADERADDR() is a macro:

#define BOOTLOADERADDR() (NRF_UICR->NRFFW[0])

So it reads the UICR register for the bootloader address.

Please make sure that you start with a completely clean flash (nrfjprog --eraseall) before you flash your bootloader, softdevice and application, so that you don't have anything old data in the flash from previous testing. The flash should not be corrupted after a DFU.

You are using the Mesh SDK 2.1.1. Was the previous application also from the same SDK version, or did you do a version upgrade?

BR,

Edvin

[Mr Sunshine]

thanks for your extended explanation. 

I did some debugging and I got the following:

BOOTLOADERADDR() = 0x70000

p_flash_end = 454656

Edvin said:

You are using the Mesh SDK 2.1.1. Was the previous application also from the same SDK version, or did you do a version upgrade?

The update is in the same SDK version. One thing that might be interesting is that the new application is on Release configuration, while the old one was in Debug.

Edvin said:

Please make sure that you start with a completely clean flash (nrfjprog --eraseall) before you flash your bootloader,

Don't worry about that. when I test a new DFU I clear and flash the original application to the device.

In addition I have tried to also update the softdevice and bootloader through the DFU to see if that solves the issue, but it still seems to get errors. I currently use this code to generate the package, did I miss something?

nrfutil pkg generate package_to_generate.zip --hw-version 52 --sd-req 0xA8,0xAF --sd-id 0 --softdevice softdevice.hex --bootloader-version 0 --bootloader bootloader.hex --key-file ./key_file.pem --application-version 0 --application application.hex

[Edvin]

Not that I can tell. Please try to check why the flash_area_is_valid() returns false, and what address it is trying to check, what is present there, and what it is expecting.

I am sorry for pushing this over on you, but we are very short staffed due to summer holidays in Norway, so in order to give some pointers to everyone, I can't spend too much time digging. These are the action points that I would investigate if I were to find out why it is not working.

Best regards,

Edvin

[Mr Sunshine]

Edvin said:

Please try to check why the flash_area_is_valid()

how can I easily check this? It becomes quite a maze if I dig deeper in the function. 

In addition I have scanned the memory with nRF Connect and see this happening. Might this help to figure out the issue? What you see is one is made before the update which shows a blank space, and the other is after the update which shows almost no blank space.

[Edvin]

Have you tried to flash the new application manually? Does that work?

What is the free space (the grey area) in your screenshot to the left? Do you have the addresses? And what is the size of your new application? does it fit within that grey space?

Are you able to debug after the DFU?

 

Mr Sunshine said:

how can I easily check this? It becomes quite a maze if I dig deeper in the function. 

 Try to set a breakpoint in flash_area_is_valid(). Try to step a bit, and set new breakpoints to find out why the flash_area_is_valid() returns false.

Is metadata_is_valid() returning false? What is your page_count? Does it assert in any of these?

NRF_MESH_ASSERT(p_manager->config.p_area[i].metadata.page_index == i);
NRF_MESH_ASSERT(p_manager->config.p_area[i].metadata.pages_in_area == p_manager->config.page_count);

Which of these checks is it that fails? 

Remember that you may need to turn off optimization in order to debug this properly. 

Have you tried to port the application to a later SDK version? Is it reproducible there?