nrf52832 mesh bricks after OTA update

[Mr Sunshine]

Hello,

for clarification I use the nrf5_SDK_for_Mesh_v2.1.1.

After an OTA update (with use of the nRF toolbox app) the nrf52832 seem to be in a state of constant reboots. After some digging I see that the error happens during the mesh initialization in the "flash_manager_add()" function located at "flash_manager.c". In my case the when updating through OTA the "flash_area_is_valid()" will turn false. In addition the following is written in the function "If the area build fails, it's because we can't fit all the metadata in the action queue. Consider increasing its size."

What can I do to fix this issue? How can I increase this action queue size?  Am I looking in the right direction or am I just fixing a symptom instead of the cause?

[Edvin]

Hello,

 

After an OTA update (with use of the nRF toolbox app) the nrf52832

 This means that you are using a BLE bootloader, and not the mesh bootloader, right?

I don't remember what combination, but I remember that there was one Mesh SDK that used a version of the nRF5 "normal BLE" SDK where this combination was not compatible with one another. The reason was that the nRF5 SDK used a new method of storing the bootloader address, which the Mesh SDK didn't check for.

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

BR,

Edvin

[Mr Sunshine]

hello Edvin,

Edvin said:

This means that you are using a BLE bootloader, and not the mesh bootloader, right?

Yes it uses BLE for updates. The system can be set in a separate DFU mode when using.

Edvin said:

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

I don't really understand what you are saying. Where is this "flash_manager_defrag.c" is this a separate file? 

I have generated the bootloader settings file if you are interested in that, but I cannot see this BOOTLOADERADDR() you are mentioning in this file. below the contents of the bootloader settings file:

:020000040007F3
:10E00000445500E70100000000000000000000008F
:10E01000000000000000000020F40100C4627BB397
:10E0200001000000000000000000000000000000EF
:10E0300000000000000000000000000000000000E0
:10E0400000000000000000000000000000000000D0
:0CE05000000000000000000000000000C4
:10F00000445500E70100000000000000000000007F
:10F01000000000000000000020F40100C4627BB387
:10F0200001000000000000000000000000000000DF
:10F0300000000000000000000000000000000000D0
:10F0400000000000000000000000000000000000C0
:0CF05000000000000000000000000000B4
:00000001FF

I hope this helps.

[Edvin]

Sorry. I was a bit short in my previous reply.

I meant the call to  BOOTLOADERADDR() in your mesh application project. it is found in flash_manager_defrag.c.

This is a macro that checks whether or not there is a bootloader present in flash.

In your case, you know that there indeed is a bootloader in the flash, but if this check returns false, then the mesh application will try to use the flash in the bootloader area to store the network data, while it actually should have tried to use the flash below the bootloader instead.

So after performing the OTA DFU, can you please try to set a breakpoint on line 591 in flash_manager_defrag.c:

bool flash_manager_defrag_init(void)
{
#ifdef FLASH_MANAGER_RECOVERY_PAGE
    mp_recovery_area = (flash_manager_recovery_area_t *) FLASH_MANAGER_RECOVERY_PAGE;
#else
    flash_manager_recovery_area_t * p_flash_end;
    if (BOOTLOADERADDR() != BLANK_FLASH_WORD &&     //<-- line 591
        BOOTLOADERADDR() != 0)
    {
        ...

and do some debugging to see what p_flash_end ends up being after these checks.

BOOTLOADERADDR() is a macro:

#define BOOTLOADERADDR() (NRF_UICR->NRFFW[0])

So it reads the UICR register for the bootloader address.

Please make sure that you start with a completely clean flash (nrfjprog --eraseall) before you flash your bootloader, softdevice and application, so that you don't have anything old data in the flash from previous testing. The flash should not be corrupted after a DFU.

You are using the Mesh SDK 2.1.1. Was the previous application also from the same SDK version, or did you do a version upgrade?

BR,

Edvin

[Mr Sunshine]

thanks for your extended explanation. 

I did some debugging and I got the following:

BOOTLOADERADDR() = 0x70000

p_flash_end = 454656

Edvin said:

You are using the Mesh SDK 2.1.1. Was the previous application also from the same SDK version, or did you do a version upgrade?

The update is in the same SDK version. One thing that might be interesting is that the new application is on Release configuration, while the old one was in Debug.

Edvin said:

Please make sure that you start with a completely clean flash (nrfjprog --eraseall) before you flash your bootloader,

Don't worry about that. when I test a new DFU I clear and flash the original application to the device.

In addition I have tried to also update the softdevice and bootloader through the DFU to see if that solves the issue, but it still seems to get errors. I currently use this code to generate the package, did I miss something?

nrfutil pkg generate package_to_generate.zip --hw-version 52 --sd-req 0xA8,0xAF --sd-id 0 --softdevice softdevice.hex --bootloader-version 0 --bootloader bootloader.hex --key-file ./key_file.pem --application-version 0 --application application.hex

[Edvin]

Not that I can tell. Please try to check why the flash_area_is_valid() returns false, and what address it is trying to check, what is present there, and what it is expecting.

I am sorry for pushing this over on you, but we are very short staffed due to summer holidays in Norway, so in order to give some pointers to everyone, I can't spend too much time digging. These are the action points that I would investigate if I were to find out why it is not working.

Best regards,

Edvin

[Mr Sunshine]

Edvin said:

Please try to check why the flash_area_is_valid()

how can I easily check this? It becomes quite a maze if I dig deeper in the function. 

In addition I have scanned the memory with nRF Connect and see this happening. Might this help to figure out the issue? What you see is one is made before the update which shows a blank space, and the other is after the update which shows almost no blank space.

[Edvin]

Have you tried to flash the new application manually? Does that work?

What is the free space (the grey area) in your screenshot to the left? Do you have the addresses? And what is the size of your new application? does it fit within that grey space?

Are you able to debug after the DFU?

 

Mr Sunshine said:

how can I easily check this? It becomes quite a maze if I dig deeper in the function. 

 Try to set a breakpoint in flash_area_is_valid(). Try to step a bit, and set new breakpoints to find out why the flash_area_is_valid() returns false.

Is metadata_is_valid() returning false? What is your page_count? Does it assert in any of these?

NRF_MESH_ASSERT(p_manager->config.p_area[i].metadata.page_index == i);
NRF_MESH_ASSERT(p_manager->config.p_area[i].metadata.pages_in_area == p_manager->config.page_count);

Which of these checks is it that fails? 

Remember that you may need to turn off optimization in order to debug this properly. 

Have you tried to port the application to a later SDK version? Is it reproducible there?

[Mr Sunshine]

Edvin said:

Have you tried to flash the new application manually? Does that work?

Flashing manually will fix the problem yes. But I want to update it through DFU. Currently some customers of us already have this product, so we don't want to call these back for a update if an OTA update is also possible. 

Edvin said:

What is the free space (the grey area) in your screenshot to the left?

I don't know what it is for, it was like this during development. I assumed it was the free space required for updates. All I know is that red is bootloader, black is bootloader settings, green is application and blue is soft device.

Edvin said:

Have you tried to port the application to a later SDK version? Is it reproducible there?

I have in the past but failed. Porting to another SDK isn't easily done, and from the experience it's probably easier to redo the project if I want to change the SDK.

I will respond later about flash_area_is_valid(). 

[Edvin]

Mr Sunshine said:

I don't know what it is for, it was like this during development. I assumed it was the free space required for updates. All I know is that red is bootloader, black is bootloader settings, green is application and blue is soft device.

 That is correct. But when you hover your mouse over the different fields. What is the end address of your application, and what is the start address of your FDS (the three green lines)?

And finally, what is the size of the application that you are updating to?

If the application image is too big to fit, perhaps it starts eating into the FDS pages. If so, you need to have a temporary middle application:

Original application -> DFU -> middle application -> DFU -> final application 

where the middle application is very reduced, much like the DFU example in the SDK.

Best regards,

Edvin

[Mr Sunshine]

Edvin said:

That is correct. But when you hover your mouse over the different fields. What is the end address of your application, and what is the start address of your FDS (the three green lines)?

I made the following images, I hope that these are clear enough:

for some clarification, the following colors are according to nRF Connect:

• first black is undefined
• second black is "MBR Paramters" 
• red is "bootloader"
• all of green is the "Application"
• blue is "Softdevice"
• orange is "MBR or Application"

It seems as if the DFU update overwrites some spaces of the Application.

Edvin said:

And finally, what is the size of the application that you are updating to?

if I look at the .hex file used to flash the original program the size is 939 KB. The new one, although not being flashed, is 864 KB. Reason why the new one is smaller is due to it being on Release configuration while toe original was on debug configuration. I don't know where I have to look to when looking for the file size for the DFU.

Edvin said:

If the application image is too big to fit, perhaps it starts eating into the FDS pages. If so, you need to have a temporary middle application:

I find this quite odd, current configuration of the devices I use is through a mesh with multiple nodes and one NUS to connect too with a Smart Device. This NUS is bigger in size than the nodes but doesn't show any issues during and after the OTA update while the nodes does. So if the image is too big than this should have surely shown on the NUS, but it currently doesn't. The NUS uses the same nrf52832 chip.

[Edvin]

The size of the nRF52832 is 512kb flash, so I don't think your application is 939kB. The .hex file may be, but this is not translatable to the application size. Could you send the old application .hex file and the new one, and I can check. That is, the old one I can see from your screenshot. If you don't want to send the .hex file, drag and drop the new one to the File Memory layout to the right, and do the same measurement (start and end flash addresses).

Did you come any further as to why the flash_area_is_valid() returns false? If not, is there any way for me to reproduce this? Preferably, a .bat script to perform the DFU is welcome, so I don't need to set it up from scratch. It will be faster for both of us

Best regards,

Edvin