nrf52832 mesh bricks after OTA update

[Mr Sunshine]

Hello,

for clarification I use the nrf5_SDK_for_Mesh_v2.1.1.

After an OTA update (with use of the nRF toolbox app) the nrf52832 seem to be in a state of constant reboots. After some digging I see that the error happens during the mesh initialization in the "flash_manager_add()" function located at "flash_manager.c". In my case the when updating through OTA the "flash_area_is_valid()" will turn false. In addition the following is written in the function "If the area build fails, it's because we can't fit all the metadata in the action queue. Consider increasing its size."

What can I do to fix this issue? How can I increase this action queue size?  Am I looking in the right direction or am I just fixing a symptom instead of the cause?

[Edvin]

Hello,

 

After an OTA update (with use of the nRF toolbox app) the nrf52832

 This means that you are using a BLE bootloader, and not the mesh bootloader, right?

I don't remember what combination, but I remember that there was one Mesh SDK that used a version of the nRF5 "normal BLE" SDK where this combination was not compatible with one another. The reason was that the nRF5 SDK used a new method of storing the bootloader address, which the Mesh SDK didn't check for.

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

BR,

Edvin

[Mr Sunshine]

hello Edvin,

Edvin said:

This means that you are using a BLE bootloader, and not the mesh bootloader, right?

Yes it uses BLE for updates. The system can be set in a separate DFU mode when using.

Edvin said:

When you have programmed the bootloader and performed the OTA DFU (or alternatively you can generate and flash bootloader settings using "nrfutil settings generate (--help)" if it is tedious to do this every time). What is the return value of BOOTLOADERADDR() in flash_manager_defrag.c. Is it 0xFFFFFFFF or the actual bootloader address?

I don't really understand what you are saying. Where is this "flash_manager_defrag.c" is this a separate file? 

I have generated the bootloader settings file if you are interested in that, but I cannot see this BOOTLOADERADDR() you are mentioning in this file. below the contents of the bootloader settings file:

:020000040007F3
:10E00000445500E70100000000000000000000008F
:10E01000000000000000000020F40100C4627BB397
:10E0200001000000000000000000000000000000EF
:10E0300000000000000000000000000000000000E0
:10E0400000000000000000000000000000000000D0
:0CE05000000000000000000000000000C4
:10F00000445500E70100000000000000000000007F
:10F01000000000000000000020F40100C4627BB387
:10F0200001000000000000000000000000000000DF
:10F0300000000000000000000000000000000000D0
:10F0400000000000000000000000000000000000C0
:0CF05000000000000000000000000000B4
:00000001FF

I hope this helps.

[Edvin]

The size of the nRF52832 is 512kb flash, so I don't think your application is 939kB. The .hex file may be, but this is not translatable to the application size. Could you send the old application .hex file and the new one, and I can check. That is, the old one I can see from your screenshot. If you don't want to send the .hex file, drag and drop the new one to the File Memory layout to the right, and do the same measurement (start and end flash addresses).

Did you come any further as to why the flash_area_is_valid() returns false? If not, is there any way for me to reproduce this? Preferably, a .bat script to perform the DFU is welcome, so I don't need to set it up from scratch. It will be faster for both of us

Best regards,

Edvin

[Mr Sunshine]

Edvin said:

Could you send the old application .hex file and the new one, and I can check.

Currently I'm stuck so I'll give you the hex files:

 flash_new.hex flash_old.hex

Edvin said:

Did you come any further as to why the flash_area_is_valid() returns false?

Sorry wan't able to. I didn't had acces to the J-link, and have to do other work activities. If I look at the flash_area_is_valid() then I see that this goes deeper into scructs. 

Edvin said:

If not, is there any way for me to reproduce this? Preferably, a .bat script to perform the DFU is welcome, so I don't need to set it up from scratch. It will be faster for both of us

Don't you mean .dat file? the generated DFU zip file only includes .dat files and I don't see .bat files. I can give you the zip file so that you can reproduce this through the nRF tools app.

new_DFU.zip

EDIT:

the new DFU will not continuously reboot because I altered the code that It will ignore the first errors at the start of the program. But if you debug it you can see errors occurring during boot. I altered this for debug purposes.

In addition I went deeper into the flash_area_is_valid() code and can confirm that the error occurs at p_area[0].

[Edvin]

Mr Sunshine said:

Don't you mean .dat file? the generated DFU zip file only includes .dat files and I don't see .bat files. I can give you the zip file so that you can reproduce this through the nRF tools app.

 No, I meant a script that will perform the DFU when I run it. This way, if I compile the application, and run the script, it would take a DK and start transferring the image. How do you normally perform the DFU?

And for the new_flash.hex, I was thinking only of the hex file that is generated when you compile the application that you use to generate the DFU image later.

BR,

Edvin

[Mr Sunshine]

Edvin said:

I can give you the zip file so that you can reproduce this through the nRF tools app.

I use the nRF toolbox app on android. 

Edvin said:

And for the new_flash.hex, I was thinking only of the hex file that is generated when you compile the application that you use to generate the DFU image later.

I use the new_DFU.zip file given in my previous message for DFU updates. I can give you the application file without the bootloader and softdevice if that is what you mean.

 6180.application.hex

in addition (I made a edit in my last message) I went deeper into the flash_area_is_valid() code and I see that the error occurs at p_area[0] of the struct "&p_manager->config.p_area[i].metadata".

[Edvin]

Which of the checks in:

static inline bool metadata_is_valid(const flash_manager_metadata_t * p_metadata)
{
    return (p_metadata->metadata_len != 0xFF &&
            p_metadata->metadata_len >= 8 &&
            p_metadata->page_index != 0xFF &&
            p_metadata->pages_in_area != 0xFF);
}

is it that returns false?

And a difficult question: What flash area is it looking at? To answer this, you need to check where is  flash_manager_add() called from in the case where it fails? add_flash_manager(), init_flash_storage() or build_flash_area()? And what is manager_config in that case?

In case:

static void add_flash_manager(void)
{

    static fm_mem_listener_t flash_add_mem_available_struct = {
        .callback = flash_manager_mem_available,
        .p_args = add_flash_manager
    };
    flash_manager_config_t manager_config;
    manager_config.write_complete_cb = flash_write_complete;
    manager_config.invalidate_complete_cb = flash_invalidate_complete;
    manager_config.remove_complete_cb = flash_remove_complete;
    manager_config.min_available_space = WORD_SIZE;
#ifdef ACCESS_FLASH_AREA_LOCATION
    manager_config.p_area = (const flash_manager_page_t *) ACCESS_FLASH_AREA_LOCATION;
#else
    manager_config.p_area = (const flash_manager_page_t *) (((const uint8_t *) dsm_flash_area_get()) - (ACCESS_FLASH_PAGE_COUNT * PAGE_SIZE));
#endif
    manager_config.page_count = ACCESS_FLASH_PAGE_COUNT;
    m_flash_not_ready = true;
    uint32_t status = flash_manager_add(&m_flash_manager, &manager_config);
    if (NRF_SUCCESS != status)
    {
        flash_manager_mem_listener_register(&flash_add_mem_available_struct);
    }
    else
    {
        m_flash_not_ready = false;
    }
}

What is dsm_flash_get()? What address is it pointing to?

In case:

static void build_flash_area(void)
{
    bool success = false;
    flash_manager_config_t manager_config;
    manager_config.write_complete_cb      = flash_write_complete;
    manager_config.invalidate_complete_cb = flash_invalidate_complete;
    manager_config.remove_complete_cb     = flash_remove_complete;
    manager_config.min_available_space    = 0;
    manager_config.p_area = dsm_flash_area_get();
    manager_config.page_count = DSM_FLASH_PAGE_COUNT;

    /* Lock the bearer event handler to ensure that we don't enter and leave the BUILDING state
     * between adding and checking. */
    bearer_event_critical_section_begin();
    if (flash_manager_add(&m_flash_manager, &manager_config) == NRF_SUCCESS)
    {
        /* If we have to build the flash manager, it means that it's new, and there's no metainfo. */
        if (m_flash_manager.internal.state == FM_STATE_BUILDING)
        {
            success = flash_store_metainfo();
        }
        else
        {
            success = true;
        }
    }
    bearer_event_critical_section_end();

    if (success)
    {
        m_flash_is_available = true;
    }
    else
    {
        /* Register the listener and wait for some memory to be freed up before we retry. */
        static fm_mem_listener_t mem_listener = {.callback = flash_mem_listener_callback,
                                                 .p_args = build_flash_area};
        flash_manager_mem_listener_register(&mem_listener);
    }
}

What is: manager_config.p_area = dsm_flash_area_get(); ?

In case:

static void init_flash_storage(void)
{
    flash_manager_config_t config;
    memset(&config, 0, sizeof(config));
    config.min_available_space = 0;
    config.p_area = net_state_flash_area_get();
    config.page_count = NET_FLASH_PAGE_COUNT;
    config.write_complete_cb = flash_write_complete;

    if (flash_manager_add(&m_flash_manager, &config) != NRF_SUCCESS)
    {
        static fm_mem_listener_t mem_listener = {.callback = flash_mem_available,
                                                 .p_args = init_flash_storage};
        flash_manager_mem_listener_register(&mem_listener);
    }
}

What is:

config.p_area = net_state_flash_area_get(); ?

In case you are curious, the net_state_flash_area_get() fetches the bootloader address:

net_state_flash_area_get() -> flash_manager_recovery_page_get() -> flash_manager_defrag_recovery_page_get() -> mp_recovery_area,

which is set in flash_manager_defrag_init():

mp_recovery_area = p_flash_end - FLASH_MANAGER_RECOVERY_PAGE_OFFSET_PAGES - 1; /* pointer arithmetic */

where p_flash_end is:

p_flash_end = (flash_manager_recovery_area_t *) (BOOTLOADERADDR() - PAGE_SIZE);

Which was why I asked for that earlier.

So, what area is it trying to initialize the flash manager on, and which of the checks is it that fails, and what is currently in the flash where you are trying to use?

As for the application .hex file, it has start address 0x26000 and end address 0x4545F, so it is 0x1F45F large (rounded up to 0x20000)

Your screenshot from nRF Connect programmer:

The previous application stops at 0x4BD58 (effectively 0x4C000), and the Flash manager starts at 0x6B000. This means that the free space between the two is 0x6B000-0x4C000 = 0x1F000. But since you are using a BLE bootloader, that should be fine (not background). However, you may want to check that your bootloader saves the correct amount of flash for FDS (flash manager). 

What is your DFU_APP_DATA_RESERVED defined as in sdk_config.h in your bootloader? Depending on what SDK version you use, it may have different, but similar name.

I your case this size should reflect the distance from the bootloader area 0x70000 to the start of your flash manager (0x6B000), so try setting it to 20480 (or alternatively, 5*CODE_PAGE_SIZE).

BR,

Edvin

[Mr Sunshine]

I might have found a solution. I altered in the sdk_config.h the NRF_DFU_SINGLE_BANK_APP_UPDATES variable from 0 to 1 and updated the bootlader separately through the DFU. After that the application update seems to function without the issues. Is single bank mode a wise way to go?

About your questions to the value of some variables, I'm afraid that what you are doing is primarily treating the symptoms instead of the cause. I think that during the OTA update the bootlader overwrites some of the configuration data which will cause the errors. 

Edvin said:

What is your DFU_APP_DATA_RESERVED defined as in sdk_config.h in your bootloader?

the sdk_config.h in the bootloader doesn't seem to have a DFU_APP_DATA_RESERVED, and I don't see anything similar in the file. I'm using the SDK 15.0.0. this version might lack it.

Edvin said:

I your case this size should reflect the distance from the bootloader area 0x70000 to the start of your flash manager (0x6B000), so try setting it to 20480 (or alternatively, 5*CODE_PAGE_SIZE).

How can I Alter this size withoud the DFU_APP_DATA_RESERVED in the sdk_config.h? 

[Edvin]

I see that in SDK15.0.0, the variable is called DFU_APP_DATA_RESERVED, and is defined in nrf_dfu_types.h.

Yes, NRF_DFU_SINGLE_BANK_APP_UPDATES set to 1 may prevent the issue to some degree, but if the application is big enough, the bootloader would start writing over your DFU area instead of throwing an error saying that the app is too big, so I would recommend you set the DFU_APP_DATA_RESERVED correctly.

The reason NRF_DFU_SINGLE_BANK_APP_UPDATES 1 works is that the bootloader will not try to use dual bank, meaning it will delete the application and store the application in the direct placement at first. In this case, it meant that the bootloader didn't use the area of the FDS pages. When it tried to use dual bank, it stored the application in the flash after the original application, because it thought it had space, since the flash manager pages was not protected by DFU_APP_DATA_RESERVED.

Dual bank is better if possible, because then you have a rollback in case something goes wrong during DFU (power loss). If not, the fallback after that would be that the device will start advertising as DfuTarg instead of running the previous application. No crisis, but it is a nice-to-have feature.

So the conclusion is: Set DFU_APP_DATA_RESERVED to protect your flash manager pages. It is a good safety to have, preventing bricked devices, like the update you currently had lead to (if you didn't have a programmer). 

Best regards,

Edvin

[Mr Sunshine]

I have altered the DFU_APP_DATA_RESERVED to be CODE_PAGE_SIZE * 5 as you suggested, and I set the NRF_DFU_SINGLE_BANK_APP_UPDATES  back to 0. The problem seems to be gone if I do a OTA update for first the bootloader and after the application, so thank you for that.

The issue isn't solved if I do an OTA update with a zip file that combines the bootloader, application, and softdevice altogether. It seems that I I use a combined zip that probably the application will be updated first instead of the bootloader. Might it be that I make a mistake when generating a new zip? The code I use is:

 nrfutil pkg generate new_DFU.zip --hw-version 52 --sd-req 0xA8,0xAF --sd-id 0 --softdevice softdevice.hex --bootloader-version 0 --bootloader bootloader.hex --key-file key_node.pem --application-version 0 --application application.hex

[Edvin]

When you do a BL + SD + APP update, you can generate this in one packet, but the process will be split in two. First it will update BL + SD in one go, because the bootloader requires that it always has a compatible SoftDevice.

When this is done, you can see that the device will reset briefly, and nrfutil/nRF Connect will reconnect to the device, and then perform the APP update.

What sort of issue do you see?

Try to only use one --sd-req (the old softdevice, that is already present on the nRF), and then you need to set the --sd-id to the new softdevice version.

But please describe what issue you have? Does the DFU complete? Are you talking about the flash_manager issue, or something else?

[Edvin]

I will leave for vacation today, so if the new problem is not related to the flash manager, I suggest you create a new ticket with a title that describes the issue.

BR,

Edvin