Mesh assert on flash_area_build()

RE: Mesh assert on flash_area_build()

Edvin — Tue, 18 Jul 2023 12:54:08 GMT

Hello,

Sorry for the late reply. I have been out for vacation, and just checking by.

It may be that this is the case. In general, it shouldn't happen, and there are recovery mechanisms that should take care of this, but we have seen that it fails before. What I recommend you to do is to add a piece of recovery for this scenario as well. If the page is not blank, and metadata_is_valid() returns false, then erase the page, and reboot (NVIC_SystemReset()), and then I believe everything should go back to normal. By the looks of things, it looks like the page should be empty, so either it was attempted erased, or it was erased, and attempted written to, possibly right before a power out (brownout) so I assume that the best option here is to erase the page, and try again.

Also, I am not sure whether you are doing this or not, but try avoiding writing to flash directly after booting up the application. If possible, add a delay of some seconds before doing so. The reason is that even if the flash manager should recover from a brownout, it may fail if it is in the middle of a recovery, and then looses power again (and again, and again...). This may be the case if your battery is running low. The chip draws current, causing the battery voltage to drop below the threshold, and the nRF shuts off. When it does, the battery voltage increases a tiny bit, because it has no current being drawn, enough for the nRF to power back on. Then you can get stuck in this cycle for a while before the battery voltage actually goes low enough for the nRF to stop power cycling.

If you are writing to flash on every boot, then the chance for it to be disrupted in a flash write operation increases significantly during this power cycle stage. Adding e.g. 1 second delay will reduce this alot!

NB: I will be out for a couple of weeks, so if you are in need of quick replies, I suggest you create a new ticket, and you can refer to this one.

Best regards,

Edvin

RE: Mesh assert on flash_area_build()

wataru_m — Thu, 29 Jun 2023 11:44:39 GMT

Hi,

Thank you for your reply.

My apologies as well.
I was confused because I did not understand the format of the hex file.
I understood what you meant.

[quote userid="26071" url="~/f/nordic-q-a/100644/mesh-assert-on-flash_area_build/432868"]On the device to your left. can you try to run nrfjprog --recover, and read out the flash again? Does it look the same, or will it be all 0xFF's then?[/quote]

I tried it and everything turned out to be FF.
From this, it seems that there is no physical damage.

Then, after chip-erasing, we wrote softdevice, bootloader, application, and devicepage to confirm that the device works as expected. (Confirmed by logs on the RTT viewer)

From that state, I tried writing code.hex, ram.hex, and uicr.hex, which I had previously dumped, using nrfjprog --program, and the assertion occurred in the same place as initially.

From the device on which we were able to reproduce the assertion, we tried to dump the code with nrfjprog --readcode as well, but it did not exactly match the hex file that we had originally dumped.

I was getting more and more confused as to what I was checking.
Is it safe to assume that the flash state was rewritten to something unintended because the battery voltage dropped and the power was turned off while some unfortunate process was taking place, as we initially suspected?

Best regards,
Wataru

RE: Mesh assert on flash_area_build()

Edvin — Mon, 26 Jun 2023 08:07:20 GMT

Sorry for being unclear.

[quote user="wataru_m"]I did not understand the meaning of "the exact same flash on another device"[/quote]

I meant that if you read out the flash on one device, and store it in a hex file, then you can program another nRF with that .hex file and you should be able to reproduce the issue there.

[quote user="wataru_m"]

What can be confirmed by doing this?

[/quote]

If you upload the hex file here, in addition to your application, I would be able to reproduce and even debug the issue.

That being said. It looks like you have some non-erased flash. Possibly even broken.

On the device to your left. can you try to run nrfjprog --recover, and read out the flash again? Does it look the same, or will it be all 0xFF's then?

Best regards,

Edvin

RE: Mesh assert on flash_area_build()

wataru_m — Thu, 22 Jun 2023 11:35:23 GMT

Hi Edvin,

Thank you for your reply, and I'm sorry for late reply.
Sorry, our application is difficult to provide.

I dumped the code on the device that failed after running in the customer's environment and another device with the same FW written to it for comparison.

Attached below is the result of the comparison near 0x000f5000.
I may be looking at the hex file wrong, but I am checking the value of the address following ":10", which is "5000".
(There were 16 addresses with the address "5000", and we have confirmed the 16th one.
We expect that this is the information for the address 0x000f5000.)

On the left is the failed device and on the right is another device prepared for comparison.

I don't know what information is being written, but most of the failed devices are marked "FF". Some of them are marked "EF". (Broken?)

I did not understand the meaning of "the exact same flash on another device"
Do you mean that the dumped hex file (using --readcode) is written to the device again using nrfjprog?
What can be confirmed by doing this?

We have not enabled the power fail comparator.

Best regards,
Wataru

RE: Mesh assert on flash_area_build()

Edvin — Fri, 16 Jun 2023 12:52:19 GMT

By the way, did you enable the power fail comparator in your application? See link:

https://infocenter.nordicsemi.com/index.jsp?topic=%2Fps_nrf52840%2Fpower.html&anchor=unique_726894245

There is an issue there where if this is enabled, and you try to write to flash, it will report that the flash write was successful but the flash is not actually written. This in itself is not an issue that should lead to corrupted flash, but we have seen incidents where this event occurring while you are in the process of writing to flash, that you can get some bitfaults, which could lead to corrupted flash.

BR,

Edvin

RE: Mesh assert on flash_area_build()

Edvin — Fri, 16 Jun 2023 12:32:10 GMT

[quote user="wataru_m"]I am thinking that the trigger for this condition could be a dead battery, but can you give me your opinion as Nordic?[/quote]

That could be, but the flash manager should be able to handle power outages. However, there may be a combination of events, timing, in combination with a brownout that could trigger a bug that we are not aware of.

[quote user="wataru_m"]How can I verify the flash address I am trying to use in this debugging environment?
[/quote]

It looks like it is on address 0x000f5000, p_area.

If you read out the flash using "nrfjprog --readcode my_flash_dump.hex" it will store the flash on the nRF chip to my_flash_dump.hex. You can use this to re-program another DK (or I can, if I can get a copy of your application, so that I can build it here in my office desk), and do some digging. If the issue is that one of the flash pages are corrupt after some period of time, the exact same flash on another device should reproduce the issue, because it would contain the same corrupted flash page.

That would require you to still have the failing piece, and that you didn't re-program it.

Best regards,

Edvin

RE: Mesh assert on flash_area_build()

wataru_m — Thu, 15 Jun 2023 09:46:08 GMT

Hi Edvin,

Thank you for your reply.

I am sorry, but I do not have an overview of the processes taking place during initialization.
(Is the call stack flow that you are pasting into the ticket reading provisioning information, etc...?)

Aside from that, I tried writing the same bootloader and firmware to another board, provisioning it to the same network, and seeing if the program would run in the same way as the board where the problem occurred.
(I specifically checked the flow during initialization.)

The process proceeded in the same way until about halfway through.
However, the flash_area_is_valid() function at line 834 of flash_manager.c returned true, confirming that the process proceeded to a different branch from the board where the problem occurred.
This means that the flash_area_build function that generated the assertion will not be executed.

The inside of p_manager (a structure of flash_manager_t type) after the flash_manager_add() function is completed is shown in the figure below.

From these confirmed results, for some reason, the flash condition? I am guessing that the "Mere Old Man" has gone wrong.

I am thinking that the trigger for this condition could be a dead battery, but can you give me your opinion as Nordic?

Please let me know about the matter you advised.
How can I verify the flash address I am trying to use in this debugging environment?
Also, if I were to dump the flash into a hex file, I am not sure how I would be able to reproduce the behavior in question.

Best Regards,
Wataru

RE: Mesh assert on flash_area_build()

Edvin — Wed, 14 Jun 2023 11:41:52 GMT

And if possible, please also try to dump the flash in the chip to a hex file using:

nrfjprog --readcode flash_dump.hex

This way, it would be easier to reproduce the issue if we have all the flash in a .hex file.

Best regards,

Edvin

RE: Mesh assert on flash_area_build()

Edvin — Wed, 14 Jun 2023 11:40:48 GMT

Hello Wataru,

Sorry for the late reply. I was unexpecedly out of office after being assigned this ticket.

Does this means that the flash page that it is trying to use is not empty (not all 0xFFs), but it contains some garbage data? If you can still reproduce the issue, can you please check:

1. the flash address it is trying to use, and:

2. read out the entire flash? You can do so using the command:

nrfjprog --readcode 0x00000000 --n 0x00100000 >> flash_dump.txt

Best regards,

Edvin