Mesh assert on mesh stack init.

RE: Mesh assert on mesh stack init.

bart — Tue, 25 Feb 2020 12:59:34 GMT

So by the time you asked for this full dump, I already decided disable persistent storage, and thus couldn't provide you with a full flash dump. However, we started using it again, and again ran into mesh asserts after some time.

I suspect it started on brownout (already have a couple cases that are crashing since we enabled mesh persistent storage), which happens when the device is unplugged.

This time it crashes at some other place in the code, but again during loading from flash at init.

GDB trace:

0x0003915a in app_error_fault_handler (id=id@entry=16386, pc=302160, info=info@entry=536935760) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/third/nrf/app_error_weak.c:101
101	    NRF_BREAKPOINT_COND;
(gdb) bt
#0  0x0003915a in app_error_fault_handler (id=id@entry=16386, pc=302160, info=info@entry=536935760) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/third/nrf/app_error_weak.c:101
#1  0x0004fd34 in mesh_assertion_handler (pc=<optimized out>) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/examples/common/src/assertion_handler_weak.c:54
#2  0x00049c52 in restore_addresses_for_model (p_model=0x200057e0 <m_model_pool+112>) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/mesh/access/src/access.c:816
#3  restore_models () at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/mesh/access/src/access.c:853
#4  access_flash_config_load () at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/mesh/access/src/access.c:989
#5  0x0004d4e2 in mesh_stack_init (p_init_params=p_init_params@entry=0x2000fdc4, p_device_provisioned=p_device_provisioned@entry=0x200040e4 <Mesh::getInstance()::instance+4>) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/mesh/stack/src/mesh_stack.c:114
#6  0x0003b698 in Mesh::init (this=0x200040e0 <Mesh::getInstance()::instance>, board=...) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/mesh/cs_Mesh.cpp:273
#7  0x0003de6c in Crownstone::startOperationMode (mode=@0x2000ffec: OperationMode::OPERATION_MODE_NORMAL, this=0x2000ff14) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:623
#8  Crownstone::startOperationMode (this=0x2000ff14, mode=@0x2000ffec: OperationMode::OPERATION_MODE_NORMAL) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:612
#9  0x0003dc30 in Crownstone::switchMode (this=this@entry=0x2000ff14, newMode=@0x2000ffec: OperationMode::OPERATION_MODE_NORMAL) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:528
#10 0x0003d6a0 in Crownstone::init1 (this=this@entry=0x2000ff14) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:248
#11 0x0003e2fc in Crownstone::init (step=1, this=0x2000ff14) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:806
#12 Crownstone::handleEvent (this=0x2000ff14, event=...) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:806
#13 0x0002f884 in EventDispatcher::dispatch (this=0x20002618 <EventDispatcher::getInstance()::instance>, event=...) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/events/cs_EventDispatcher.cpp:33
#14 0x0002f2e4 in Storage::handleFileStorageEvent (this=0x20002818 <Storage::getInstance()::instance>, p_fds_evt=0x20002a28 <Timer::init()::APP_SCHED_BUF+264>) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/drivers/cs_Storage.cpp:906
#15 0x00040162 in app_sched_execute () at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/nrf5_sdk/components/libraries/scheduler/app_scheduler.c:280
#16 0x0003e2c8 in Crownstone::run (this=this@entry=0x2000ff14) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:784
#17 0x0003e44c in main () at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/source/src/cs_Crownstone.cpp:960
(gdb) f 2
#2  0x00049c52 in restore_addresses_for_model (p_model=0x200057e0 <m_model_pool+112>) at /home/vliedel/dev/bluenet-workspace-cmake/bluenet/tools/mesh_sdk/mesh/access/src/access.c:816
816	        NRF_MESH_ERROR_CHECK(dsm_address_publish_add_handle(p_model->model_info.publish_address_handle));

Config:

ACCESS_FLASH_PAGE_COUNT = 1
DSM_FLASH_PAGE_COUNT = 1
FLASH_MANAGER_RECOVERY_PAGE_OFFSET_PAGES = 6

Flash dump: www.dropbox.com/.../flash.log

RE: Mesh assert on mesh stack init.

Mttrinh — Thu, 19 Sep 2019 09:20:44 GMT

Hi,

We haven't been able to reproduce this assert even after rebooting the device several thousand times. Looks like the cause of this assertion is possibly corrupted length field of the written entry. Such corruption can occur if the flash power supply is unstable, and if flash data gets corrupted such assertion can trigger.

Seems like the problem occurred for the page starting at 0x6D000 (since p_end is 0x6E000 as per the debug logs) and a flash dump for that page haven't been provided. Could you provide a full flash dump when this issue happens and also ensure that the chip power supply is stable?

RE: Mesh assert on mesh stack init.

Mttrinh — Fri, 30 Aug 2019 13:18:38 GMT

You are correct. I have forwarded this to the team. I will update you when they figured out what is causing this assert.

RE: Mesh assert on mesh stack init.

bart — Fri, 23 Aug 2019 12:31:09 GMT

How is that different than using the back trace I pasted? You can see exactly where in the mesh sdk code it fails.

RE: Mesh assert on mesh stack init.

Mttrinh — Fri, 23 Aug 2019 10:22:12 GMT

If you put a breakpoint at app_error_fault_handler you can read out the line number to see which function are triggering this error. You should also get an error code. The error seems to be somewhere in your application.

RE: Mesh assert on mesh stack init.

bart — Tue, 20 Aug 2019 08:21:57 GMT

What more debug would you need? The backtrace i pasted shows which assert fails, but i'm not that much in the mesh code to understand what it means. I was hoping you could tell me this.

I rebooted it about 1000 times, but I didn't try to reproduce this state yet, as i have the option to simply disable mesh persistent storage for now.

Bart

RE: Mesh assert on mesh stack init.

Mttrinh — Fri, 16 Aug 2019 10:53:58 GMT

Hi,

I'm not sure what is causing this issue but you should debug your application and see what error you get when this happens. Put a breakpoint at app_error_fault_handler and check the error code.

If you re-flash your application on the device, does this still occurr? How many reboots until this happens? Is it the same amount of reboots each time?