SOFTDEVICE: ASSERTION FAILED while running mesh.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Tue, 10 Sep 2019 08:27:26 GMT

Very sorry for the delayed response. Received an update: "Using the FDS shouldn't normally cause the mesh to kill the Softdevice, neither should the SD advertiser. Since increasing the safety margin helps, it sounds like there's either some IRQ locking going on in the application or some excessive clock drift that is causing this problem. My bet is on the former.

To rule out any clock drift issues, they can try running the application on a DK if they're on a custom HW platform. They can also try running the normal examples on the same HW with proxy enabled. If this is a xtal HW problem, any softdevice activity on an unedited example application should invoke the same crash.

Next, I'd profile any and all IRQ locking (including SDK modules). A full IRQ lock preferably shouldn't last for more than 20us, and definitely not for more than 50us. Both the softdevice API for locking (sd_nvic_critical_region_enter) and the hardware API (_DISABLE_IRQS(), __disable_irq()) could be causing issues here. Also make sure that no application interrupts are executing in any of the Softdevice IRQ levels.

If none of this explains it, I think we'd need more details to determine the cause. What happens right before the crash? Is it possible to get a logic analyzer trace of the crash with mesh debug pins enabled and initialized for timeslot? What's the XTAL accuracy passed to mesh_stack_init? Does it fit with the XTAL HW?"

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Tue, 27 Aug 2019 15:17:16 GMT

Ah, I read this as FDS will get in some trouble (write timeouts), not the mesh getting into trouble, and definitely not getting SD asserts. Also, above your quote, the documentation mentions:

To allow safe coexistence of the flash storage module Flash manager in the mesh stack and the flash storage module fstorage in the Nordic nRF5 SDK, add the following code block to nrf_mesh_config_app.h

So that makes one think it's fine to use FDS next to flash manager. Can you maybe explain why FDS would lead to SD asserts?

Regarding the advertising, doesn't the mesh use the timeslot API for that? If so, why would it matter that I also use SD advertiser? Seeing how mesh messages are also just advertisements, you'd expect it doesn't matter for the mesh whether it's advertising mesh messages or iBeacon messages. How would SD advertising result in SD asserts?

Also, how come setting TIMESLOT_END_SAFETY_MARGIN_US to 1000 makes these asserts vanish?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Tue, 27 Aug 2019 13:09:37 GMT

Another update: "As I see, they are using FDS. If you see the mesh SDK documentation:
https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.meshsdk.v3.2.0/md_doc_getting_started_how_to_nordicSDK.html?cp=5_2_1_7

We mention that:

nRF5 SDK NVM storage modules

Using nRF5 SDK modules such as fstorage, pstorage, or ble_flash for writing to flash may be problematic due to long timeslot events occupied by the mesh stack. Use the Flash manager module provided by the mesh stack instead.

He can either use flash manager or better, mesh_config, as used in various other internal modules.

Also, for beacon advertising, if possible, use the Mesh advertiser instead of SD advertiser."

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Tue, 27 Aug 2019 08:19:21 GMT

So sorry for the delayed reponse again! I have asked internally & the quick answer is do not use FDS. FDS with Bluetooth Mesh will lead to timeslot issues & SD asserts. Much easier to use the provided flash manager in the mesh sdk. A lot easier to use too from what I have heard.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Fri, 23 Aug 2019 08:04:52 GMT

Hey Bjorn,

I'm aware of the mesh using the timeslot API. And I looked at the ble_app_proximity_coexist example, is the uart one more relevant?

Like I said before, even without the iBeacon advertising, I still got the crashes, so I fail to see how that can be the main cause. Did you maybe mean the advertising i do directly via the SD (which are not iBeacon advertisements)? The ibeacon advertising is done via the mesh api, not directly via the SD.

Also like I said before, ever since I set TIMESLOT_END_SAFETY_MARGIN_US to 1000, I don't get the crash anymore:

I tried setting TIMESLOT_END_SAFETY_MARGIN_US to 1000. In that case, I still quite often get the FDS timeouts on boot, but it doesn't result in the event NRF_EVT_RADIO_SESSION_IDLE, and a softdevice assert anymore.

Does that mean the 100 is too tight? How do you determine the correct value?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Thu, 22 Aug 2019 16:03:26 GMT

Very sorry for the delayed response. As you can see from this blog post, you use the Timeslot API directly to communicate between the Mesh stack & the softdevice. The softdevice is only used for the iBeacon advertising. From your code, it seems there is definitely something wrong with the timeslot of the iBeacon advertisement.

Have you taken a look at one of the coexist examples between Mesh & BLE UART for example? I will take a closer look tomorrow morning.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Thu, 15 Aug 2019 14:45:01 GMT

I tried, but only get ??:0 Don't you need the softdevice elf for that address?

Here you can see the code, but as i said, i had the crashes even without the mesh doing the advertising. The softdevice is also advertising a different payload.

So far, i didn't get a crash yet with the margin set to 1000, while both mesh and softdevice are advertising.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Thu, 15 Aug 2019 13:43:13 GMT

Have you tried using the addr2line tool? How are you doing the iBeacon advertising? Could you upload some code so we can double check how you are doing this? Are you using the softdevice to do the iBeacon advertising?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Wed, 14 Aug 2019 09:38:38 GMT

Does that mean the 100 is too tight? How do you determine the correct value?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Tue, 13 Aug 2019 15:57:41 GMT

Also tried with PERSISTENT_STORAGE 0, but still getting the same crash.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Mon, 12 Aug 2019 07:05:15 GMT

All mesh functions are called from thread, from the main loop (I'm using the app scheduler).

Edit: while checking your link, I do notice that I hadn't set NRF_SDH_DISPATCH_MODEL to 1 yet. I will try and see if that helps. Until now, I manually put events that I actually use on the app scheduler. Does this setting matter for the mesh code?

Edit2: With NRF_SDH_DISPATCH_MODEL 1, I'm still getting the crash.

The one that's easiest to reproduce is the one on boot, where i get timeouts from FDS, and then this event 7 (NRF_EVT_RADIO_SESSION_IDLE).

The back trace I have in that case:

#0 0x0003b012 in app_error_fault_handler (id=1, pc=86552, info=0) at (...)/SDK-15-3/components/libraries/util/app_error_weak.c:100
#1 0x00023de8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) info registers
r0             0x0   0
r1             0x1   1
r2             0xe000e100   3758153984
r3             0x80000000   2147483648
r4             0x10001   65537
r5             0x0   0
r6             0x1   1
r7             0x15218   86552
r8             0x0   0
r9             0x20005eac   536895148
r10            0x20000000   536870912
r11            0x0   0
r12            0x0   0
sp             0x2000fd98   0x2000fd98
lr             0x3c0b9   245945
pc             0x3b012   0x3b012 <app_error_fault_handler+110>
xpsr           0x810f0003   2165243907
msp            0x2000fd98   536935832
psp            0x0   0
primask        0x1   1
basepri        0x0   0
faultmask      0x0   0
control        0x0   0
fpscr          0x80000010   2147483664

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Mon, 12 Aug 2019 07:01:30 GMT

Which priorities are you running in your example (see link)?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Fri, 09 Aug 2019 15:33:51 GMT

Tried completely without mesh advertising, but that didn't help.

Also tried with mesh 3.2.0, that didn't help either.

Every crash I've seen last days, I always got this event 7 (NRF_EVT_RADIO_SESSION_IDLE).

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Thu, 08 Aug 2019 13:00:06 GMT

Another thing I just noticed: while I don't have DEBUG defined, the device does not reboot.

Edit: never mind, this is just the NRF_BREAKPOINT_COND.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Thu, 08 Aug 2019 12:06:25 GMT

Maybe some additional info that helps:

Sometimes at boot, the SD seems to hang for seconds: In this time, I don't get any advertisements, and FDS gives me write timeouts. It also sometimes ends in the event 7 (NRF_EVT_RADIO_SESSION_IDLE).

I don't get that if I don't start the mesh.

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

bart — Mon, 05 Aug 2019 10:19:41 GMT

Same sort of thing:

app_error_fault_handler (id=1, pc=86552, info=0)

bt: #1 0x00023de8 in ?? ()

My soc handler printed event 7 (NRF_EVT_RADIO_SESSION_IDLE).

The mesh debug:

<t:    1353824>, net_beacon.c, 265, BEACON TX F8:D2:7A:2E
<t:    1772985>, net_packet.c, 230, Unencrypted data: : C51E60D00D80AD09ADA0E236A8337182EA6A
<t:    1772992>, transport.c, 934, Message decrypted
<t:    1772995>, access.c, 246, RX: [aop: 0x00C0]
<t:    1773660>, net_packet.c, 230, Unencrypted data: : C51E6016BCC5AAA3AE307F3A8CDE8874138C
<t:    1773667>, transport.c, 934, Message decrypted
<t:    1773671>, access.c, 246, RX: [aop: 0x00C0]
<t:    1774446>, net_packet.c, 230, Unencrypted data: : C51E60F7DC92944C0A997FC9EF9AFF507FAE
<t:    1774454>, transport.c, 934, Message decrypted
<t:    1774457>, access.c, 246, RX: [aop: 0x00C0]
<t:    1776676>, net_packet.c, 230, Unencrypted data: : C51E60B12F7637719B5817BF995101639455
<t:    1776684>, transport.c, 934, Message decrypted
<t:    1776687>, access.c, 246, RX: [aop: 0x00C0]
<t:    1777588>, net_packet.c, 230, Unencrypted data: : C51E601F09C849EDAE1495D76BB7AA0936DC
<t:    1777596>, transport.c, 934, Message decrypted
<t:    1777599>, access.c, 246, RX: [aop: 0x00C0]
<t:    1778394>, net_packet.c, 230, Unencrypted data: : C51E60B374117D1E4D154091045930567D09
<t:    1778402>, transport.c, 934, Message decrypted
<t:    1778405>, access.c, 246, RX: [aop: 0x00C0]
<t:    1852661>, net_packet.c, 230, Unencrypted data: : C51E60A68964C124FDB04E7C1D7C73
<t:    1852668>, transport.c, 934, Message decrypted
<t:    1852671>, access.c, 246, RX: [aop: 0x00C0]
<t:    2009184>, net_beacon.c, 265, BEACON TX F8:D2:7A:2E
<error> app: SOFTDEVICE: ASSERTION FAILED

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Fri, 02 Aug 2019 13:00:03 GMT

What happens if you decrease the iBeacon advertising interval to say every 2 seconds? Do you get the assert then?

RE: SOFTDEVICE: ASSERTION FAILED while running mesh.

Bjørn Kvaale — Thu, 01 Aug 2019 12:04:20 GMT

Let me see if I can get a hold of a softdevice elf file. Then we could use the addr2line tool to see where the softdevice assert is occuring. I will get back to you soon.