Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Fri, 13 Mar 2020 08:02:12 GMT

It's not possible to read the WD counter, unfortunately. It starts over again when you reload it.

[quote user="cdunster"]This makes it seem like either the watchdog is not being kicked properly or as you previously said it is incrementing in the background building up overtime getting close to the overflow value, then when the system wakes and performs it's actions this CPU usage is long enough that it pushes the watchdog to overflow before it can idle and get kicked.[/quote]

What's the WDT_CONFIG_BEHAVIOUR configured to in your sdk_config.h file? " Run in SLEEP" would make it count continuously anyway.

[quote user="cdunster"]By reducing the watchdog time to about 10ms instead of 3 seconds then we can see the same problem but much quicker. With this quicker watchdog we see that the system can work fine indefinitely, connecting and disconnecting phones; sending commands; etc., but then if we disconnect the phones, leaving the system to idle for about a minute, and afterwards we connect a single phone then the watchdog kicks in and reboots the system.[/quote]

Assume you had to increase the frequency of your reloading task to make 10 ms work?

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Thu, 12 Mar 2020 11:46:50 GMT

We have done power profiling and it seems that the CPU usage does not change at the time we see the issue. If it was CPU usage that it would likely happen every time and not just after a prolonged idle period.

By reducing the watchdog time to about 10ms instead of 3 seconds then we can see the same problem but much quicker. With this quicker watchdog we see that the system can work fine indefinitely, connecting and disconnecting phones; sending commands; etc., but then if we disconnect the phones, leaving the system to idle for about a minute, and afterwards we connect a single phone then the watchdog kicks in and reboots the system.

This makes it seem like either the watchdog is not being kicked properly or as you previously said it is incrementing in the background building up overtime getting close to the overflow value, then when the system wakes and performs it's actions this CPU usage is long enough that it pushes the watchdog to overflow before it can idle and get kicked.

We use the nrfx_wdt_channel_feed(m_channel_id); function to kick the watchdog, is this correct?

Is it possible to print the current watchdog timer value? So that we can see if it does actually increase over time and that it is reloaded correctly.

Thanks.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Tue, 10 Mar 2020 14:42:11 GMT

So I've discussed this with a colleague who is experienced with Freertos and he thought the problem could be that the BLE task is not yielding to other thread responsible for feeding the WD. Have you tried to do any profiling to see if there's heavy CPU usage around the time a connection is established?

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Tue, 10 Mar 2020 10:05:27 GMT

This was our worry too but, strangely, we can reduce the failure rate but we are struggling to increase the failure rate. We have tested it a few times with the watchdog timeout increased to 10 seconds (from 3 seconds) and the system seems to work without crashing and also doesn't hang anywhere. We have also tried to decrease the watchdog to 1 second but the failure stays the same.

Is it possible to print the current value of the watchdog timer so that we can see if it is being fed and if it is incrementing in the background as you said it might be.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Mon, 09 Mar 2020 14:01:45 GMT

In the failing cases, maybe the WD is already close to expiring when you get the connection event, and the code responsible for reloading is being delayed enough by that to cause a reset. Have you noticed any impact on failure rate if you increase the timeout and or frequency of the periodic tasks that reload the WD?

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Thu, 05 Mar 2020 11:56:21 GMT

It does seem that it is a genuine watchdog reset but without the watchdog the system runs fine so it is not stuck but instead it seems that a task is just taking too long meaning the watchdog is not kicked.

The weird thing is still the fact that it only happens after a phone connects via BLE and the system has been on for a few hours and not every time a phone connects, even though the same tasks run.

I was wondering if it is the fact you mentioned about the watchdog incrementing in the background so I reduced the watchdog time and let it run thinking it would keep incrementing until it overflowed and caused a reset, but it didn't.

So, it does seem like it is a problem in our code where something takes too long which we can fix. However, I still don't understand why it only happens after some time, any ideas?

Thank you!

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Tue, 03 Mar 2020 14:26:42 GMT

Ok, good to know that it is being reloaded. I understand it's time-consuming to run this test, but have you tried to extend the WD timeout as well? As for the CPU halting at 0xA80, I recently found out that the debugger will halt the CPU on reset if the "reset" breakpoint is enabled (you can see this breakpoint is disabled in the first screenshot I posted here). The problem is then to find out what caused it to reset in the first place. Maybe it's always by the WD timing out.

Has this never happened with the WDT disabled?

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Mon, 02 Mar 2020 15:32:52 GMT

I believe the WDT is being fed as we are using FreeRTOS and we have some periodic tasks which run during this period. Currently, the watchdog itself is actually being fed by a periodic task but with it's priority set to the same level as idle.

That's good to know about it being incremented in the background though, thanks.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Mon, 02 Mar 2020 14:17:17 GMT

Is the WD being fed during this idle period? Note that Softdevice tasks such as advertisement events are run in the background without waking the application and will cause the WDT counter to be incremented even if you have enabled the pause on sleep option.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Mon, 02 Mar 2020 13:11:20 GMT

So, it seems like the watchdog is the cause of the crash. However, this crash only occurs if the system is left idle for hours and then we connect a phone via BLE. We have repeated the test but with the watchdog disabled and the system runs fine (we were hoping that it would get stuck in a loop and we could find it without the Watchdog but it just keeps running).

Are you aware of anything that might change after a long period of time? We can leave the system for days and it works fine but as soon as we connect a phone it reboots.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Wed, 26 Feb 2020 13:38:55 GMT

The MBR reset handler starts at 0xA80 so it looks like it just came out of a system reset. I suggest you check RESETREAS register to find out what the reset source was (RESETREAS)

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Wed, 26 Feb 2020 10:53:42 GMT

We are using SoftDevice 7.0.1

This is a screenshot of the problem. As you can see we are stopped at 0xA80 and there is noting on the Call Stack. The ISR register is 0.

Thanks for your help.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Fri, 21 Feb 2020 14:26:32 GMT

I would have to check that. What Softdevice version are you using?

Maybe you can check and see what the ISR number is when you hit the breakpoint at 0xA80. As you can see from the screenshot below, the ISR number is 3 (=hardfault) when the program halts at 0xa60.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

PRvdO — Fri, 21 Feb 2020 13:35:57 GMT

Thanks for the quick response.

Do you know what is at address 0xA80?

The 0xA60 reset only seems to happen after 3 hours so it will take a while to reproduce, in the mean time, the 0xA80 reset happens when multiple BLE devices are connected. Each connected device sends data over BLE which is forwarded via the UART so could it be to do with either of those?

Thanks again.

RE: Breakpoint and reset in SoftDevice addresses 0xA60 and 0xA80

Vidar Berg — Fri, 21 Feb 2020 12:32:18 GMT

Hi,

0xA60 is where the hardfault handler is located in the MBR. Please disable the breakpoint on hardfault as shown by the screenshot below (assuming you use SES) and check if the program enters the hardfault handler in your application or if it ends up in the app error handler.