How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Anthony W — Tue, 03 May 2022 14:53:58 GMT

In fact, before removing the peripheral UART code, that "lfclk_spinwait" is exactly where I found it getting stuck, so something similar was going on I bet. Thanks for your help!

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Elfving — Fri, 29 Apr 2022 07:48:59 GMT

Great!

I do not have a good explanation for why the presence off the UART code would be a problem, but I have actually seen this being the case before. A quick search gave me for instance this case here, which looks very similar.

[quote user="Anthony W"] but I suspect it might be because the UART was not initialized (or even present on the custom board) and somehow this non-existent UART was otherwise interfering with BT operation. [/quote]

That might be. But in any case, I am glad you figured it out.

Regards,

Elfving

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Anthony W — Thu, 28 Apr 2022 17:03:23 GMT

Actually, I finally found the issue yesterday, so I can (somewhat) fill in here. You're correct, the BMD-350 module does not have an external LF clock source, but both the dev board and the custom board do, accurate to 20ppm (on mine, not sure about the dev board). But that wasn't the issue. I based my code off of the peripheral uart sample actually, but I wasn't using UART at all. I removed all of the UART-specific implementation code (basically, just initialized BT and NUS) and the problem went away. I didn't debug exactly why that code was causing a failure (since it was unused in my application anyway and I don't care to keep it around) but I suspect it might be because the UART was not initialized (or even present on the custom board) and somehow this non-existent UART was otherwise interfering with BT operation. It seems like a stretch, and none of the code I removed should have been reached in operation, but removing it fixed the issue. So I guess it will remain a partial mystery for now.

Thank you for your advice though, in particular pointing out the clock source. While it wasn't necessarily applicable to this issue I do have a greater appreciation for the importance of the clocks on BT operation.

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Elfving — Thu, 28 Apr 2022 16:15:37 GMT

Hi Anthony, sorry about the wait.

[quote user="Anthony W"]However, both the development kit and my custom board have crystal oscillators onboard. I did try other clock sources anyway, all yield the same behavior.[/quote]

Did you try both high and low frequency clocks? It seems to me that the BMD-350 doesn't have an external low-frequency one. If your project isn't configured to use the internal RC, it might be an issue. You can do this by adding CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y to your configuration file.

If that doesn't help, have you tried our other examples as well? Does it for instance stop you from testing the peripheral uart sample?

devzone.nordicsemi.com/.../zephyr.hex

If it occurs there as well, could you provide me a sniffer trace?

Regards,

Elfving

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Anthony W — Fri, 22 Apr 2022 15:19:12 GMT

I was really hoping this would work, easy fix. However, both the development kit and my custom board have crystal oscillators onboard. I did try other clock sources anyway, all yield the same behavior.

The UART has also not been helpful. It will also stop after this failure mode. But I can still connect and disconnect from Bluetooth, there's just no longer bluetooth activity. It almost seems like the device is being put to sleep or something, but attempting to connect via RTT doesn't correct this issue. And again, with the RTT or debugger (SWD) connected in the first place, this issue will not arise.

There doesn't seem to be a way (at least in VSCode or through west) to attach a debugger to chipset already running without restarting the MCU and losing that error state. Although if it is just putting itself to sleep, I might not see the error state anyway.

RE: How can one debug a Zephyr RTOS application which crashes only with the debugger detached?

Elfving — Fri, 22 Apr 2022 10:16:25 GMT

Hello Anthony,

[quote user=""]Any thoughts as to how one might go about debugging a bug like this?[/quote]

You could try outputting something on the UART in order to get an understanding of the problem.

But in this case I have a theory already: BMD-350 doesn't have a crystal on it. You need to configure the nRF52832 to use the internal RC.

Regards,

Elfving