Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Tue, 06 Aug 2024 08:29:26 GMT

@taspon98

Hi, we don't use CONFIG_BT_LL_SW_SPLIT anymore. This is also because in a later certification only QDIDs will be available for the Nordic stack, but not for the Zephyr SW stack (CONFIG_BT_LL_SW_SPLIT). That's why I didn't pursue the IRQ 24 issue any further.

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

PuffinEgg — Sat, 20 Jul 2024 09:26:45 GMT

Hi, do you happen to know the other config that might be conflicting with CONFIG_BT_LL_SW_SPLIT? I am also running into the same issue with multi handler registration for irq 24 when migrating to 2.7.0

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Thu, 11 Jul 2024 14:00:43 GMT

I have now found a useful guide for monitor mode.
It allows me to debug my BLE project with SDK 2.6.1.

Here I found a guide that worked for me:

Section "7.1 Add the following Kconfig flags to prj.conf"

Add:
CONFIG_CORTEX_M_DEBUG_MONITOR_HOOK=yCONFIG_SEGGER_DEBUGMON=y

and section "7.4 Run the debug action"

Command in the debug console of nRF Connect for VS Code:

-exec monitor exec SetMonModeDebug=1

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Thu, 11 Jul 2024 11:55:38 GMT

Note on SDK 2.6.1:

No patch works here anymore!
I get the following error when compiling the project:

gen_isr_tables.py: error: multiple registrations at table_index 24 for irq 24 (0x18)
Existing handler 0x1a001, new handler 0x28e2f
Has IRQ_CONNECT or IRQ_DIRECT_CONNECT accidentally been invoked on the same irq multiple times?

The reason is the use of CONFIG_BT_LL_SW_SPLIT!

As soon as I remove CONFIG_BT_LL_SW_SPLIT and use the Nordic stack, I can compile successfully again.

Unfortunately, debugging then no longer works!

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 22 Mar 2024 08:48:50 GMT

A note about SDK 2.5.2:
The patch I mentioned no longer works here!
The call to LL_ASSERT_MSG was removed in GIT on 2023-02-13 and is now not included in SDK 2.5.2 (possibly also in earlier SDK versions).

Interestingly, on 2023-06-28 the define LL_ASSERT_OVERHEAD was added in zephyr\subsys\bluetooth\controller\hal\debug.h. Here you can prevent LL_ASSERT_MSG from being triggered via "!CONFIG_BT_CTLR_ASSERT_OVERHEAD_START".

See also https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/kconfig/index.html#CONFIG_BT_CTLR_ASSERT_OVERHEAD_START

The background seems a bit different, but it could also be a solution to my debug problem.
I still have to check whether this is the "built-in" solution for my problem.

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Wed, 13 Sep 2023 23:03:08 GMT

Hi Marko,

I'm glad that you're able to debug your BLE application sufficiently with this "one extra step" into the connection!

If you want to get an option included for this, I think the best way to proceed is to raise an issue or make a pull request on GitHub, and explain your reasoning. However, I don't think the option you suggest is likely to be added.

I think your case is a rare one, and that most people will be expecting to debug further than this. But the assert was added for a reason - halting a BLE enabled application won't work without the connection breaking down.

Besides this, we primarily develop with the SoftDevice Controller in mind, and the assert is firmly in place there.

I haven't tried this out myself yet, but please check out this documentation on the "monitor mode" debugging that I mentioned earlier:

https://docs.zephyrproject.org/latest/services/debugging/debugmon.html#cortex-m-debug-monitor

If you need to debug further than you're currently doing, look into this. And if it ends up working for you, I recommend you to move over to the SoftDevice Controller.

In the future I hope we can offer monitor mode debugging in some convenient way.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 08 Sep 2023 06:36:33 GMT

Hello Raoul,
I'll get back on this topic.

In the last NCS versions 2.4.0 to 2.4.2 I change the following code in the lll.c file:

#if !IS_ENABLED(CONFIG_DEBUG)
LL_ASSERT_MSG(false, "%s: Actual EVENT_OVERHEAD_START_US = %u", __func__,
HAL_TICKER_TICKS_TO_US(diff));
#endif // CONFIG_DEBUG

This has helped me well so far, but it is consuming to do this with every new NCS version. Especially not just on my computer but also with work colleagues.

Maybe could create an own config define to disable the strict timing requirements during development?

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Wed, 29 Mar 2023 14:44:37 GMT

Hi Marko,

I just wanted to say that I've shared your feedback with our NCS team. I know that BLE can be tightly integrated with business logic and so being able to set breakpoints is a huge benefit.

The strict timing requirements for BLE operation means that ordinary halting can never be expected to work. But I assume you are asking for something like the Monitor Mode debugging I mentioned earlier. I've asked the team about support for this, and if I hear anything useful I'll share it with you.

Meanwhile, for your very specific use case (continuing to do what you were able to do in v2.1.3 and v1.9.1), I want to reiterate that the only way this is possible at the moment is to use the Zephyr BLE controller and then commenting out the newly added asserts that I linked to in a previous reply.

However, please note that the Nordic SoftDevice controller is the only BLE stack that we officially support, and the only stack that gets QDID's which helps with final certification. So it might be better to find a way around the issue for the moment, until hopefully Monitor Mode debugging support arrives.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 17 Mar 2023 07:03:13 GMT

Hi Raoul,
does this mean that with newer SDK's it will never be possible to debug an application using BLE?

This would mean the end of our projects related to Zephyr and Nordic MCU's!
It is impossible to develop complex BLE projects without a debugger or to localize reported errors.

You can't always remove BLE from the projects to be able to debug. This would be a high risk when creating a release candidate, as you might not re-enable what is commented out correctly.
And I see another problem when you need the BLE communication to start your own processes that you want to debug.
This is exactly what I need right now and it works with NCS 2.1.3 as long as I don't set breakpoints in the BLE communication but in my own threads.
With NCS 2.1.3, even breakpoints when receiving BLE packets work, but only once, since the connection then breaks down (this is acceptable). But that's enough to be able to debug your own processing at least once.

It is also interesting that the breakpoint in the sample project samples/bluetooth/peripheral already has problems, only when advertising is already activated (without a connection).

Here the developers should create a way to also be able to debug with BLE. Either via CONFIG_DEBUG=y or a new explicit configuration parameter.

It would be nice if the developers could find a way.
Thanks :-)

Best regards,
Marko

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Wed, 15 Mar 2023 18:15:30 GMT

Hi again,

I realised that the MPSL assert was only related to the case where you enabled the Nordic SoftDevice controller, but that wasn't your original issue. Sorry for the confusion.

Your original issue was related to the Zephyr BLE controller - you were able to set a breakpoint at the start of your BLE enabled app before, but now no longer. Regarding that, I've heard back from a developer:

Zephyr Controller has recently added strict assertion on delayed events. If he strictly for debugging, he can comment out the said assertion:

https://github.com/zephyrproject-rtos/zephyr/commit/ebf723626704aaffa95a2a22fb4415a4ae59bf00

The Nordic SD Controller has never allowed breakpoints while it is enabled. Now the Zephyr controller doesn't either. So please note that this newly added assert should be seen more as a bugfix than anything else; timing is critical in BLE.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Sun, 05 Mar 2023 23:35:12 GMT

Hi Mark, thanks a lot for sharing these details.

I now realise that we have received multiple cases recently related to MPSL ASSERT: 112, 2195 being triggered by a breakpoint. I've informed the developers and shared some of your findings.

I'll let you know when they find out what to do about it.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 03 Mar 2023 14:24:17 GMT

Hi Raoul,
We are already using CONFIG_BT_LL_SW_SPLIT in the current projects because we had problems with BLE + UART that could be solved with it.

For this troubleshooting I would only refer to the example project samples/bluetooth/peripheral (with nRF52 DK NRF52832).

I added a breakpoint to the line

error = bt_enable(NULL);

and

base_notify();

I only use the first breakpoint (bt_enable) to determine a restart (only stops at main() the first time)
I want to debug the second breakpoint (bas_notify).

Result of the different (Pristine) builds:
NCS 2.1.3 without CONFIG_BT_LL_SW_SPLIT: crash while debugging
NCS 2.1.3 with CONFIG_BT_LL_SW_SPLIT=y: Debugging works fine

NCS 2.2.0 without CONFIG_BT_LL_SW_SPLIT: crash while debugging
NCS 2.2.0 with CONFIG_BT_LL_SW_SPLIT: crash while debugging

I hope I can help with this information to fix the error.

Best regards
Mark

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Sun, 26 Feb 2023 22:24:16 GMT

Hi Marko,

Could I ask you to try switching to the Zephyr BLE controller? This can be enabled through CONFIG_BT_LL_SW_SPLIT.

I'm curious if it works. Meanwhile, the original issue might be the result of a bug related to stack space, I'm not certain yet.

If you are able to share your project with us, that would be helpful.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Wed, 15 Feb 2023 09:28:42 GMT

Hi Marko, thanks for sharing details on this! So this sounds like there is a bug on our side. I'll share your findings internally and get back to you when I know more.

Best regards,

Raoul

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Mon, 13 Feb 2023 12:04:37 GMT

I tested my application code again with different SDK versions:

2.0.2 Debug OK
2.1.0 Debug OK
2.1.2 Debug OK
2.1.3 Debug OK
2.2.0 Faulting instruction address (r15/pc): 0x00000fb6

According to nRF Debug Memory Viewer, the symbol name of address 0x00000fb6 is "lll_preempt_calc".

In a test with the Bluetooth Peripheral example, however, it still crashes with 2.1.3:

Faulting instruction address (r15/pc): 0x0001fb78

According to nRF Debug Memory Viewer, the symbol name of address 0x0001fb78 is "m_assert_handler".

As another test, I have added the CONFIG_BT_LL_SW_SPLIT option, which I had to use in my project, to the example. The example can now also be debugged (however, only up to SDK 2.1.3).

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 10 Feb 2023 13:11:49 GMT

I ported one of my own projects back from NCS v2.2.0 to v.1.9.1 today.
Now I can easily debug after bt_le_adv_start!!!
Pause and resume, display variables... no problem.

The project has the same settings, only <zephyr/..." and <zephyr/kernel.h> have been ported back to the old syntax and %s have been removed from log output.

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Marko W — Fri, 10 Feb 2023 07:51:41 GMT

Hi, I'm working with Sven and can add one more thing to the problem.

The crash occurs as soon as advertisements are launched when there is no connection at all.

I have earlier observed that a breakpoint in the processing of Bluetooth data (when connected) results in a disconnection at remote participant. However, I think that earlier (must have been NCS v1.9.x) I was able to debug at least one received packet to the end (without crashing).

Turning off Bluetooth certainly helps when debugging application logic that has nothing to do with Bluetooth. However, our current development is receiving and sending data via Bluetooth.

I've earlier had a crash (Instruction Address Error (r15/pc)) related to binding a UART after the advertising started. At that time I had found a hint on the web to use CONFIG_BT_LL_SW_SPLIT=y which fixed the crash (with "warning: Experimental symbol BT_LL_SW_SPLIT is enabled."). We have already tested this setting here, unfortunately without success.

Regarding monitor mode debugging, I haven't been able to find anything for NCS/Zephyr so far.
According to the Nordic recommendation: "SEGGER Embedded Studio Nordic Edition is no longer tested and recommended for new projects." (Release Notes 2.0.0) we have used Visual Studio Code for all new projects and those still under development.

The question now is whether that was a good decision and whether NCS is perhaps not yet suitable for productive use at all?

Thanks for the support!

Marko

RE: Debuggen of BLE Applications based on Connect SDK 2.2.0 not possible -> Faulting instruction address (r15/pc)

Raoul — Fri, 10 Feb 2023 06:29:53 GMT

Hi Sven,

Are you setting a breakpoint somewhere? Unfortunately it is simply not possible to do halting debugging of BLE applications. The Bluetooth controller has strict timing requirements and will hardfault when these can't be met. See this post for a more detailed explanation: RE: Debugging while BT is working

If you need to debug your application, it's best to disable BLE temporarily. If you really have to debug with BLE, there is another debugging mode called Monitor Mode Debugging: https://www.segger.com/products/debug-probes/j-link/technology/monitor-mode-debugging/

It kind of keeps the CPU running in a loop at the breakpoint, allowing other timing critical things to continue running in the background.

I'm not familiar with Monitor Mode Debugging, and from what I can see, support for monitor mode debugging is still limited in NCS: Problem using monitor mode debugging with the nrf connect sdk Debugging with Monitor Mode on NCS

So I hope that you are able to debug your application with BLE temporarily disabled.

Best regards,

Raoul