strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Hieu — Wed, 03 Jul 2024 20:41:27 GMT

Hi Volker,

Sorry I wasn't clear. I meant to just take a look at the relevant source code so that you can put more details in when you report the issue to the Zephyr project.

I don't mean to recommend you to move to latest Zephyr version.

Regards,

Hieu

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Volker — Wed, 03 Jul 2024 12:32:57 GMT

Hi Hieu,

for our application the upgrade from zephyr 2.4 to zephyr 3.2 took about 3 month until everything was running again, especially with having the same low power consumption as before. So I am seeing almost no chance to check this on the latest zephyr.

Regards,

Volker

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Hieu — Wed, 03 Jul 2024 09:02:25 GMT

Hi Volker,

What you discovered about overlapping priorities check out and explain the situation we see. If you no longer see the same issue anymore, I believe you have got the correct fix.

Since you know the details, do you think it is a good idea to check if the problem is still in the latest version of Zephyr, and report it on Zephyr GitHub?

Regards,

Hieu

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Volker — Wed, 03 Jul 2024 08:15:40 GMT

Hi Hieu,

thank you for your detailed reply.

I found some youtube comparison of zephyr-vanialla and ncs. There was mentioned some difference in the licensing model. Is it right that using ncs is ok, as long as it runs on some nrf hardware (cpu)?

As soon as I realized, that CONFIG_PRINTK_SYNC increased the problem, I deactivated it immediately.

In the meantime I was able to make some progress ... and found out, that there was some overlap between application thread priorities and bluetooth related thread priorities. When giving all the application threads lower priorities than the bluetooth threads, the problems seems to disappear.

This was kind of surprising, as the same application was running on zephyr 2.4 without having these problems.

Regards,

Volker

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Hieu — Tue, 02 Jul 2024 14:27:58 GMT

Hi Volker,

[quote user="Volker"]Yes, we are using vanilla zephyr since several years. We are within the step to upgrade from zephyr 2.4 to zephyr 3.2 so we might have some minor chance to use this drive to eventually switch over to nRF Connect SDK. Are there some major benefits from doing so?[/quote]

The nRF Connect SDK (NCS) supports the SoftDevice Controller (SDC), which is Nordic's proprietary controller that was designed to work best with Nordic's products. It also includes the (also proprietary) Multiprotocol Service Layer, MPSL, which as its name says, enables multi-protocol support.

We also perform Bluetooth qualification on the Zephyr Host and SDC combination, so as long as there is no modification to the Bluetooth LE stack, you can refer to our QDID during your product's qualification process. The Zephyr Controller doesn't have this.

Last but not least, as Nordic own the SDC completely, we can provide you better technical support for issues regarding it.

[quote user="Volker"]We had several stack overflows during the migration process to zephyr 3.2 and now I would assume that we have increased the stack size of all relevant threads. We used CONFIG_THREAD_ANALYZER to get regular output about the actual used stack size, but I am not quite sure if this covers all dynamic of stack usage properly. Do you have some more hints of how to make sure that all stacks are big enough?[/quote]

I had a similar issue back in NCS 2.3.0, which coincidentally (?) is also based on Zephyr 3.2. What I observed is that even with thread analyzer showing that the stack uses are almost always fine, the system still crashes, and a further increase in stack size did help.

I guess that a fairly large spike in stack use caused the overflow, and thus the thread analyzer didn't have a chance to show that the impending issue.

[quote user="Volker"]Reducing the amount of logging output is not really an option, as we are just using our application specific logging, which is almost essential. Other built in component logging has been disabled already.[/quote]

I see. But then you should at least disable CONFIG_PRINTK_SYNC. If you are worried about losing logs, it's better to increase the buffer size instead.

If you are using printk() for logging, the config is CONFIG_PRINTK_BUFFER_SIZE.

If you are using the logging module, the config is CONFIG_LOG_BUFFER_SIZE.

Is this possible? If so, please try it and see if the situation improve.

Regards,

Hieu

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Volker — Mon, 01 Jul 2024 14:33:13 GMT

I found the CONFIG_BT_CTLR_PROFILE_ISR config option, but I am not quite sure about how to interpret the output. Can you give me some advise?

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Volker — Mon, 01 Jul 2024 13:20:47 GMT

Hi Hieu,

Thank you for your reply.

Yes, we are using vanilla zephyr since several years. We are within the step to upgrade from zephyr 2.4 to zephyr 3.2 so we might have some minor chance to use this drive to eventually switch over to nRF Connect SDK. Are there some major benefits from doing so?

We had several stack overflows during the migration process to zephyr 3.2 and now I would assume that we have increased the stack size of all relevant threads. We used CONFIG_THREAD_ANALYZER to get regular output about the actual used stack size, but I am not quite sure if this covers all dynamic of stack usage properly. Do you have some more hints of how to make sure that all stacks are big enough?

Reducing the amount of logging output is not really an option, as we are just using our application specific logging, which is almost essential. Other built in component logging has been disabled already.

Do you have any other suggestions of how to track down and potentially fix the root cause for the bluetooth processing delays? Or how to optimize this?

Regards,

Volker

RE: strange bluetooth low energy behavior and assertions in zephyr 3.2 when using debug mode for compilation

Hieu — Mon, 01 Jul 2024 12:11:27 GMT

Hi Volker,

You mentioned Zephyr 3.2, and the log shows use of the Zephyr Controller. Are you using vanilla Zephyr RTOS? We recommend using the nRF Connect SDK for development.

For your issue, debug mode builds often cause less efficient code, so the application requires more memory and timing margins.

One common issue is threads running out of stack, so please try increasing your threads' stack size first.

If the problem persists, please try to turn off logging or reducing the log level of the modules that you are not debugging.

Please let me know if any of those helps.

Regards,

Hieu