Integrate NUS into BLE Audio code

RE: Integrate NUS into BLE Audio code

Brian Kim — Tue, 16 Apr 2024 07:55:12 GMT

Hello Dekel,

Sorry for this delay. I just saw this reply. Now that the new request is totally different one, we should discuss on another thread. I believe this is already requested with another case:

nRF5340 Audio - configure gateway unit also as ACL peripheral

If that is about it, we can close this case and keep talking in the the case. Feel free to comment.

//Brian

RE: Integrate NUS into BLE Audio code

Dekel — Mon, 08 Apr 2024 12:59:21 GMT

Hi Brian,

It's actually not completely fixed.

We've figured out that it will be easier if the broadcaster side, which is in peripheral role in terms of audio, will also be the peripheral in terms of ACL connection. Meaning, we'd like it to advertise both audio stream and ACL connection for commands.

I was able to take the code you've supplied and apply it on the headset side. Namely, I have now a headset which is also able to connect through ACL as central (in parallel to audio stream receival) and communicate through NUS.

I would appreciate your help with the broadcast side implementation.

I tried it myself and I'm on the brink of success, but calling bt_le_ext_adv_create() twice (once for audio and once for ACL) fails with EIO (I/O error). It's unclear to me why it happens.

[quote userid="132468" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/475788"]I came across the following repository through a DevZone post (I am unable to find the original post through a simple google search)[/quote]

Thank you for that! I took a look on it, but since it relates to CIS while my project is BIS, it's difficult for me to understand the necessary modifications.

RE: Integrate NUS into BLE Audio code

Brian Kim — Tue, 02 Apr 2024 07:15:03 GMT

Thanks for the good news, Dekel!

Now that it seems to be fixed, please close this case. You can create new case any time if you faced another issue. :)

RE: Integrate NUS into BLE Audio code

mk102 — Tue, 26 Mar 2024 00:39:30 GMT

Hi Dekel,

I came across the following repository through a DevZone post (I am unable to find the original post through a simple google search)

https://github.com/nrfconnect/sdk-nrf/compare/v2.3-branch...rick1082:sdk-nrf:audio_with_nus_example?diff=split&w=

This combines the audio example with NUS example with minimal changes. You will still have to play around with the settings like `CONFIG_BT_BUF_ACL_RX_SIZE`, `CONFIG_BT_BUF_ACL_TX_SIZE`, `CONFIG_BT_AUDIO_BITRATE_UNICAST_SINK` based on your requirements. You can refer to samples like ble-throughput to understand what each of these configs do.

This was written based on nrf SDK v2.3.0, and I was able to get it to work with the latest v2.6.0 SDK. I am able to communicate simultaneously transfer data over ACL while playing/receiving audio between two Audio DKs.

RE: Integrate NUS into BLE Audio code

Dekel — Sun, 24 Mar 2024 08:25:37 GMT

Hi Brian,

[quote userid="2124" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/475140"]If you want to send the other way, it's already implemented[/quote]

I can see that on my devices, similar to the screenshot you've shared.👍🏼

[quote userid="2124" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/475140"][P.S] you don't need to use the option,-ctlr="SDC" any more because it's default option since NCS 2.6.0.[/quote]

Good to know 🙂

[quote userid="2124" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/475140"]

3. In order to integrate the ACL into the BLE Audio system, it's necessary for the headset units to be able to serve as preipherals. Is this implementation easier now that we have your implementation on gateway side?

4. Following my previous question, I'll share that in order to achieve the goal of our project, a full solution of this would be to be able to configure the gateway unit both as central and peripheral in terms of ACL connection (for connection with a smartphone app). Is it achieveable? What will it require?

[/quote]

I would truely appreciate your help with these topics, they're crucial to our project's progress.

Thank you,

Dekel

RE: Integrate NUS into BLE Audio code

Brian Kim — Thu, 21 Mar 2024 13:43:32 GMT

Hello Dekel,

Thanks for the good news!

[quote user="Dekel"]1. Regarding the automatic ACL connection that the gateway establishes - is it name-based? does it mean that it will try to connect to all units with the name "Nordic_UART_Service"?[/quote]

Yes, it corresponds with REMOTE_DEVICE_NAME_PEER.

[quote user="Dekel"]

2. How can I test the other direction of messaging? Namely, sending data from the gateway to the peripheral over ACL?

[/quote]

The screen shot I shared below was about sending data from the gateway(ACL central) to the peripherals. If you want to send the other way, it's already implemented. Once data is transmitted from any peripheral, you will see it in ble_data_received.

I simply tried and confirmed it works as intended like this. :)

[P.S] you don't need to use the option,-ctlr="SDC" any more because it's default option since NCS 2.6.0.

//Brian

RE: Integrate NUS into BLE Audio code

Dekel — Wed, 20 Mar 2024 16:57:29 GMT

Hi Brian,

First of all, thank you so much for your efforts and for sharing all that information! this is very helpful to us.

Based on my tests, things look good! I'll describe my process:

I've installed NCS2.6.0 through the Toolchain Manager and replaced the 3 files you've attached in the zip file.

I've successfully built and flashed the program using this command (flashed gateway & headset units):

python buildprog.py -c both -b debug -d both -p --ctlr="SDC" --recover_on_fail

The gateway receives audio data through USB (I've removed CONFIG_AUDIO_SOURCE_I2S from proj.conf for that) and it's successfully heard on the headset side, to which I've connected a speaker.

I've also flashed a third unit, nRf52833DK, with the "peripheral UART" example code. On startup, while the broadcaster is already running and music is playing, an ACL connection was established automatically and I was able to write from the peripheral (nRf52833DK) to the central (gateway) using TeraTerm.

This is great! It may serve as infrastructure for sending sensor data from the headset units towards the gateway.

I'd like to ask a few questions please:

1. Regarding the automatic ACL connection that the gateway establishes - is it name-based? does it mean that it will try to connect to all units with the name "Nordic_UART_Service"?

2. How can I test the other direction of messaging? Namely, sending data from the gateway to the peripheral over ACL?

Thank you,

Dekel

RE: Integrate NUS into BLE Audio code

Brian Kim — Tue, 19 Mar 2024 15:03:00 GMT

Hello Dekel,

I am sharing very initial reference for your application working as a BIS broadcaster and NUS central simultaneously based on NCS1.6.0.

6-8 concurrent ACL link might be challenging, but you can tweak related parameters properly , like the connection interval(currently 20ms) or CONFIG_BT_MAX_CONN(currently 4).

Please note that this is just a rough reference not guarantee the stability, so you should verify and purify it.

The modified files :

nrf/applications/nrf5340_audio/broadcast_source/main.c
nrf/applications/nrf5340_audio/prj.conf
nrf/applications/nrf5340_audio/src/bluetooth/bt_management/bt_mgmt.c

And also please refer the screen shot of the operation. I tested with three devices in total, BIS broadcaster, BIS headset, nRF52DK(1st NUS server) and nRF52840 DK(2nd NUS server).

[updated 20.03] replaced with one zip file for easier access.( also changed one parameter(CONFIG_BT_AUDIO_RETRANSMITS=3 from 10)on prj.conf for faster ACL communication.

devzone.nordicsemi.com/.../nrf5340_5F00_audio_5F00_gw_5F00_central.zip

RE: Integrate NUS into BLE Audio code

Dekel — Mon, 04 Mar 2024 08:39:39 GMT

I'll share current code modifications with a hope for help:

devzone.nordicsemi.com/.../modifications.rar

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Wed, 28 Feb 2024 10:32:26 GMT

Hello,

[quote user="Dekel"]No, the NUS is not integrated yet. Only the ability to have a connectable advertisement in parallel to BLE Audio.[/quote]

Thank you for clarifying. Being able to establish the additional ACL connection in parallel is a big step in the right direction - now all that remains is to include and enable the NUS service (and to debug the instability issue, of course).

[quote user="Dekel"]I assume I'll have to use SoftDevice in order to be able to debug with VSCode and have logs. Is that correct?[/quote]

In general yes - and please remember that the ISO support in the SoftDevice Controller is still experimental - but in this case you can also just provide the log output over the UART from your debugger, if you are using the LE Audio controller.
I.e run the application as usual, and open a serial monitor (like PuTTy, or a nRF Terminal in VSC) to see the applications logs.

[quote user="Dekel"]I'm not familiar with it, but I'll take a look, thanks.[/quote]

All right, great! The sniffer is a very powerful tool to wield when developing BLE applications. Please keep in mind that the nRF Sniffer only supports sniffing of ALC ('regular' BLE communication), and not the contents of ISO channels.
For the debugging we are doing now all we need is to look at what is being sent between the devices leading up to the instability you have observed.
It can be daunting to familiarize with the nRF Sniffer tool at first, so please do not hesitate to ask if you should encounter any issues or questions! :)

Best regards,
Karl

RE: Integrate NUS into BLE Audio code

Dekel — Wed, 28 Feb 2024 07:22:56 GMT

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/470926"]Does this mean that you have got NUS up and running on your headset device alongside the ISO functionality, so that you are able to see and interact with the NUS service when connecting to the peripheral advertisement that is running alongside the ISO functionality?[/quote]

No, the NUS is not integrated yet. Only the ability to have a connectable advertisement in parallel to BLE Audio.

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/470926"]If so, could you share the logs from your peripheral device when it fails those 4-5 times before achieving a stable connection?[/quote]

I assume I'll have to use SoftDevice in order to be able to debug with VSCode and have logs. Is that correct?

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/470926"]If nothing stands out in the logs we will need to get a sniffer trace of the communication when you attempt to connect to the peripheral's advertisements - are you familiar with using the nRF Sniffer tool?[/quote]

I'm not familiar with it, but I'll take a look, thanks.

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Tue, 27 Feb 2024 07:39:13 GMT

Hello,

Thank you both for your patience with this.

[quote user="elecia"]

However, the bt_nus_shell.py python script expects the NUS to be part of the scan response of the advertisement. (As do the other NUS tools.)

It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.

Anyway, looking at nrf/samples/bluetooth/multiple_adv_sets, I can see how two advertisements are being created with different data. I haven't gotten it working with NUS yet but it seems like I need to send a additional advertisements and include the NUS data as scan response? Probably in bt_mgmt_adv.c'advertising_process()?

Or is there a simpler way?

[/quote]

You can add the NUS UUID to the scan reponse data by populating the second to last parameter in the extended advertising creation call, or you can modify the filter parameters of the NUS central devices to connect to the name (or other data) advertised by your device.

[quote user="Dekel"]How can I configure that?[/quote]

To use buildprog.py, we can add the command --ctlr="SDC" parameter to switch to the SDC controller.
E.g.,
$ python buildprog.py -c both -b debug -d both -p --pristine--ctlr="SDC"

To use west instead, you can add -DCONFIG_BT_LL_ACS_NRF53=n
E.g.,
$ west build -b nrf5340_audio_dk_nrf5340_cpuapp -d build_headset -p -- -DCONFIG_AUDIO_DEV=1 -DCONFIG_BT_LL_ACS_NRF53=n; west flash --build-dir=build_headset

[quote user="Dekel"] I'm able to connect to my customized advertisement in an unstable way - seems that my phone has to be in constant connecting retries and only then the DK should be powered ON, and that sometimes works only after 4-5 tries. Bottom line, I've established an ACL connection in parallel to LE Audio broadcasting and I'm able to read basic data from the MCU (e.g. model number and manufacturer). I don't have much clue about how to integrate a bidirectional communication platform in order to send data from the app to the MCU. I would appreciate ideas about that.[/quote]

Does this mean that you have got NUS up and running on your headset device alongside the ISO functionality, so that you are able to see and interact with the NUS service when connecting to the peripheral advertisement that is running alongside the ISO functionality?
If so, could you share the logs from your peripheral device when it fails those 4-5 times before achieving a stable connection?
If nothing stands out in the logs we will need to get a sniffer trace of the communication when you attempt to connect to the peripheral's advertisements - are you familiar with using the nRF Sniffer tool?

Best regards,
Karl

RE: Integrate NUS into BLE Audio code

Dekel — Wed, 21 Feb 2024 15:29:07 GMT

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/469905"]so long as you make sure that you are using the SoftDevice Controller.[/quote]

How can I configure that?

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/469905"]Which functionality from the 'multiple advertising sets' example code will you be merging into the nRF5340 LE Audio reference application?[/quote]

The connectable advertisement functionality. I would like to have it in parallel to LE Audio.

[quote userid="44804" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/469916"]Thank you both, Karl and Dekel, it has been really useful to read this thread.[/quote]

I'm glad to hear that! thank you for contributing, it really seems that we are headed in the same direction.

[quote userid="44804" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/469916"]It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.[/quote]

I encountered the same issue. The NUS seems to be deeply rooted within the SDK.

I'll share that at this point I'm able to find both LE Audio broadcast and my customized connectable advertisement with nRF Connect Mobile (with my iPhone). In parallel to audio receiving with another DK (as headset), I'm able to connect to my customized advertisement in an unstable way - seems that my phone has to be in constant connecting retries and only then the DK should be powered ON, and that sometimes works only after 4-5 tries. Bottom line, I've established an ACL connection in parallel to LE Audio broadcasting and I'm able to read basic data from the MCU (e.g. model number and manufacturer). I don't have much clue about how to integrate a bidirectional communication platform in order to send data from the app to the MCU. I would appreciate ideas about that.

RE: Integrate NUS into BLE Audio code

EchoCharlieWhiskey — Wed, 21 Feb 2024 00:40:48 GMT

I've been following this with interest and the multiple advertising sets was quite helpful as a hint for adding NUS to LE Audio.

I'm starting with nrf/application/nrf5340_audio (2.5.1 SDK). I want to add NUS to make a Zephyr console to my audio application (headset).

I can get the console on NUS from nrf/samples/bluetooth/shell_bt_nus. Standalone, that works with nrf/scripts/shell/bt_nus_shell.py.

I've merged shell_bt_nus into nrf5340_audio, adding CONFIG_SHELL_BT_NUS=y and CONFIG_BT_MAX_CONN=2 into my prj.conf then modifying bt_mgmt.c's connect to call shell_bt_nus_enable (and disconnect to call disable).

However, the bt_nus_shell.py python script expects the NUS to be part of the scan response of the advertisement. (As do the other NUS tools.)

It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.

Or is there a simpler way?

(I did look at the older iso_peripheral_example but that code looks very different than the current LE audio applications, I couldn't see how to fit it in.)

It also seems like the Zephyr Settings API is being used to set the name from CONFIG_BT_DEVICE_NAME (and possibly other advertisement settings). I haven't explored that much. I know that is where the name comes from (commit_settings calls bt_set_name) but I couldn't see a way to add the NUS scan response in that code.

Finally, I too would like step-through debugging with VSC. The buildprog.py script is super handy getting going but now that I'm digging into development on my headset features, I want to focus on that code, including stepping through my algorithms.

Thank you both, Karl and Dekel, it has been really useful to read this thread. I think Dekel and I are headed in the same direction which is why we have such similar needs and questions. I hope I've represented Dekel's concerns in my summary.

Best wishes,
ECW

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Tue, 20 Feb 2024 19:54:29 GMT

Hello again, Dekel

In the past thread debugging in VSC was not supported by the LE Audio controller, but now that we have ISO support for the SoftDevice Controller there should be no issue using all the tools available in the nRF Connect VSC extension to work with the application, so long as you make sure that you are using the SoftDevice Controller.
I am however having some troubles verifying that all the functionality works as expected, but I suspect that this is due to some recent changes I've made to my development setup, and so I'll keep looking into that.

[quote user="Dekel"]I'll mention that I encountered the "multiple advertising sets" example code, which seems like a good direction to follow. It's not completely strightforward to me, but I've made some progress. I'll keep you updated.[/quote]

Which functionality from the 'multiple advertising sets' example code will you be merging into the nRF5340 LE Audio reference application? Please do not hesitate to ask if you should encounter any issues or questions while working with this! :)

Best regards,
Karl

RE: Integrate NUS into BLE Audio code

Dekel — Sun, 18 Feb 2024 07:43:43 GMT

Thanks Karl.

I'll mention that I encountered the "multiple advertising sets" example code, which seems like a good direction to follow. It's not completely strightforward to me, but I've made some progress. I'll keep you updated.

Thanks,

Dekel

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Thu, 15 Feb 2024 16:14:14 GMT

Hello again Dekel,

Thank you for your extreme patience with this!

I have been out of office for some days, but now I am back.
I am checking some things and I will get back to you with a proper answer to your questions tomorrow!

Best regards,
Karl

RE: Integrate NUS into BLE Audio code

Dekel — Thu, 01 Feb 2024 08:42:55 GMT

[quote userid="87869" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/466945"]but the command line and buildprog approaches are very convenient once you get familiar with them! :)[/quote]

Do these approaches allow runtime debugging?

[quote userid="114336" url="~/f/nordic-q-a/107763/integrate-nus-into-ble-audio-code/466354"]How should I continue from here?[/quote]

I would appreciate your comment for that, since I'm stuck at this point.

Thanks,

Dekel

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Wed, 31 Jan 2024 13:15:51 GMT

Hello again,

[quote user="Dekel"]I appreciate your quick and detailed response![/quote]

No problem at all, Dekel - I am happy to help! :)

[quote user="Dekel"]In addition, I'm not sure how to work with the example code that you've mentioned (NUS integration) - they use there bt_le_adv_start(), compared to bt_mgmt_adv_start() in the audio example code. I assume only one should be used. Which one is it?[/quote]

This example is very old - it is made using NCS v2.3.0 - so if you wish to run it as is you will need to also use the v2.3.0 code. However, the example still serves as a good illustration of the general approach you must use, and the things you will need to add, in order to merge NUS functionality into the existing nRF5340 LE Audio reference application.

[quote user="Dekel"]What are the currently available options for debugging the audio example code? I'm asking this since this is my base-code, and when building & flashing with buildproj.py it's obviously not possible. In addition, when using VSCode with the nRF extension things don't work as expected, which aligns with the note on the example's documentation page.[/quote]

Which issues are you encountering with the nRF extensions?
You are correct that building and flashing is not supported for the nRF5340 LE Audio application through VSC yet, unfortunately - but the command line and buildprog approaches are very convenient once you get familiar with them! :)

Best regards,
Karl

RE: Integrate NUS into BLE Audio code

Dekel — Sun, 28 Jan 2024 15:54:41 GMT

Hi Karl,

I appreciate your quick and detailed response!

First, I've followed these steps:

1. Cloned NCS repository and checked-out on the latest commit (main branch).

2. Initialized West workspace in the repository's directory ("west init" and then "west update", which both completed successfully).

3. As a sanity check, I tried to build the code with the original buildproj.py. Unfortunately, I got stuck with this error:

Therefore, I decided to go with the nRF Toolchain Manager for installing SDK v2.5.1. After successfully achieving that, once again I've done my sanity check. This time it worked. So my first question is what am I doing wrong with the cloned NCS? How can I fix that?

Continuing with NCS v2.5.1 from the Toolchain Manager, I've followed these steps:

1. Replaced 6 files in the NCS with the files from the two pull-requests you've mentioned (3 files from each PR).

2. As a sanity check, I tried to build the code with the new buildproj.py by using this command:

python buildprog.py -c both -b debug -d both -p --pristine --ctlr="SDC"

Unfortunately, I'm stuck with the error of "no such file or directory" for ble_hci_vsc.h, which is included by bt_mgmt_ctlr_cfg.c and bt_mgmt.c:

How should I continue from here?

In addition, I'm not sure how to work with the example code that you've mentioned (NUS integration) - they use there bt_le_adv_start(), compared to bt_mgmt_adv_start() in the audio example code. I assume only one should be used. Which one is it?

Another important thing I've been wondering about:

What are the currently available options for debugging the audio example code? I'm asking this since this is my base-code, and when building & flashing with buildproj.py it's obviously not possible. In addition, when using VSCode with the nRF extension things don't work as expected, which aligns with the note on the example's documentation page.

This is a critical part of my development process.

Thank you,

Dekel

RE: Integrate NUS into BLE Audio code

Karl Ylvisaker — Thu, 25 Jan 2024 15:07:42 GMT

Hello Dekel,

[quote user=""]Is it possible? If so, which BT controller should be used? How should proj.conf should look like? What other configurations are required?[/quote]

Yes, this is possible - what other requirements do you have to the application / Audio streams?
With the Packetcraft controller there is a hard limitation on the total number of concurrent connections: 2 ISO + 2 ACL, and so you could for instance use this for 2 CIS (1 CIS requires 1 ISO + 1 ACL), or a combination of 1 or 2 BIS + 1 or 2 ACLs.
In case of the latter, you can see an example of how to do this here, but please keep in mind that this is an old example that we do not maintain, but it highlights the general approach you should take to achieve this.

With the SoftDevice Controller there is no such hard limit on the number of concurrent connections, but the downside to using it is that it was very recently updated with ISO functionality, and so it is still experimental and undergoing testing and integration with the nRF5340 LE Audio reference application.
If you would like to give this controller a try (which I recommend), you be on the ncs main, and you must follow the same outline for the NUS inclusion as demonstrated in the referenced example above, but you must also perform these steps when building and flashing the application using the buildprog script:

Checkout latest NCS main, since SDC with ISO is merged after NCS v2.5.0, and also some modification are needed for application to run with SDC
The PR which brings ISO on SDC: https://github.com/nrfconnect/sdk-nrf/pull/12508
The PR which update buildprog.py for switching between SDC and PCFT https://github.com/nrfconnect/sdk-nrf/pull/12996
For using buildprog.py, we can add the command --ctlr="SDC" in the end
E.g., $ python buildprog.py -c both -b debug -d both -p --pristine--ctlr="SDC"

Please also keep in mind that the SoftDevice Controller still does not have support for DFU with the nRF5340 LE Audio application for the time being.

Best regards,
Karl