Integrate NUS into BLE Audio code

[Dekel]

Hi,

I'm working with nRF5340 Audio DK.

I'm able to successfully run the audio example in BIS mode (with gateway and headset devices), and also to run the Peripheral UART Bluetooth example separately on my DK.

My goal is to open another BT connection, in parallel to the audio broadcast on my gateway device, to which I can connect with the nRF Toolbox App.

Is it possible? If so, which BT controller should be used? How should proj.conf should look like? What other configurations are required?

I've noticed that the UART example code uses adv.c for advertising, while the audio example code uses bt_mgmt.c.

Thank you,

Dekel

[Karl Ylvisaker]

Hello Dekel,

Is it possible? If so, which BT controller should be used? How should proj.conf should look like? What other configurations are required?

Yes, this is possible - what other requirements do you have to the application / Audio streams?
With the Packetcraft controller there is a hard limitation on the total number of concurrent connections: 2 ISO + 2 ACL, and so you could for instance use this for 2 CIS (1 CIS requires 1 ISO + 1 ACL), or a combination of 1 or 2 BIS + 1 or 2 ACLs.
In case of the latter, you can see an example of how to do this here, but please keep in mind that this is an old example that we do not maintain, but it highlights the general approach you should take to achieve this.

With the SoftDevice Controller there is no such hard limit on the number of concurrent connections, but the downside to using it is that it was very recently updated with ISO functionality, and so it is still experimental and undergoing testing and integration with the nRF5340 LE Audio reference application.
If you would like to give this controller a try (which I recommend), you be on the ncs main, and you must follow the same outline for the NUS inclusion as demonstrated in the referenced example above, but you must also perform these steps when building and flashing the application using the buildprog script:

1. Checkout latest NCS main, since SDC with ISO is merged after NCS v2.5.0, and also some modification are needed for application to run with SDC
   The PR which brings ISO on SDC: https://github.com/nrfconnect/sdk-nrf/pull/12508
   The PR which update buildprog.py for switching between SDC and PCFT https://github.com/nrfconnect/sdk-nrf/pull/12996
2. For using buildprog.py, we can add the command --ctlr="SDC" in the end
   E.g., $ python buildprog.py -c both -b debug -d both -p --pristine--ctlr="SDC"

Please also keep in mind that the SoftDevice Controller still does not have support for DFU with the nRF5340 LE Audio application for the time being.

Best regards,
Karl

[Dekel]

Hi Karl,

I appreciate your quick and detailed response!

First, I've followed these steps:

1. Cloned NCS repository and checked-out on the latest commit (main branch).

2. Initialized West workspace in the repository's directory ("west init" and then "west update", which both completed successfully).

3. As a sanity check, I tried to build the code with the original buildproj.py. Unfortunately, I got stuck with this error:

Therefore, I decided to go with the nRF Toolchain Manager for installing SDK v2.5.1. After successfully achieving that, once again I've done my sanity check. This time it worked. So my first question is what am I doing wrong with the cloned NCS? How can I fix that?

Continuing with NCS v2.5.1 from the Toolchain Manager, I've followed these steps:

1. Replaced 6 files in the NCS with the files from the two pull-requests you've mentioned (3 files from each PR).

2. As a sanity check, I tried to build the code with the new buildproj.py by using this command:

python buildprog.py -c both -b debug -d both -p --pristine --ctlr="SDC"

Unfortunately, I'm stuck with the error of "no such file or directory" for ble_hci_vsc.h, which is included by bt_mgmt_ctlr_cfg.c and bt_mgmt.c:

How should I continue from here?

In addition, I'm not sure how to work with the example code that you've mentioned (NUS integration) - they use there bt_le_adv_start(), compared to bt_mgmt_adv_start() in the audio example code. I assume only one should be used. Which one is it?

Another important thing I've been wondering about:

What are the currently available options for debugging the audio example code? I'm asking this since this is my base-code, and when building & flashing with buildproj.py it's obviously not possible. In addition, when using VSCode with the nRF extension things don't work as expected, which aligns with the note on the example's documentation page.

This is a critical part of my development process.

Thank you,

Dekel

[Karl Ylvisaker]

Hello again, Dekel

In the past thread debugging in VSC was not supported by the LE Audio controller, but now that we have ISO support for the SoftDevice Controller there should be no issue using all the tools available in the nRF Connect VSC extension to work with the application, so long as you make sure that you are using the SoftDevice Controller.
I am however having some troubles verifying that all the functionality works as expected, but I suspect that this is due to some recent changes I've made to my development setup, and so I'll keep looking into that.

Dekel said:

I'll mention that I encountered the "multiple advertising sets" example code, which seems like a good direction to follow. It's not completely strightforward to me, but I've made some progress. I'll keep you updated.

Which functionality from the 'multiple advertising sets' example code will you be merging into the nRF5340 LE Audio reference application? Please do not hesitate to ask if you should encounter any issues or questions while working with this! :) 

Best regards,
Karl

[EchoCharlieWhiskey]

I've been following this with interest and the multiple advertising sets was quite helpful as a hint for adding NUS to LE Audio.

I'm starting with nrf/application/nrf5340_audio (2.5.1 SDK). I want to add NUS to make a Zephyr console to my audio application (headset).

I can get the console on NUS from nrf/samples/bluetooth/shell_bt_nus. Standalone, that works with nrf/scripts/shell/bt_nus_shell.py.

I've merged shell_bt_nus into nrf5340_audio, adding CONFIG_SHELL_BT_NUS=y and  CONFIG_BT_MAX_CONN=2 into my prj.conf then modifying bt_mgmt.c's connect to call shell_bt_nus_enable (and disconnect to call disable).

However, the bt_nus_shell.py python script expects the NUS to be part of the scan response of the advertisement. (As do the other NUS tools.)

It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.

Anyway, looking at nrf/samples/bluetooth/multiple_adv_sets, I can see how two advertisements are being created with different data. I haven't gotten it working with NUS yet but it seems like I need to send a additional advertisements and include the NUS data as scan response? Probably  in bt_mgmt_adv.c'advertising_process()?

Or is there a simpler way?

(I did look at the older iso_peripheral_example but that code looks very different than the current LE audio applications, I couldn't see how to fit it in.)

It also seems like the Zephyr Settings API is being used to set the name from CONFIG_BT_DEVICE_NAME (and possibly other advertisement settings). I haven't explored that much. I know that is where the name comes from (commit_settings calls bt_set_name) but I couldn't see a way to add the NUS scan response in that code. 

Finally, I too would like step-through debugging with VSC. The buildprog.py script is super handy getting going but now that I'm digging into development on my headset features, I want to focus on that code, including stepping through my algorithms. 

Thank you both, Karl and Dekel, it has been really useful to read this thread. I think Dekel and I are headed in the same direction which is why we have such similar needs and questions. I hope I've represented Dekel's concerns in my summary. 

Best wishes,
ECW

[Dekel]

Karl Ylvisaker said:

so long as you make sure that you are using the SoftDevice Controller.

How can I configure that?

Karl Ylvisaker said:

Which functionality from the 'multiple advertising sets' example code will you be merging into the nRF5340 LE Audio reference application?

The connectable advertisement functionality. I would like to have it in parallel to LE Audio.

EchoCharlieWhiskey said:

Thank you both, Karl and Dekel, it has been really useful to read this thread.

I'm glad to hear that! thank you for contributing, it really seems that we are headed in the same direction.

EchoCharlieWhiskey said:

It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.

I encountered the same issue. The NUS seems to be deeply rooted within the SDK.

I'll share that at this point I'm able to find both LE Audio broadcast and my customized connectable advertisement with nRF Connect Mobile (with my iPhone). In parallel to audio receiving with another DK (as headset), I'm able to connect to my customized advertisement in an unstable way - seems that my phone has to be in constant connecting retries and only then the DK should be powered ON, and that sometimes works only after 4-5 tries. Bottom line, I've established an ACL connection in parallel to LE Audio broadcasting and I'm able to read basic data from the MCU (e.g. model number and manufacturer). I don't have much clue about how to integrate a bidirectional communication platform in order to send data from the app to the MCU. I would appreciate ideas about that.

[Karl Ylvisaker]

Hello,

Thank you both for your patience with this.

EchoCharlieWhiskey said:

However, the bt_nus_shell.py python script expects the NUS to be part of the scan response of the advertisement. (As do the other NUS tools.)

It looks like the nrf5340_audio starts the advertisement via bt_mgmt_adc.c's bt_mgmt_adv_start(...) in streamctrl_unicast_server.c. I don't see how to modify that to add NUS to this.

Anyway, looking at nrf/samples/bluetooth/multiple_adv_sets, I can see how two advertisements are being created with different data. I haven't gotten it working with NUS yet but it seems like I need to send a additional advertisements and include the NUS data as scan response? Probably  in bt_mgmt_adv.c'advertising_process()?

Or is there a simpler way?

You can add the NUS UUID to the scan reponse data by populating the second to last parameter in the extended advertising creation call, or you can modify the filter parameters of the NUS central devices to connect to the name (or other data) advertised by your device.

Dekel said:

How can I configure that?

To use buildprog.py, we can add the command --ctlr="SDC" parameter to switch to the SDC controller.
E.g.,
$ python buildprog.py -c both -b debug -d both -p --pristine--ctlr="SDC"

To use west instead, you can add -DCONFIG_BT_LL_ACS_NRF53=n
E.g.,
$ west build -b nrf5340_audio_dk_nrf5340_cpuapp -d build_headset -p -- -DCONFIG_AUDIO_DEV=1 -DCONFIG_BT_LL_ACS_NRF53=n; west flash --build-dir=build_headset

Dekel said:

I'm able to connect to my customized advertisement in an unstable way - seems that my phone has to be in constant connecting retries and only then the DK should be powered ON, and that sometimes works only after 4-5 tries. Bottom line, I've established an ACL connection in parallel to LE Audio broadcasting and I'm able to read basic data from the MCU (e.g. model number and manufacturer). I don't have much clue about how to integrate a bidirectional communication platform in order to send data from the app to the MCU. I would appreciate ideas about that.

Does this mean that you have got NUS up and running on your headset device alongside the ISO functionality, so that you are able to see and interact with the NUS service when connecting to the peripheral advertisement that is running alongside the ISO functionality?
If so, could you share the logs from your peripheral device when it fails those 4-5 times before achieving a stable connection?
If nothing stands out in the logs we will need to get a sniffer trace of the communication when you attempt to connect to the peripheral's advertisements - are you familiar with using the nRF Sniffer tool?

Best regards,
Karl

[Dekel]

Karl Ylvisaker said:

Does this mean that you have got NUS up and running on your headset device alongside the ISO functionality, so that you are able to see and interact with the NUS service when connecting to the peripheral advertisement that is running alongside the ISO functionality?

No, the NUS is not integrated yet. Only the ability to have a connectable advertisement in parallel to BLE Audio.

Karl Ylvisaker said:

If so, could you share the logs from your peripheral device when it fails those 4-5 times before achieving a stable connection?

I assume I'll have to use SoftDevice in order to be able to debug with VSCode and have logs. Is that correct?

Karl Ylvisaker said:

If nothing stands out in the logs we will need to get a sniffer trace of the communication when you attempt to connect to the peripheral's advertisements - are you familiar with using the nRF Sniffer tool?

I'm not familiar with it, but I'll take a look, thanks.

[Karl Ylvisaker]

Hello,

Dekel said:

No, the NUS is not integrated yet. Only the ability to have a connectable advertisement in parallel to BLE Audio.

Thank you for clarifying. Being able to establish the additional ACL connection in parallel is a big step in the right direction - now all that remains is to include and enable the NUS service (and to debug the instability issue, of course).

Dekel said:

I assume I'll have to use SoftDevice in order to be able to debug with VSCode and have logs. Is that correct?

In general yes - and please remember that the ISO support in the SoftDevice Controller is still experimental - but in this case you can also just provide the log output over the UART from your debugger, if you are using the LE Audio controller.
I.e run the application as usual, and open a serial monitor (like PuTTy, or a nRF Terminal in VSC) to see the applications logs.

Dekel said:

I'm not familiar with it, but I'll take a look, thanks.

All right, great! The sniffer is a very powerful tool to wield when developing BLE applications. Please keep in mind that the nRF Sniffer only supports sniffing of ALC ('regular' BLE communication), and not the contents of ISO channels.
For the debugging we are doing now all we need is to look at what is being sent between the devices leading up to the instability you have observed.
It can be daunting to familiarize with the nRF Sniffer tool at first, so please do not hesitate to ask if you should encounter any issues or questions! :) 

Best regards,
Karl

[Dekel]

I'll share current code modifications with a hope for help:

modifications.rar