nrf5340 audio DK: transmit two headsets' mic data to one gateway

[lincoln-lu]

Hi,

I'd like to evaluate dual mic recording on three 5340 audio DKs: two of them work as CIS headsets(Unicast server?) and the other one as CIS gateway(Unicast client?).
The two headsets will capture the voice data by the onboard PDM mic and encode and transmit mic data to the gateway via LE audio CIS stream from left and right channel respectively.

And when the gateway is configured to use I2S audio then it should playback the recorded mic data via its headphone jack, otherwise it can send that to USB host as a soundcard.

My question: Is there any workable example code from Nordic for this kind of application?

I found that there is a branch https://github.com/rick1082/sdk-nrf/commits/dual_mic_sample_2_4_2 crreated by Nordic engineer Rick in 2022, based on NCS2.4.2, I compiled and flash both CIS headset and gateway firmware images into my three 5340 audio DKs based on that code, but it does not work at all!  I see error "Failed to establish CIS, ret = -22" in gateway log,

Here are the more logs from the gateway:

GW [00:01:46.169,555] <dbg> cis_gateway: ep_sta*** Booting Zephyr OS build v3.3.99-ncs1-2 ***
GW [00:00:00.255,249] <dbg> main: main: nRF5340 APP core started
GW [00:00:00.255,462] <inf> fw_info: 
	 nRF5340 Audio nRF5340 Audio DK cpuapp 			    
	 NCS base version: 2.4.2 			    
	 Cmake run : Tue Dec 17 16:51:52 2024
GW [00:00:00.255,462] <inf> fw_info: ------- DEBUG BUILD -------
GW [00:00:00.255,493] <inf> fw_info: Compiled for GATEWAY device
GW [00:00:00.266,113] <inf> board_version: Compatible board/HW version found: 1.0.0
GW [00:00:00.303,741] <wrn> bt_hci_core: Controller to host flow control not supported
GW [00:00:00.306,884] <inf> ble: MAC: 00:00:00:00:00:00 (public)
GW [00:00:00.307,464] <inf> ble: Controller version: 3349
GW [00:00:00.309,417] <inf> bt_keys: SC LTK: 0x1b081f1d4c6e0c8e9cc78196b5bfb645
GW [00:00:00.309,417] <inf> bt_keys: SC LTK: 0xc27ee2a3d8723fdc80e002fc08b0b3a3
GW [00:00:00.427,185] <inf> cis_gateway: Stored bonding found: CD:28:C6:83:F0:4D (random)
GW [00:00:00.427,276] <inf> cis_gateway: Stored bonding found: EA:1F:9F:80:A9:0A (random)
GW [00:00:00.427,276] <inf> cis_gateway: All bonded slots filled, will not accept new devices
GW [00:00:00.429,290] <inf> cis_gateway: Scanning successfully started
GW [00:00:01.186,523] <dbg> cis_gateway: bond_connect: Found bonded device
GW [00:00:01.393,768] <inf> cis_gateway: Connected: EA:1F:9F:80:A9:0A (random)
GW [00:00:01.394,012] <dbg> cis_gateway: connected_cb: TX power set to 0 dBm for connection 0x20005e98
GW [00:00:01.574,462] <dbg> cis_gateway: security_changed_cb: Security changed: level 2
GW [00:00:02.014,617] <inf> cis_gateway: Stored bonding found: CD:28:C6:83:F0:4D (random)
GW [00:00:02.014,709] <inf> cis_gateway: Stored bonding found: EA:1F:9F:80:A9:0A (random)
GW [00:00:02.014,709] <inf> cis_gateway: All bonded slots filled, will not accept new devices
GW [00:00:02.015,319] <inf> cis_gateway: Scanning successfully started
GW [00:00:02.223,114] <dbg> cis_gateway: bond_connect: Found bonded device
GW [00:00:02.326,232] <inf> cis_gateway: Connected: CD:28:C6:83:F0:4D (random)
GW [00:00:02.326,446] <dbg> cis_gateway: connected_cb: TX power set to 0 dBm for connection 0x20005f60
GW [00:00:02.414,642] <dbg> cis_gateway: available_contexts_cb: conn: EA:1F:9F:80:A9:0A (random), snk ctx 7 src ctx 7

GW [00:00:02.514,495] <dbg> cis_gateway: security_changed_cb: Security changed: level 2
GW [00:00:02.954,589] <inf> cis_gateway: All headsets connected
GW [00:00:03.054,565] <dbg> cis_gateway: discover_source_cb: Source discover complete: err 0
GW [00:00:03.054,595] <dbg> bt_bap_stream: bt_bap_stream_config: conn 0x20005e98 stream 0x20001f0c, ep 0x20011af4 codec 0x200003b4 codec id 0x06 codec cid 0x0000 codec vid 0x0000
GW [00:00:03.054,595] <dbg> bt_bap_stream: bt_bap_stream_attach: conn 0x20005e98 stream 0x20001f0c ep 0x20011af4 codec 0x200003b4
GW [00:00:03.135,406] <inf> cis_gateway: LEFT source stream configured
GW [00:00:03.135,406] <dbg> bt_bap_stream: bt_bap_stream_qos: conn 0x20005e98 group 0x20012234
GW [00:00:03.135,437] <dbg> bt_bap_stream: bt_bap_stream_verify_qos: Latency 20 higher than preferred max 10
GW [00:00:03.214,721] <dbg> cis_gateway: stream_qos_set_cb: Set LEFT to PD: 10000
GW [00:00:03.214,721] <dbg> bt_bap_stream: bt_bap_stream_enable: stream 0x20001f0c
GW [00:00:03.214,782] <inf> cis_gateway: Enable stream 0x20001f0c
GW [00:00:03.295,257] <inf> cis_gateway: Stream enabled: 0x20001f0c
GW [00:00:03.295,257] <dbg> cis_gateway: stream_enabled_cb: k_msg_put: ch: 0, dir: 1, retries 0
GW [00:00:03.295,288] <dbg> cis_gateway: stream_enabled_cb: k_msg_put: ch: 0, dir: 2, retries 0
GW [00:00:03.295,318] <dbg> cis_gateway: work_stream_start: k_msg_get: ch: 0, dir: 1, retries 0
GW [00:00:03.295,349] <dbg> bt_bap_stream: bt_bap_stream_start: stream 0x20001cec ep (nil)
GW [00:00:03.295,349] <dbg> bt_bap_stream: bt_bap_stream_start: Invalid stream
GW [00:00:03.295,349] <err> cis_gateway: Failed to establish CIS, ret = -22
GW [00:00:03.295,379] <dbg> cis_gateway: work_stream_start: k_msg_get: ch: 0, dir: 2, retries 0
GW [00:00:03.295,379] <dbg> bt_bap_stream: bt_bap_stream_start: stream 0x20001f0c ep 0x20011af4
GW [00:00:03.354,675] <dbg> cis_gateway: available_contexts_cb: conn: CD:28:C6:83:F0:4D (random), snk ctx 7 src ctx 7

GW [00:00:03.654,663] <inf> cis_gateway: Stream 0x20001f0c started
GW [00:00:03.654,724] <dbg> streamctrl: le_audio_msg_sub_thread: Received event = 2, current state = 1
GW [00:00:03.654,754] <dbg> streamctrl: le_audio_msg_sub_thread: LE audio evt streaming
GW [00:00:03.659,240] <dbg> sw_codec_select: sw_codec_init: Encode: 48000Hz 16bits 10000us 64000bps 2 channel(s)
GW [00:00:03.659,759] <dbg> sw_codec_select: sw_codec_init: Decode: 48000Hz 16bits 10000us 2 channel(s)
GW [00:00:03.681,152] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.681,182] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.681,182] <wrn> streamctrl: Problem with sending LE audio data, ret: -140
GW [00:00:03.692,169] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.692,199] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.701,416] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.701,446] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.711,791] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.711,791] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.721,252] <dbg> cis_gateway: ep_state_check: Endpoint is NULL
GW [00:00:03.721,252] <dbg> cis_gateway: ep_state_check: Endpoint is NULL

Can you take a look at it? If there is any new branch that implemented what I want, please let me know. 

Thanks

[Simonr]

Hi Lincoln

So, the demo you link to is an old version with running on the packetcraft controller, which we are no longer able to support unfortunately.

However, one of our devs has made a newer version of the dual mic demo running in the nRF Connect SDK. It's available here, but please note that it is an experimental demo, so we do not have any warranty or guarantees that this project is at a production level quality, and if you decide to use it for development and/or production, thorough testing should be done on your end to make sure it is working as intended. I hope it is helpful, at least for demo purposes.

https://github.com/rick1082/sdk-nrf/tree/2t1r_mic_demo 

Best regards,

Simon

[lincoln-lu]

Simonr Thanks for your support!

In fact, during your holiday weeks, I ported the code from the old branch https://github.com/rick1082/sdk-nrf/commits/dual_mic_sample_2_4_2 to NCS 2.9.0 by myself and it seem 'work' at some extend. I can hear the mic data from both left and right headsets on the headphone jack of the 5340 audio DK as Gateway.

Now one thing I want to confirm with you: I connected a stereo headphone to the gateway audio DK's headphone jack, in my left and right ears, I found the voice from left and right headsets are mixed together, instead of true "stereo". Is it because the  HW codec CS47L63 on the Audio DK can ONLY support MONO DAC? 

Thanks

[Simonr]

Hi Lincoln

lincoln-lu said:

I found the voice from left and right headsets are mixed together, instead of true "stereo". Is it because the  HW codec CS47L63 on the Audio DK can ONLY support MONO DAC?

This is the default behavior of the old sample that you have ported to 2.9.0, as this was a request from another customer at that time. You can check the last commit from that branch for details on what changes were made. Although, even removing the last commit, you'll likely still hear the mix of L and R from the gateway because of the CS47L63 only being able to output one channel, and the codec will mix L and R somewhat before outputting.

lincoln-lu said:

I see that 2 CIS connections are OK, but each time only one headset's mic data can be decoded by gateway.

This is a controller issue that occurs at 96kbps bit rate I'm pretty sure. The developer in charge has updated the demo branch to avoid this, so please try doing a pull of the branch and test again. 

In addition to this, the new SDC controller that took over from packetcraft, supports a different transport latency, so there's a chance that the data receiving form the gateway is out of order (L, L, R, R, L, R, R, L), so the old sample might not work as intended in NCS 2.9.0

Best regards,

[lincoln-lu]

Simonr said:

This is a controller issue that occurs at 96kbps bit rate I'm pretty sure. The developer in charge has updated the demo branch to avoid this, so please try doing a pull of the branch and test again. 

Yes, I just notice that your developer has commited new code to enable 124kbps, and removed 96kbps today. as below
#CONFIG_BT_AUDIO_BITRATE_UNICAST_SRC=160000
CONFIG_BT_AUDIO_BITRATE_UNICAST_SRC=124000

And I was using the code of 2025-01-07 https://github.com/rick1082/sdk-nrf/tree/2t1r_mic_demo that is defaulted to 96kbps.

Did you mean 96kbps still has known issue and you can not fix it  so far? Which bitrate do you recommend then?

Simonr said:

In addition to this, the new SDC controller that took over from packetcraft, supports a different transport latency, so there's a chance that the data receiving form the gateway is out of order (L, L, R, R, L, R, R, L), so the old sample might not work as intended in NCS 2.9.0

I have already been building code of https://github.com/rick1082/sdk-nrf/tree/2t1r_mic_demo by NCS 2.9.0 that was downloaded by VSCode NCS extention.
Can you tell me how to check if I am using the new SDC controller that took over from packetcraft?

[Simonr]

We recommend 124kbps in this demo, as there is an issue with 96kbps that we have not been able to fix on our end.