nRF5340 Audio - configure gateway unit also as ACL peripheral

[Dekel]

Hi,

I'm working with nRF5340 Audio DKs.

I'm able to successfully run the audio example in BIS mode (with gateway and headset devices).

My goal is to open another BT connection, in parallel to the audio broadcast on my gateway device, to which I can connect with the headset units and the nRF Toolbox App via NUS. I'd like the gateway device to act as peripheral in that manner and to be able to connect to 6 different devices concurrently via NUS (5 headset units + 1 smartphone app).

I tried implementing it by myself, and I feel like on the brink of it, but calling bt_le_ext_adv_create() twice (once for audio and once for ACL) fails with EIO (I/O error). It's unclear to me why it happens. As I see it, the challange is to create 2 concurrent advertisements in the gateway unit - audio streaming and ACL connection.

On this ticket I received great help from Brian Kim, which implemented the gateway device as central in terms of ACL connection.

I'll share that I was able to deduce from his development and successfully configure the headset device as central in terms of ACL connection (in parallel to audio streaming receival). here are the relevant files: headset_as_ACL_central.rar

I would appreciate any help with such implementation.

Thank you,

Dekel

[Brian Kim]

Thanks for your patience, Dekel.

Please the attached file. As you know, adding ACL link affects the LE audio performance directly. so you can tune the related parameters number of retransmissions, bitrate, advertising interval and so on. For better performance, you should tune the numbers accordingly. 

  

Best regards,

BrianModifiedFiles.7z

[Dekel]

Hi Brian,

I tried to run the code you've shared as it is. When trying to connect to the gateway unit with my iPhone, either through nRF connect for mobile or nRF Toolbox, it doesn't succeed.

I receive the error of BT_HCI_ERR_CONN_FAIL_TO_ESTAB and the unit constantly connects and disconnects from the iPhone:

I'll mention that there's no I2S audio source connected to the gateway unit and also there's no headset unit around it. That's in order to avoid interference.

I tried to manipulate prj.conf, but with no success.

I would appreciate your help.

Thanks,

Dekel

[Dekel]

Hi Brian,

In order to test the Android functionality properly, I've cloned the sdk-nrf to my PC and switched to the "v2.6.0" tag. As a sanity check, I've built it as is using the original buildprog.py (with --pristine). It worked well.

The next thing I did was replacing the 4 files you've mentioned above and then using buildprog again. It was built & flashed successfully to 2 units (one headset and one gateway).

I'm able to stream audio through the gateway unit and receive it on the headset unit, but when I try to connect to the NUS advertisment with an Android smartphone, I get this error (BT_HCI_ERR_CONN_FAIL_TO_ESTAB):

I've tried multiple times, both with nRF Connect For Mobile and nRF Toolbox apps. Furthermore, my Android's bonding list is empty.

I would appreciate your help with this.

In addition, I'd appreciate an update regarding the iOS communication. Our project will include both Android and iPhone support.

Thank you,

Dekel

[Brian Kim]

Hello Dekel,

Thanks for your update.

Dekel said:

I'm able to stream audio through the gateway unit and receive it on the headset unit, but when I try to connect to the NUS advertisment with an Android smartphone, I get this error (BT_HCI_ERR_CONN_FAIL_TO_ESTAB):

Can you share the binary(merged-domains.hex) of the gateway? I will try it on my side to check. And also, can you tell me what Android model you are testing?

Dekel said:

In addition, I'd appreciate an update regarding the iOS communication. Our project will include both Android and iPhone support.

I believe iOS seems to have trouble with ADV-EXT for connection(NUS). We keep on eye on their response or guide. 

Thank you,

Brian

 

[Dekel]

Hi Brian,

thanks for the quick response!

Brian Kim said:

Can you share the binary(merged-domains.hex) of the gateway?

Coudln't find such file. Here's my build folder: 5822.build.rar

Brian Kim said:

can you tell me what Android model you are testing?

Samsung Galaxy A03s, Android version 13.

Brian Kim said:

I believe iOS seems to have trouble with ADV-EXT for connection(NUS). We keep on eye on their response or guide. 

Do you think it will be possible to connect with iOS?

Thank you,

Dekel

[Brian Kim]

Dekel said:

Coudln't find such file. Here's my build folder: 5822.build.rar

It didn't work on my side either, but it was different error(BT_HCI_ERR_CONN_TIMEOUT : 0x08).  Please find the attached my binary and try it on your side. You can just drag & drop and program it using nRF Programmer tool.

Dekel said:

Samsung Galaxy A03s, Android version 13.

On my side, the test device is Pixel 6, Android version 14.

Dekel said:

Do you think it will be possible to connect with iOS?

We've been talking to the team and I will keep you updated. At the worst case, we can advertise with legacy ADV for the connection and that should work without issues.

2337.merged_domains_BK.zip

Best regards,

Brian

[Dekel]

Brian Kim said:

Please find the attached my binary and try it on your side. You can just drag & drop and program it using nRF Programmer tool.

I did just that - still receive the same error (BT_HCI_ERR_CONN_FAIL_TO_ESTAB).

Brian Kim said:

At the worst case, we can advertise with legacy ADV for the connection and that should work without issues.

Sounds like the legacy ADV could be an adequate solution for the Android as well, isn't it?

[Brian Kim]

Dekel said:

I did just that - still receive the same error (BT_HCI_ERR_CONN_FAIL_TO_ESTAB).

It looks like QoS issue. You should find the optimized parameters tweaking them for your device. This information should be useful to figure out proper number of the parameters.

Dekel said:

Sounds like the legacy ADV could be an adequate solution for the Android as well, isn't it?

For iOS, if Apple doesn't change anything about this issue, that could be an alternatives. For Android, as I said above, this would be more of QoS timing issue rather than ADV-EXT, but it's worth a shot.

[Dekel]

Thanks for the info.

Brian Kim said:

You should find the optimized parameters tweaking them for your device.

Interestingly, when trying to connect with another smartphone, running Android 12, everything works well using your original code! That's awesome. I'm able to communicate through nRF Toolbox app and even send data from the device back to the smartphone (with a small code modification).

Therefore, I assume that's something about the smartphone I've been using. I'll try with a few more devices to get better statistics.

Brian Kim said:

For iOS, if Apple doesn't change anything about this issue, that could be an alternatives. For Android, as I said above, this would be more of QoS timing issue rather than ADV-EXT, but it's worth a shot.

Is it complicated to implement the legacy ADV? If it's relatively simple and you assume it should work without issues, I prefer proceeding in that direction in order to make progress. I'd really appreciate your guidance with that.

[Brian Kim]

Thanks for the good news!

Dekel said:

Is it complicated to implement the legacy ADV? If it's relatively simple and you assume it should work without issues, I prefer proceeding in that direction in order to make progress. I'd really appreciate your guidance with that.

It would be a little complicated, but it's very promising. However, replacing with legacy ADV might connect iOS devices, but that won't completely fix this issue because this comes from timing-activities in the end. Are you satisfied with the current audio quality? Also, even slightly lower quality than this would be acceptable? If yes, that will bring more stable ACL link.

Best regards,

Brian

[Brian Kim]

Hello Dekel,

In the meantime, we tried it out for legacy advertising for the ACL connection.

It's not fully optimized yet, but it looks good for both iPhone and Android. you can simply merge the attached one file((\broadcast_source\main.c) to your previous patch. Feel free to comment about your result.

//Brian

Fullscreen 3580.main.c Download

/*
 * Copyright (c) 2023 Nordic Semiconductor ASA
 *
 * SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
 */

#include "streamctrl.h"

#include <zephyr/bluetooth/audio/audio.h>
#include <zephyr/kernel.h>
#include <zephyr/zbus/zbus.h>

#include "nrf5340_audio_common.h"
#include "nrf5340_audio_dk.h"
#include "broadcast_source.h"
#include "led.h"
#include "button_assignments.h"
#include "macros_common.h"
#include "audio_system.h"
#include "bt_mgmt.h"

//Adding for NUS peri.
#include <bluetooth/services/nus.h>
//

#include <zephyr/logging/log.h>
LOG_MODULE_REGISTER(main, CONFIG_MAIN_LOG_LEVEL);

ZBUS_SUBSCRIBER_DEFINE(button_evt_sub, CONFIG_BUTTON_MSG_SUB_QUEUE_SIZE);

ZBUS_MSG_SUBSCRIBER_DEFINE(le_audio_evt_sub);

ZBUS_CHAN_DECLARE(button_chan);
ZBUS_CHAN_DECLARE(le_audio_chan);
ZBUS_CHAN_DECLARE(bt_mgmt_chan);
ZBUS_CHAN_DECLARE(sdu_ref_chan);

ZBUS_OBS_DECLARE(sdu_ref_msg_listen);

static struct k_thread button_msg_sub_thread_data;
static struct k_thread le_audio_msg_sub_thread_data;

//Adding for NUS peri.
static struct bt_le_ext_adv *adv_for_conn;
//

static k_tid_t button_msg_sub_thread_id;
static k_tid_t le_audio_msg_sub_thread_id;

K_THREAD_STACK_DEFINE(button_msg_sub_thread_stack, CONFIG_BUTTON_MSG_SUB_STACK_SIZE);
K_THREAD_STACK_DEFINE(le_audio_msg_sub_thread_stack, CONFIG_LE_AUDIO_MSG_SUB_STACK_SIZE);

static enum stream_state strm_state = STATE_PAUSED;

//Advertise with Legacy mode(instead of ADV-EXT) for ACL connection..
#define CONFIG_BT_LEGACY_ADV4ACL

#ifndef CONFIG_BT_LEGACY_ADV4ACL //Adding for NUS peri. w/ ADV-EXT.
#define BT_LE_EXT_ADV_CONN BT_LE_ADV_PARAM(BT_LE_ADV_OPT_EXT_ADV | \
						BT_LE_ADV_OPT_CONNECTABLE, \
						BT_GAP_ADV_FAST_INT_MIN_1, \
						BT_GAP_ADV_FAST_INT_MAX_1, \
						NULL)
#else	//Adding for NUS peri. w/ legacy ADV.
#define BT_LE_ADV_CONN_CUSTOM BT_LE_ADV_PARAM(BT_LE_ADV_OPT_CONNECTABLE, \
						BT_GAP_ADV_FAST_INT_MIN_1, \
						BT_GAP_ADV_FAST_INT_MAX_1, \
						NULL)
#endif

#define DEVICE_NAME "BROADCASTER_NUS"
#define DEVICE_NAME_LEN	(sizeof(DEVICE_NAME) - 1)

#ifndef CONFIG_BT_LEGACY_ADV4ACL
static const struct bt_data ad[] = {
	BT_DATA_BYTES(BT_DATA_FLAGS, (BT_LE_AD_GENERAL | BT_LE_AD_NO_BREDR)),
	BT_DATA(BT_DATA_NAME_SHORTENED, DEVICE_NAME, DEVICE_NAME_LEN),
	BT_DATA_BYTES(BT_DATA_UUID128_ALL, BT_UUID_NUS_VAL),
};
#else
static const struct bt_data ad[] = {
	BT_DATA_BYTES(BT_DATA_FLAGS, (BT_LE_AD_GENERAL | BT_LE_AD_NO_BREDR)),
	BT_DATA(BT_DATA_NAME_SHORTENED, DEVICE_NAME, DEVICE_NAME_LEN),
};
static const struct bt_data sd[] = {
	BT_DATA_BYTES(BT_DATA_UUID128_ALL, BT_UUID_NUS_VAL),
};
#endif

static void bt_receive_cb(struct bt_conn *conn, const uint8_t *const data, uint16_t len)
{
	LOG_HEXDUMP_INF(data, len, " NUS");
}

static struct bt_nus_cb nus_cb = {
	.received = bt_receive_cb,	
};
//

/* Function for handling all stream state changes */
static void stream_state_set(enum stream_state stream_state_new)
{
	strm_state = stream_state_new;
}

/**
 * @brief	Handle button activity.
 */
static void button_msg_sub_thread(void)
{
	int ret;
	const struct zbus_channel *chan;

	while (1) {
		ret = zbus_sub_wait(&button_evt_sub, &chan, K_FOREVER);
		ERR_CHK(ret);

		struct button_msg msg;

		ret = zbus_chan_read(chan, &msg, ZBUS_READ_TIMEOUT_MS);
		ERR_CHK(ret);

		LOG_DBG("Got btn evt from queue - id = %d, action = %d", msg.button_pin,
			msg.button_action);

		if (msg.button_action != BUTTON_PRESS) {
			LOG_WRN("Unhandled button action");
			return;
		}

		switch (msg.button_pin) {
		case BUTTON_PLAY_PAUSE:
			if (strm_state == STATE_STREAMING) {
				ret = broadcast_source_stop();
				if (ret) {
					LOG_WRN("Failed to stop broadcaster: %d", ret);
				}
			} else if (strm_state == STATE_PAUSED) {
				ret = broadcast_source_start(NULL);
				if (ret) {
					LOG_WRN("Failed to start broadcaster: %d", ret);
				}
			} else {
				LOG_WRN("In invalid state: %d", strm_state);
			}

			break;

		case BUTTON_4:
			if (IS_ENABLED(CONFIG_AUDIO_TEST_TONE)) {
				if (strm_state != STATE_STREAMING) {
					LOG_WRN("Not in streaming state");
					break;
				}

				ret = audio_system_encode_test_tone_step();
				if (ret) {
					LOG_WRN("Failed to play test tone, ret: %d", ret);
				}

				break;
			}

			break;

		default:
			LOG_WRN("Unexpected/unhandled button id: %d", msg.button_pin);
		}

		STACK_USAGE_PRINT("button_msg_thread", &button_msg_sub_thread_data);
	}
}

/**
 * @brief	Handle Bluetooth LE audio events.
 */
static void le_audio_msg_sub_thread(void)
{
	int ret;
	const struct zbus_channel *chan;

	while (1) {
		struct le_audio_msg msg;

		ret = zbus_sub_wait_msg(&le_audio_evt_sub, &chan, &msg, K_FOREVER);
		ERR_CHK(ret);

		LOG_DBG("Received event = %d, current state = %d", msg.event, strm_state);

		switch (msg.event) {
		case LE_AUDIO_EVT_STREAMING:
			LOG_DBG("LE audio evt streaming");

			audio_system_encoder_start();

			if (strm_state == STATE_STREAMING) {
				LOG_DBG("Got streaming event in streaming state");
				break;
			}

			audio_system_start();
			stream_state_set(STATE_STREAMING);
			ret = led_blink(LED_APP_1_BLUE);
			ERR_CHK(ret);

			break;

		case LE_AUDIO_EVT_NOT_STREAMING:
			LOG_DBG("LE audio evt not_streaming");

			audio_system_encoder_stop();

			if (strm_state == STATE_PAUSED) {
				LOG_DBG("Got not_streaming event in paused state");
				break;
			}

			stream_state_set(STATE_PAUSED);
			audio_system_stop();
			ret = led_on(LED_APP_1_BLUE);
			ERR_CHK(ret);

			break;

		default:
			LOG_WRN("Unexpected/unhandled le_audio event: %d", msg.event);

			break;
		}

		STACK_USAGE_PRINT("le_audio_msg_thread", &le_audio_msg_sub_thread_data);
	}
}

/**
 * @brief	Create zbus subscriber threads.
 *
 * @return	0 for success, error otherwise.
 */
static int zbus_subscribers_create(void)
{
	int ret;

	button_msg_sub_thread_id = k_thread_create(
		&button_msg_sub_thread_data, button_msg_sub_thread_stack,
		CONFIG_BUTTON_MSG_SUB_STACK_SIZE, (k_thread_entry_t)button_msg_sub_thread, NULL,
		NULL, NULL, K_PRIO_PREEMPT(CONFIG_BUTTON_MSG_SUB_THREAD_PRIO), 0, K_NO_WAIT);
	ret = k_thread_name_set(button_msg_sub_thread_id, "BUTTON_MSG_SUB");
	if (ret) {
		LOG_ERR("Failed to create button_msg thread");
		return ret;
	}

	le_audio_msg_sub_thread_id = k_thread_create(
		&le_audio_msg_sub_thread_data, le_audio_msg_sub_thread_stack,
		CONFIG_LE_AUDIO_MSG_SUB_STACK_SIZE, (k_thread_entry_t)le_audio_msg_sub_thread, NULL,
		NULL, NULL, K_PRIO_PREEMPT(CONFIG_LE_AUDIO_MSG_SUB_THREAD_PRIO), 0, K_NO_WAIT);
	ret = k_thread_name_set(le_audio_msg_sub_thread_id, "LE_AUDIO_MSG_SUB");
	if (ret) {
		LOG_ERR("Failed to create le_audio_msg thread");
		return ret;
	}

	ret = zbus_chan_add_obs(&sdu_ref_chan, &sdu_ref_msg_listen, ZBUS_ADD_OBS_TIMEOUT_MS);
	if (ret) {
		LOG_ERR("Failed to add timestamp listener");
		return ret;
	}

	return 0;
}

//Adding for NUS peri.
static struct k_work adv_work;
static void advertising_process(struct k_work *work)
{
	int ret;
#ifndef CONFIG_BT_LEGACY_ADV4ACL	//ADV-EXT
	ret = bt_le_ext_adv_start(adv_for_conn, BT_LE_EXT_ADV_START_DEFAULT);
	if (ret) {
		printk("Failed to set advertising data (err %d)\n", ret);
	}
#else	//legacy ADV.
	ret = bt_le_adv_start(BT_LE_ADV_CONN_CUSTOM, ad, ARRAY_SIZE(ad), sd, ARRAY_SIZE(sd));
	if(ret) {
		LOG_WRN("Failed to start advertiser, ret = %d", ret);
	}
#endif
}
//

/**
 * @brief	Zbus listener to receive events from bt_mgmt.
 *
 * @param[in]	chan	Zbus channel.
 *
 * @note	Will in most cases be called from BT_RX context,
 *		so there should not be too much processing done here.
 */
static void bt_mgmt_evt_handler(const struct zbus_channel *chan)
{
	int ret;
	const struct bt_mgmt_msg *msg;

	msg = zbus_chan_const_msg(chan);

	switch (msg->event) {
	//Adding for NUS peri.
	case BT_MGMT_CONNECTED:
		LOG_INF("BT_MGMT_CONNECTED");
		break;

	case BT_MGMT_DISCONNECTED:
		LOG_INF("BT_MGMT_DISCONNECTED");
#ifndef CONFIG_BT_LEGACY_ADV4ACL	//ADV-EXT
		k_work_submit(&adv_work);
#endif		
		break;

	case BT_MGMT_SECURITY_CHANGED:
		LOG_INF("BT_MGMT_SECURITY_CHANGED");
		break;
	//
	case BT_MGMT_EXT_ADV_WITH_PA_READY:
		LOG_INF("Ext adv ready");

		ret = broadcast_source_start(msg->ext_adv);
		if (ret) {
			LOG_ERR("Failed to start broadcaster: %d", ret);
		}

		break;

	default:
		LOG_WRN("Unexpected/unhandled bt_mgmt event: %d", msg->event);
		break;
	}
}

ZBUS_LISTENER_DEFINE(bt_mgmt_evt_listen, bt_mgmt_evt_handler);

/**
 * @brief	Link zbus producers and observers.
 *
 * @return	0 for success, error otherwise.
 */
static int zbus_link_producers_observers(void)
{
	int ret;

	if (!IS_ENABLED(CONFIG_ZBUS)) {
		return -ENOTSUP;
	}

	ret = zbus_chan_add_obs(&button_chan, &button_evt_sub, ZBUS_ADD_OBS_TIMEOUT_MS);
	if (ret) {
		LOG_ERR("Failed to add button sub");
		return ret;
	}

	ret = zbus_chan_add_obs(&le_audio_chan, &le_audio_evt_sub, ZBUS_ADD_OBS_TIMEOUT_MS);
	if (ret) {
		LOG_ERR("Failed to add le_audio sub");
		return ret;
	}

	ret = zbus_chan_add_obs(&bt_mgmt_chan, &bt_mgmt_evt_listen, ZBUS_ADD_OBS_TIMEOUT_MS);
	if (ret) {
		LOG_ERR("Failed to add bt_mgmt listener");
		return ret;
	}

	return 0;
}

uint8_t stream_state_get(void)
{
	return strm_state;
}

void streamctrl_send(void const *const data, size_t size, uint8_t num_ch)
{
	int ret;
	static int prev_ret;

	struct le_audio_encoded_audio enc_audio = {.data = data, .size = size, .num_ch = num_ch};

	if (strm_state == STATE_STREAMING) {
		ret = broadcast_source_send(enc_audio);

		if (ret != 0 && ret != prev_ret) {
			if (ret == -ECANCELED) {
				LOG_WRN("Sending operation cancelled");
			} else {
				LOG_WRN("Problem with sending LE audio data, ret: %d", ret);
			}
		}

		prev_ret = ret;
	}
}

int main(void)
{
	int ret;
	static const struct bt_data *ext_adv;
	static const struct bt_data *per_adv;

	LOG_DBG("nRF5340 APP core started");

	ret = nrf5340_audio_dk_init();
	ERR_CHK(ret);

	ret = nrf5340_audio_common_init();
	ERR_CHK(ret);

	size_t ext_adv_size = 0;
	size_t per_adv_size = 0;

	ret = zbus_subscribers_create();
	ERR_CHK_MSG(ret, "Failed to create zbus subscriber threads");

	ret = zbus_link_producers_observers();
	ERR_CHK_MSG(ret, "Failed to link zbus producers and observers");

	ret = broadcast_source_enable();
	ERR_CHK_MSG(ret, "Failed to enable broadcaster");

	ret = audio_system_config_set(
		bt_audio_codec_cfg_freq_to_freq_hz(CONFIG_BT_AUDIO_PREF_SAMPLE_RATE_VALUE),
		CONFIG_BT_AUDIO_BITRATE_BROADCAST_SRC, VALUE_NOT_SET);
	ERR_CHK_MSG(ret, "Failed to set sample- and bitrate");

	broadcast_source_adv_get(&ext_adv, &ext_adv_size, &per_adv, &per_adv_size);

	ret = bt_mgmt_adv_start(ext_adv, ext_adv_size, per_adv, per_adv_size, false);
	ERR_CHK_MSG(ret, "Failed to start advertiser");


	//Adding for NUS peri.
#ifndef CONFIG_BT_LEGACY_ADV4ACL	//ADV-EXT
	int err;

	err = bt_le_ext_adv_create(BT_LE_EXT_ADV_CONN, NULL, &adv_for_conn);
	if (err) {
		printk("Failed to create advertising set (err %d)\n", err);
	}

	err = bt_le_ext_adv_set_data(adv_for_conn, ad, ARRAY_SIZE(ad), NULL, 0);
	if (err) {
		printk("Failed to set advertising data (err %d)\n", err);
	}

	err = bt_le_ext_adv_start(adv_for_conn, BT_LE_EXT_ADV_START_DEFAULT);
	if (err) {
		printk("Failed to set advertising data (err %d)\n", err);
	}
#else	//legacy ADV.
	ret = bt_le_adv_start(BT_LE_ADV_CONN_CUSTOM, ad, ARRAY_SIZE(ad), sd, ARRAY_SIZE(sd));
	ERR_CHK_MSG(ret, "Failed to start advertiser");
#endif
	k_work_init(&adv_work, advertising_process);
	bt_nus_init(&nus_cb);
	//
	return 0;
}

[Dekel]

Brian Kim said:

In the meantime, we tried it out for legacy advertising for the ACL connection.

It's not fully optimized yet, but it looks good for both iPhone and Android. you can simply merge the attached one file((\broadcast_source\main.c) to your previous patch. Feel free to comment about your result.

That's great news, I really appreciate it! I've been on vacation this past week, hope to test it during the next one. I'll keep you updated.

Brian Kim said:

Are you satisfied with the current audio quality? Also, even slightly lower quality than this would be acceptable? If yes, that will bring more stable ACL link.

I am satisfied, and it absolutely can be lowered (more than 'slightly'), since audio quality may be set to minimum in our project.

Which parameters should I tweak to achieve this?

Thanks,

Dekel