Help with streaming BLE audio with device as source

RE: Help with streaming BLE audio with device as source

ovrebekk — Tue, 21 May 2024 12:45:27 GMT

Hi Ben

Sorry for the slow response. I don't have any immediate clever ideas, but have you tried to use the thread analyzer to figure out how much CPU time you are using, and which threads are using the most?

I guess most of the heavy number crunching happens within the same thread, so you won't be able to figure out exactly which operations are causing it, but at least it would give you an impression of how much CPU is used overall, and how much is used by your algorithms, compared to the other parts of the application.

Best regards
Torbjørn

RE: Help with streaming BLE audio with device as source

benwefers — Tue, 14 May 2024 16:10:53 GMT

Hi Torbjorn!

Thanks for the quick reply and again for all the info. I will do more testing and hopefully have some more details later today. I currently am running everything with default configuration, which from what I can tell is 192-byte blocks and 10 blocks per frame.

In my program, I am manually triggering the audio_usb.c data_received() callback every 1000us with 192 bytes. From what I can tell, the audio_system.c task is just waiting and collecting 10 of these blocks before encoding them and sending, so I think if I can drop the block number down to just 1 and then pass in all 1920 bytes at once, that would give me a lot more processing time where I don't have to worry about yielding the cpu to the audio_system.c task. Would that make sense?

Thanks

Ben

EDIT: I just tried this in a simplistic way by changing my app logic to the following:

    if (empty) {
        get_next_block(frame, FRAME_SIZE_BYTES);
        empty = false;
    }

    // Send data block
    if (timer < usTime()) {
        if (timer < usTime() - 2500) LOG_ERR("Missed deadline by %d us", usTime() - timer);
		for (int i = 0; i < CONFIG_FIFO_FRAME_SPLIT_NUM; i++)
        	audio_manual_send(&frame[BLOCK_SIZE_BYTES * i], BLOCK_SIZE_BYTES);
        timer += 10000;
        empty = true;
    }

Now instead of generating each block in 192-byte chunks, I generate it in 1920-byte chunks (the whole frame), and then queue up each subsection. This dramatically improved the audio quality (it is almost passable now)! 😃

The only issue is if I mix lots of tones (3 or more) it starts to get choppy again. I think I can improve performance further by increasing the frame size, and potentially bypassing the audio_system altogether by directly calling sw_codec_encode() and streamctrl_encoded_data_send(). Thoughts?

RE: Help with streaming BLE audio with device as source

ovrebekk — Tue, 14 May 2024 14:23:53 GMT

Hi Ben

[quote user="benwefers"]Zephyr is showing that I have 448 kb of ram, but I am wondering are there cache layers to this? Is there a performance penalty for accessing sections of this array compared to other variables?[/quote]

No, there is no cache for the RAM, it essentially runs at the same clock speed as the CPU, but there could be slow downs if multiple peripherals want to access the same RAM block at the same time. Still, I doubt this is the reason you experience slow downs here.

Normally you would use const data (flash) for large arrays like this, and then there will be a cache involved, but again I doubt this would slow you down significantly.

One way to make memory transfers more efficient is to treat your data as 32-bit values rather than 8-bit values, if possible, since the RAM bus can transfer 32-bits in a single cycle.

[quote user="benwefers"]I will perform some in depth speed testing to get to the bottom of which operations are hurting me the most.[/quote]

Please do. As I mentioned earlier simple pin toggling is often the most direct way to measure the speed of low level functions like these.

[quote user="benwefers"]EDIT: If accessing the array is in fact coming with a penalty, would it make sense to increase the packet size? [/quote]

Generally, yes, the larger you make your buffers the more efficient the processing will be, but it is a matter of diminishing returns as you make the buffers very large. Do you know what the size of your buffers are currently?

Best regards
Torbjørn

RE: Help with streaming BLE audio with device as source

benwefers — Mon, 13 May 2024 20:42:57 GMT

Hi Torbjorn!

Thanks for the info and sorry for the slow reply. I switched to 32-bit floats, but that seems to have had no noticeable impact. I ran through the entire datapath and verified there are no other instances of doubles.

Currently, my note table is stored like so in a file piano.h:

static unsigned char piano_wav[] = {
  0x52, 0x49, 0x46, 0x46, 0xa4, 0xb2, 0x00, 0x00, 0x57, 0x41, 0x56, 0x45,
  0x66, 0x6d, 0x74, 0x20, 0x10, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00,
  0x80, 0xbb, 0x00, 0x00, 0x00, 0x77, 0x01, 0x00, 0x02, 0x00, 0x10, 0x00,
  0x64, 0x61, 0x74, 0x61, 0x80, 0xb2, 0x00, 0x00, 0x0a, 0x00, 0x08, 0x00,
  0x01, 0x00, 0xfe, 0xff, 0xfb, 0xff, 0xff, 0xff, 0x01, 0x00, 0x0b, 0x00,
  0x09, 0x00, 0x0d, 0x00, 0x09, 0x00, 0x07, 0x00, 0x00, 0x00, 0x02, 0x00,
  0xfc, 0xff, 0xfe, 0xff, 0xfb, 0xff, 0xf9, 0xff, 0xfb, 0xff, 0xfc, 0xff,
  0x00, 0x00, 0x05, 0x00, 0x06, 0x00, 0x0a, 0x00, 0x08, 0x00, 0x0f, 0x00,
  0x09, 0x00, 0x13, 0x00, 0x04, 0x00, 0x0f, 0x00, 0xfe, 0xff, 0x03, 0x00,
  0xf6, 0xff, 0xf5, 0xff, 0xe9, 0xff, 0xe5, 0xff, 0xd8, 0xff, ...

I then include this in my main app.h and index the array as needed. The array is very long (roughly 45kb). Zephyr is showing that I have 448 kb of ram, but I am wondering are there cache layers to this? Is there a performance penalty for accessing sections of this array compared to other variables?

I will perform some in depth speed testing to get to the bottom of which operations are hurting me the most.

Thanks for your continued support!

EDIT: If accessing the array is in fact coming with a penalty, would it make sense to increase the packet size? I'm perfectly fine trading off less responsive audio if it means smoother playback. So perhaps instead of these really small chunks that need to be sent frequently, can we instead spread them out a bit giving my mixing logic more time to execute before the sending logic needs the cpu again? Less context switches generally means faster...

From what I can tell, frame and block size seem to be configurable, though I'm not sure how, nor how big I can make them before the bluetooth driver will start to get angry. Can you give me some tips for adjusting these? For my purposes, even a response time of around 50ms is fine.

RE: Help with streaming BLE audio with device as source

ovrebekk — Fri, 03 May 2024 12:30:19 GMT

Have you measured the runtime of the various steps in the mix_tone(..) function, to see if one or more of them could be running slower than you expect?

double math should generally be avoided, unless absolutely necessary. The Cortex M33 is a 32-bit architecture with no hardware dedicated to 64-bit floating point (double) processing, and double calculations will run very slow and require a lot of CPU number crunching. 32-bit floats on the other hand can take advantage of hardware acceleration, in addition to being less compute and memory intensive to start with.

For the note table I would recommend just storing a static table in RAM, and fill it with values at the start of your program.

Do you know if either of the tone_gen(..), contin_array_create(..) or pcm_mix(..) functions use double?

Best regards
Torbjørn

RE: Help with streaming BLE audio with device as source

benwefers — Thu, 02 May 2024 16:22:53 GMT

Hi Torbjorn!

No worries, and I really appreciate your help. I was able to confirm that I am indeed underrunning the buffer (my audio_manual_send is not being triggered frequently enough). I confirmed this by adding the following to my code.

    // Send data block
    if (timer < usTime()) {
        if (timer < usTime() - 1000) LOG_ERR("Missed deadline by %d us", usTime() - timer);
        audio_manual_send(frame, 192);
        timer += 1000;
        empty = true;
        // LOG_INF("Sending...");
    }

This produced these log messages:

--- 11 messages dropped ---
GW [00:00:13.796,508] <wrn> audio_system: Failed to get last filled block
--- 11 messages dropped ---
GW [00:00:13.816,650] <err> app: Missed deadline by 3080 us
--- 11 messages dropped ---
GW [00:00:13.836,944] <err> app: Missed deadline by 2375 us
--- 10 messages dropped ---
GW [00:00:13.857,025] <err> app: Missed deadline by 2425 us
--- 11 messages dropped ---
GW [00:00:13.877,349] <err> app: Missed deadline by 1780 us
--- 11 messages dropped ---
GW [00:00:13.897,369] <err> app: Missed deadline by 1799 us
--- 11 messages dropped ---
GW [00:00:13.917,297] <err> app: Missed deadline by 1727 us
--- 10 messages dropped ---
GW [00:00:13.936,798] <err> app: Missed deadline by 3228 us
--- 11 messages dropped ---
GW [00:00:13.956,573] <wrn> audio_system: Failed to get last filled block
--- 10 messages dropped ---
GW [00:00:13.986,663] <wrn> audio_system: Failed to get last filled block
--- 10 messages dropped ---
GW [00:00:13.997,711] <err> app: Missed deadline by 1141 us
--- 11 messages dropped ---
GW [00:00:14.017,364] <err> app: Missed deadline by 1795 us
--- 11 messages dropped ---
GW [00:00:14.037,445] <err> app: Missed deadline by 1845 us
--- 12 messages dropped ---
GW [00:00:14.057,006] <err> app: Missed deadline by 2406 us
--- 10 messages dropped ---
GW [00:00:14.077,636] <err> app: Missed deadline by 1067 us

So it seems my my app thread is not getting ticked nearly as frequently as it needs to be. Just as a quick check, I bumped the thread priority to 1 and that unsurprisingly made my "Missed deadline" messages go away, but the audio was much worse (presumably the audio_system thread was no longer getting ticked frequently enough).

Here is a snippet of some of my audio mixing code.

// Mixes a test tone into the buffer
void mix_note(char *buffer, size_t size, int note, int index, bool pressed) {
    if (!pressed) {
        test_tone_finite_pos[index] = 0;
        return;
    }
	int freq = 440 * pow(2, (double) (note + octave * 12) / 12);
	tone_gen(test_tone_buf, &test_tone_size, freq, CONFIG_AUDIO_SAMPLE_RATE_HZ, volume);
	int ret = contin_array_create(test_tone, size, test_tone_buf, test_tone_size, &test_tone_finite_pos[index]);
	ERR_CHK_MSG(ret, "Failed to create continous array");

	ret = pcm_mix(buffer, size, test_tone, size, B_MONO_INTO_A_MONO);
	ERR_CHK_MSG(ret, "Failed to mix tones");
}

This mixes a simple test tone at a specified note into the provided buffer. What's interesting, is that when pressed is true (i.e. the tone is actually being mixed), the amount by which I miss the deadline steadily. Then when I release (i.e. stop mixing the tone) and the amount I miss the deadline by is sufficiently high, the program immediately crashes. See logs below.

GW [00:00:23.362,640] <err> app: Missed deadline by 910040 us
--- 68 messages dropped ---
GW [00:00:23.426,239] <err> app: Missed deadline by 918638 us
--- 64 messages dropped ---
GW [00:00:23.493,591] <err> app: Missed deadline by 920991 us
--- 70 messages dropped ---
GW [00:00:23.556,488] <err> app: Missed deadline by 928888 us
--- 66 messages dropped ---
GW [00:00:23.624,786] <err> app: Missed deadline by 932186 us
--- 68 messages dropped ---
GW [00:00:23.687,988] <err> app: Missed deadline by 938388 us
--- 67 messages dropped ---
GW [00:00:23.754,150] <err> app: Missed deadline by 941580 us
--- 66 messages dropped ---
GW [00:00:23.820,892] <err> app: Missed deadline by 947292 us
--- 69 messages dropped ---
GW [00:00:23.884,857] <err> app: Missed deadline by 955257 us
--- 65 messages dropped ---
GW [00:00:23.950,744] <err> app: Missed deadline by 959144 us
--- 69 messages dropped ---
GW [00:00:24.014,739] <err> app: Missed deadline by 966139 us
--- 69 messages dropped ---
GW [00:00:24.082,977] <err> app: Missed deadline by 970377 us
--- 66 messages dropped ---
GW [00:00:24.146,087] <err> app: Missed deadline by 975487 us
--- 69 messages dropped ---
GW [00:00:24.211,822] <err> app: Missed deadline by 980222 us
--- 67 messages dropped ---
GW [00:00:24.276,947] <err> app: Missed deadline by 986347 us
--- 69 messages dropped ---
GW [00:00:24.341,186] <err> app: Missed deadline by 991586 us
--- 67 messages dropped ---
GW [00:00:24.407,958] <err> app: Missed deadline by 995358 us
--- 68 messages dropped ---
GW [00:00:24.471,435] <err> app: Missed deadline by 1003835 us
--- 68 messages dropped ---
GW [00:00:24.540,557] <err> app: Missed deadline by 1006957 us
--- 68 messages dropped ---
GW [00:00:24.604,980] <err> app: Missed deadline by 1012410 us
--- 72 messages dropped ---
GW [00:00:24.670,898] <err> app: Missed deadline by 1017298 us
--- 66 messages dropped ---
GW [00:00:24.743,408] <wrn> audio_system: Failed to get last filled block
--- 80 messages dropped ---
GW [00:00:24.809,234] <err> app: Missed deadline by 1030634 us
--- 65 messages dropped ---
GW [00:00:24.876,434] <err> app: Missed deadline by 1033834 us
--- 69 messages dropped ---
GW [00:00:24.940,673] <err> app: Missed deadline by 1041073 us
--- 66 messages dropped ---
GW [00:00:25.007,293] <err> app: Missed deadline by 1042693 us
--- 69 messages dropped ---
GW [00:00:25.071,411] <err> app: Missed deadline by 1049811 us
--- 66 messages dropped ---
GW [00:00:25.138,153] <err> app: Missed deadline by 1055553 us
--- 67 messages dropped ---
GW [00:00:25.205,505] <err> app: Missed deadline by 1060905 us
--- 69 messages dropped ---
GW [00:00:25.268,402] <err> app: Missed deadline by 1067802 us
--- 65 messages dropped ---
GW [00:00:25.335,662] <err> app: Missed deadline by 1071093 us
--- 68 messages dropped ---
GW [00:00:25.401,367] <err> app: Missed deadline by 1077767 us
--- 70 messages dropped ---
GW [00:00:25.472,808] <err> app: Missed deadline by 1086208 us
--- 68 messages dropped ---
GW [00:00:25.538,452] <err> app: Missed deadline by 1091852 us
--- 68 messages dropped ---
GW [00:00:25.604,248] <err> app: Missed deadline by 1097648 us
--- 67 messages dropped ---
GW [00:00:25.671,234] <err> app: Missed deadline by 1100634 us
--- 74 messages dropped ---
GW [00:00:25.735,107] <err> app: Missed deadline by 1106507 us
--- 64 messages dropped ---
GW [00:00:25.802,001] <err> app: Missed deadline by 1111401 us
--- 68 messages dropped ---
GW [00:00:25.868,225] <err> app: Missed deadline by 1117625 us
--- 69 messages dropped ---
GW [00:00:25.933,197] <err> app: Missed deadline by 1121597 us
--- 67 messages dropped ---
GW [00:00:25.999,694] <err> app: Missed deadline by 1127094 us
--- 69 messages dropped ---
GW [00:00:26.063,781] <err> app: Missed deadline by 1135181 us
--- 65 messages dropped ---
GW [00:00:26.130,462] <err> app: Missed deadline by 1136862 us
--- 70 messages dropped ---
GW [00:00:26.194,732] <err> app: Missed deadline by 1144132 us
--- 67 messages dropped ---
GW [00:00:26.269,012] <wrn> audio_system: Failed to get last filled block
--- 73 messages dropped ---
GW [00:00:26.336,853] <err> app: Missed deadline by 1155253 us
--- 69 messages dropped ---
GW [00:00:26.401,397] <err> app: Missed deadline by 1160797 us
--- 68 messages dropped ---
GW [00:00:26.468,811] <err> app: Missed deadline by 1165211 us
--- 68 messages dropped ---
GW [00:00:26.533,020] <err> app: Missed deadline by 1173420 us
--- 67 messages dropped ---
GW [00:00:26.599,853] <err> app: Missed deadline by 1175253 us
--- 68 messages dropped ---
GW [00:00:26.664,642] <err> app: Missed deadline by 1182042 us
--- 70 messages dropped ---
GW [00:00:26.731,079] <err> app: Missed deadline by 1188479 us
--- 64 messages dropped ---
GW [00:00:26.795,959] <err> app: Missed deadline by 1191359 us
--- 69 messages dropped ---
GW [00:00:26.861,145] <err> app: Missed deadline by 1196545 us
--- 66 messages dropped ---
GW [00:00:26.933,654] <wrn> audio_datapath: Not able to drop FIFO RX block
--- 78 messages dropped ---
GW [00:00:27.000,030] <err> app: Missed deadline by 1211430 us
--- 67 messages dropped ---
GW [00:00:27.068,115] <err> app: Missed deadline by 1213545 us
--- 68 messages dropped ---
GW [00:00:27.134,307] <err> app: Missed deadline by 1219707 us
--- 71 messages dropped ---
GW [00:00:27.200,073] <err> app: Missed deadline by 1227503 us
--- 65 messages dropped ---
GW [00:00:27.272,155] <wrn> audio_system: Failed to get last filled block
--- 73 messages dropped ---
GW [00:00:27.340,057] <err> app: Missed deadline by 1239457 us
--- 69 messages dropped ---
GW [00:00:27.406,677] <err> app: Missed deadline by 1246077 us
--- 65 messages dropped ---
GW [00:00:27.471,984] <err> app: Missed deadline by 1249384 us
--- 71 messages dropped ---
GW [00:00:27.536,926] <err> app: Missed deadline by 1255326 us
--- 66 messages dropped ---
GW [00:00:27.603,454] <err> app: Missed deadline by 1259854 us
--- 72 messages dropped ---
GW [00:00:27.667,480] <err> app: Missed deadline by 1264880 us
--- 64 messages dropped ---
GW [00:00:27.739,593] <wrn> audio_system: Failed to get last filled block
--- 73 messages dropped ---
GW [00:00:27.808,044] <err> app: Missed deadline by 1278444 us
--- 70 messages dropped ---
GW [00:00:27.873,901] <err> app: Missed deadline by 1284301 us
--- 67 messages dropped ---
GW [00:00:27.940,826] <err> app: Missed deadline by 1288226 us
--- 70 messages dropped ---
GW [00:00:28.004,302] <err> app: Missed deadline by 1295702 us
--- 68 messages dropped ---
GW [00:00:28.073,760] <err> app: Missed deadline by 1299160 us
--- 67 messages dropped ---
GW [00:00:28.136,138] <err> app: Missed deadline by 1305538 us
--- 68 messages dropped ---
GW [00:00:28.203,216] <err> app: Missed deadline by 1309616 us
--- 67 messages dropped ---
GW [00:00:28.270,111] <err> app: Missed deadline by 1315511 us
--- 71 messages dropped ---
GW [00:00:28.333,251] <err> app: Missed deadline by 1321651 us
--- 66 messages dropped ---
GW [00:00:28.406,097] <err> app: Missed deadline by 1330497 us
--- 70 messages dropped ---
GW [00:00:28.465,576] <err> app: Missed deadline by 1330976 us
--- 69 messages dropped ---
GW [00:00:28.538,269] <wrn> audio_datapath: Not able to drop FIFO RX block
--- 71 messages dropped ---
GW [00:00:28.607,482] <err> app: Missed deadline by 1343882 us
--- 69 messages dropped ---
GW [00:00:28.671,112] <err> app: Missed deadline by 1351512 us
--- 68 messages dropped ---
GW [00:00:28.739,440] <err> app: Missed deadline by 1355840 us
--- 68 messages dropped ---
GW [00:00:28.803,131] <err> app: Missed deadline by 1361531 us
--- 69 messages dropped ---
GW [00:00:28.876,464] <wrn> audio_system: Failed to get last filled block
--- 74 messages dropped ---
GW [00:00:28.945,800] <err> app: Missed deadline by 1374200 us
--- 69 messages dropped ---
GW [00:00:29.008,605] <err> app: Missed deadline by 1381005 us
--- 65 messages dropped ---
GW [00:00:29.075,927] <err> app: Missed deadline by 1383327 us
--- 69 messages dropped ---
GW [00:00:29.141,754] <err> app: Missed deadline by 1390154 us
--- 70 messages dropped ---
GW [00:00:29.208,953] <err> app: Missed deadline by 1394353 us
--- 69 messages dropped ---
GW [00:00:29.273,590] <err> app: Missed deadline by 1399990 us
--- 70 messages dropped ---
GW [00:00:29.338,165] <err> app: Missed deadline by 1405565 us
--- 65 messages dropped ---
GW [00:00:29.411,254] <wrn> audio_system: Failed to get last filled block
--- 76 messages dropped ---
GW [00:00:29.477,050] <err> app: Missed deadline by 1420450 us
--- 68 messages dropped ---
GW [00:00:29.545,440] <err> app: Missed deadline by 1421840 us
--- 68 messages dropped ---
GW [00:00:29.611,236] <err> app: Missed deadline by 1428636 us
--- 70 messages dropped ---
GW [00:00:29.676,422] <err> app: Missed deadline by 1432822 us
--- 67 messages dropped ---
GW [00:00:29.741,455] <err> app: Missed deadline by 1437855 us
--- 68 messages dropped ---
GW [00:00:29.805,908] <err> app: Missed deadline by 1446308 us
--- 68 messages dropped ---
GW [00:00:29.873,626] <err> app: Missed deadline by 1449026 us
--- 69 messages dropped ---
GW [00:00:29.938,476] <err> app: Missed deadline by 1454876 us
--- 69 messages dropped ---
GW [00:00:30.010,681] <wrn> audio_system: Failed to get last filled block
--- 74 messages dropped ---
GW [00:00:30.080,688] <err> app: Missed deadline by 1467088 us
--- 70 messages dropped ---
GW [00:00:30.144,226] <err> app: Missed deadline by 1472626 us
--- 66 messages dropped ---
GW [00:00:30.209,655] <err> app: Missed deadline by 1477055 us
--- 70 messages dropped ---
GW [00:00:30.262,969] <err> app: Missed deadline by 1470400 us
--- 67 messages dropped ---
GW [00:00:30.304,595] <err> app: Missed deadline by 1452026 us
--- 67 messages dropped ---
GW [00:00:30.345,886] <err> app: Missed deadline by 1433316 us
--- 66 messages dropped ---
GW [00:00:30.392,303] <err> app: Missed deadline by 1414733 us
--- 85 messages dropped ---
GW [00:00:30.415,405] <wrn> audio_usb: USB RX overrun
--- 79 messages dropped ---
GW [00:00:30.442,291] <err> app: Missed deadline by 1351721 us
--- 88 messages dropped ---
GW [00:00:30.465,698] <wrn> audio_usb: USB RX overrun
--- 78 messages dropped ---
GW [00:00:30.493,011] <err> app: Missed deadline by 1288441 us
--- 92 messages dropped ---
GW [00:00:30.516,174] <wrn> audio_usb: USB RX overrun
--- 80 messages dropped ---
GW [00:00:30.542,449] <err> app: Missed deadline by 1227880 us
--- 87 messages dropped ---GW [00:00:30.574,371] <err> app: Missed deadline by 1185801 us
--- 46 messages dropped ---
GW [00:00:30.574,401] <wrn> audio_usb: USB RX overrun
GW [00:00:30.574,645] <err> app: Missed deadline by 1185076 us
GW [00:00:30.574,676] <wrn> audio_usb: USB RX overrun
GW [00:00:30.574,920] <err> app: Missed deadline by 1184351 us
GW [00:00:30.574,951] <wrn> audio_usb: USB RX overrun
GW [00:00:30.575,195] <err> app: Missed deadline by 1183625 us
GW [00:00:30.575,225] <wrn> audio_usb: USB RX overrun
GW [00:00:30.575,469] <err> app: Missed deadline by 1182900 us
GW [00:00:30.575,500] <wrn> audio_usb: USB RX overrun
GW [00:00:30.575,744] <err> app: Missed deadline by 1182175 us
GW [00:00:30.575,775] <wrn> audio_usb: USB RX overrun
GW [00:00:30.576,019] <err> app: Missed deadline by 1181449 us
GW [00:00:30.576,049] <wrn> audio_usb: USB RX overrun
GW [00:00:30.576,354] <err> app: Missed deadline by 1180785 us
GW [00:00:30.576,660] <wrn> audio_usb: USB RX overrun
GW [00:00:30.576,690] <err> audio_usb: ERR_CHK Err_code: [-35] @ line: 99
GW [00:00:30.576,690] <err> os: r0/a1:  0x00000003  r1/a2:  0x00000022  r2/a3:  0x00000021
GW [00:00:30.576,690] <err> os: r3/a4:  0x00066c37 r12/ip:  0x00000010 r14/lr:  0x00066cc1
GW [00:00:30.576,721] <err> os:  xpsr:  0x61100000

It seems to be failing here in audio_usb.

ret = data_fifo_pointer_last_filled_get(fifo_rx, &temp, &temp_size, K_NO_WAIT);
ERR_CHK(ret);

But I am not sure I understand why. Perhaps you have some insight.

Anyway, I think these issues are primarily side effects of me underrunning the buffer, so I'm gonna do a few more tests to see if I can come up with a better strategy around that.

Thanks for your help and let me know if there are any other details I can provide that would be useful.

EDIT: I guess I would also like to know what might be using so much cpu power that I am not able to run my thread fast enough? Apart from my app thread, the main thread, and the audio_system thread, what else is running? And can you think of any that I can disable to help speed things up?

RE: Help with streaming BLE audio with device as source

ovrebekk — Thu, 02 May 2024 13:33:08 GMT

Hi Ben

Sorry for the slow response, it has been some hectic weeks. I will set aside some time tomorrow to look into this, but if you have made anymore discoveries in the mean time please let me know 🙂

Best regards
Torbjørn

RE: Help with streaming BLE audio with device as source

benwefers — Wed, 24 Apr 2024 23:39:35 GMT

Hi Torbjørn!

Thanks for the info. I went ahead and switch over to using USB as the source and you are right, that makes things a lot simpler. It seems that in this setup (where there is no actual USB source) the audio_system thread waits indefintiely for PCM chunks to arrive, which would normally be done though the data_received() callback in audio_usb.c. I added a little wrapper shown below to trigger this with my own data.

void audio_manual_send(uint8_t* buf, size_t size) {
	struct net_buf *buffer = net_buf_alloc_len(&pool_out, size, K_NO_WAIT);
	net_buf_add_mem(buffer, buf, size);
	data_received(NULL, buffer, size);
}

Because I am on my own now for timing my packets, I added the following to my app thread.

// Get current time in us
uint32_t usTime() {
    return k_cyc_to_us_near32(k_cycle_get_32());
}



... and in my loop ...

// Generate next data block
if (empty) {
    get_next_block(frame, 192);
    empty = false;
}

// Send data block
if (timer < usTime()) {
    audio_manual_send(frame, 192);
    timer += 500;
    empty = true;
    // LOG_INF("Sending...");
}

This setup therefore doesn't rely on the precise timing of the zephyr kernel, but so long as my loop is run at least once in the time period my timer will be seen and the data should be added to the queue. This has proved successful, and I am now getting smooth audio... for about 1 second.

After the first second of audio, it starts to get choppy again, and I ran some tests and it does not seem to be a result of me underrunning the queue. I am already running the loop much faster than I should be (an issue I hope you can help me with). I have it configured for 48kHz 16-bit audio with all the default sizing (192-byte blocks, 10 blocks per frame). At this rate, I would expect to need to shove a 192-byte block using my audio_manual_send() function every 2 ms. Based on my tested results this does not seem to be the case, I only get smooth audio at 500us. Maybe my math is wrong.

My other obvious issue is I have constant warning messages like the following:

GW [00:01:25.455,200] <wrn> audio_system: Failed to get last filled block
GW [00:01:25.455,200] <wrn> audio_datapath: Not able to drop FIFO RX block
GW [00:01:25.465,332] <wrn> audio_system: Failed to get last filled block
GW [00:01:25.465,332] <wrn> audio_datapath: Not able to drop FIFO RX block
GW [00:01:25.475,097] <wrn> audio_system: Failed to get last filled block
GW [00:01:25.475,097] <wrn> audio_datapath: Not able to drop FIFO RX block
GW [00:01:25.485,198] <wrn> audio_system: Failed to get last filled block
GW [00:01:25.485,198] <wrn> audio_datapath: Not able to drop FIFO RX block
GW [00:01:25.495,178] <wrn> audio_system: Failed to get last filled block
GW [00:01:25.495,208] <wrn> audio_datapath: Not able to drop FIFO RX block

These are just constantly coming out. Any thoughts?

Thanks again for the help!

UPDATE: I think it is also worth noting though that the test tone (added using CONFIG_AUDIO_TEST_TONE=y) seems to be perfectly smooth. My understanding is that the audio_system is mixing this in to the compiled frame before encoding the data.

To give bit more context, I have a number of buttons that each produce different sounds, which I am mixing together using the zephyr pcm_mix() function. This is what is being created in the call to get_next_block() call in the snippet above.

The test tone is smooth as long as I am not actually mixing any audio (i.e. get_next_block() is filling the buffer with all 0's since no buttons are pressed). Once I press any of the buttons (while the test tone is also still playing), it is a smooth tone for about a second and then it starts to get choppy (test tone included). This to me contradicts my previous assessment that I am not underrunning the buffer, and suggests that the issue is in fact that I am not filling the buffer fast enough. As far as I can tell though my app thread is getting triggered frequently enough. Also, my app mixing (other than the call to pcm_mix) is a single loop through the block size (192 bytes) with some simple arithmetic, which I imagine should not be so cpu intensive that it is causing such issues, but perhaps I am mistaken.

RE: Help with streaming BLE audio with device as source

RM — Tue, 23 Apr 2024 20:52:47 GMT

Hi Ben,

In general, I2S (PCM as well) requires very precise clock, especially if using an external codec to produce I2S signals. To process incoming streams (either PCM or I2S) in the processor, a very quick and precise processing is required, as loosing frames may result in choppy or noisy audio. You may need to use PPI and potentially bypass Zephyr OS scheduler to process these signals properly.

Usually to properly process audio (with some filters) you would need a dedicated core, unless you just pass it through.

Thanks,
Reuven

RE: Help with streaming BLE audio with device as source

ovrebekk — Tue, 23 Apr 2024 08:20:00 GMT

Hi Ben

I think you might have an easier time if set up the example for USB rather than I2S, using the audio_usb.c module as shown here.

In many ways the USB interface is easier to manage than the I2S one, and it is possible that the choppy audio you experience is caused by the synchronization of the I2S interface.

Could you give this a go and see if you get better results?

If you are still having issues please let me know what you tried, and I will take a look at it.

Best regards
Torbjørn