Best place for audio processing adjustments.

[Jorritv3]

Hello all,

I am working on a project with 2 nrf5340 Audio DK's, and I have been working for a while now on getting myself familiar with the audio application, but I have a few questions. First of all I assumed it would be wise to develop on top of the provided audio application since writing the entire pairing and data transmission yourself would be near impossible, is this a correct assumption?

Secondly I have the walkie talkie demo running, and am now trying to encode and decode the data that I am sending over the line, for now I just want a simple xor operation to encrypt the data and decrypt it. Right now I am doing it before the codec encode in the audio_system.c and decrypting it after the decode, but I just get a horrible noise instead of the orginal audio. I tried a few other spots like inside the callback, but I saw another thread saying this was ill advised.

So my question is what would be the correct places to place the encrypt and decrypt of the data? 

[Einar Thorsrud]

Hi,

The codec is a lossy codec, so you cannod encrypt (with simple xoring or any other method) before coding. Any such operations will have to happen after coding, and then the reverse operation must be performed before decoding. (If you were doing simple one way audio processing where the output signal is also an audio signal, and there is no reverse operation in the other end, that could have been doen before encoding).

[Jorritv3]

That makes a lot of sense, thank you.

So I implemented it the way you are saying, and it seems to work. However if I only implement the encoding side without the decoding to check if the audio gets scrambled it doesn't output anything anymore. This feels like a checksum of some sort is being implemented somewhere.

Do you have an idea of why this doesn't audio anymore? I would like to check wether the audio gets encrypted correctly, so it would be nice if it still reaches the speaker of my headset.

Thank you.

P.S.

I placed a very simple

const uint16_t xor_key = 0xAA;

for (size_t i = 0; i < audio_frame->len; i++) {
	audio_frame->data[i] ^= xor_key;
}

before the sw_codec_decode in audio.datapath.c and the same after the sw_codec_encode in audio_system.c.

[Einar Thorsrud]

Hi,

I would have expected that to work. Have you done any debugging to see what happens to the data and why there is no audio output? (Start with seeing if there are errors in the log, then move deeper from there)?

PS: I assume this is just for experimentation, but to be on the safe side, I want to mention that if the end goal is encrypting the data, you should proably use standard Bluetooth security.

[Jorritv3]

Hi,

First of all thank you for your help.

So I tried to follow the audio path because there were no warnings or errors. I found the 

sw_codec_lc3_dec_run function, which I assumed makes the PCM signal, to return only zeroes as PCM signal. I can not figure out why, but it seems to happen at LC3DecodeSessionData (I might be wrong). Also it is not noted anywhere as a bad frame so it seems like the codec is decoding it but just writes all zeroes.

The only other maybe notable thing is that I set CONFIG_SW_CODEC_LC3=y in my conf.

Any help is appreciated, thank you in advance. 

[Einar Thorsrud]

Hi,

I am not sure I understand. Can you elaborate, and perhaps show which changes you made in the codet?

[Jorritv3]

Hi,

Thank you for taking the time.

So first I looked at what was coming in on the decode side, but there the audio frame is all zeroes. 

next I looked if the audio even arrived correctly at the other device, so after bleutooth transmission, which it does. In the file audio_datapath.c I see the audioframe having values before the sw_codec_decode, but the decoded data after the decode function is all zeroes, so somewhere in this function it goes wrong.

In this function I saw that it depends on CONFIG_SW_CODEC_LC3 so I tried turning this on and placing some more print statements to see where the audio data is lost.

In the file sw_codec_select.c there is this piece of code:

case SW_CODEC_MONO: {
			if (meta->bad_data && SW_CODEC_OVERRIDE_PLC) {
				memset(decoded_data_mono[0], 0, PCM_NUM_BYTES_MONO);
				decoded_data_size = PCM_NUM_BYTES_MONO;
			} else {

				ret = sw_codec_lc3_dec_run(
					audio_frame->data, audio_frame->len, LC3_PCM_NUM_BYTES_MONO,
					0, decoded_data_mono[0], (uint16_t *)&decoded_data_size,
					meta->bad_data);

				if (ret) {

					return ret;
				}

And I see that the data in the audio frame before this function has values, but the decoded data after this function is all zeroes. So I followed the sw_codec_lc3_dec_run function to see where it goes wrong there. 

And finally I end up at this line in sw_codec_lc3.c:

ret = LC3DecodeSessionData(dec_handle_ch[audio_ch], &LC3DecodeInput, &LC3DecodeOutput);

The audio is fine before this function, but is all zeroes after this function. I placed this logging:

if (frame_counter % 200 == 0)
	{
		LOG_INF("---- LC3 Decode #%d ----", frame_counter);
		LOG_INF("Ch:%d, FrameLen:%d, PCMbuf:%d, bad_frame:%d",
				audio_ch, lc3_data_size, pcm_data_buf_size, bad_frame);
		LOG_HEXDUMP_INF(lc3_data, MIN(lc3_data_size, 16), "LC3 Input (first 16 bytes)");
		LOG_INF("Decoded bytesWritten:%d, PLCCounter:%d",
				LC3DecodeOutput.bytesWritten, LC3DecodeOutput.PLCCounter);

		if (LC3DecodeOutput.bytesWritten > 0)
		{
			uint16_t *pcm16 = (uint16_t *)pcm_data;
			size_t samples_to_print = MIN(8, LC3DecodeOutput.bytesWritten / sizeof(uint16_t));
			printk("PCM samples (first %zu): ", samples_to_print);
			for (size_t i = 0; i < samples_to_print; i++)
			{
				printk("%d ", pcm16[i]);
			}
			printk("\n");
		}

		LOG_INF("---------------------------");
	}

Which returns 0 bad_frames correct LC3 Input and no errors, but the PCM Samples printed are all zeroes.

I don't know what this function does but is there a way to make it work, because I don't understand why it doesn't output anything when I encrypt my data even though I expect random noise, (which I want to hear)

I hope this clears up my issue, please let me know if you need more information.

Kind regards.

[Einar Thorsrud]

Hi,

I am sorry for the late reply. Is it correct that you pass data to LC3DecodeOutput(), meaning non-empty input frame, but the output frame is all zero non the less? You write that there are no errors, does that include from LC3DecodeOutput? What is the return value? If it returns 0 that means no error, and the output buffer should be a valid decoded frame containing raw PCM data.

Also, is this on unmodified data, or do you see ths issue only when you have manipulatedc the input data (before it was encoded I assume)?

[Jorritv3]

Hi,

So I pass valid data to LC3DecodeSessionData(), not LC3DecodeOutput(), and it returns 0, so it should be valid. This is modified data in the sense that I applied an XOR operation to the bytes after encoding. And indeed they are not empty, but still the output frames are zero.

This problem only happens when I only encrypt the data (so after encoding). What I would expect is to hear noise (meaning in this case encrypted audio), but when I also apply the decryption it sounds normal again. I want to hear the noise to prove the audio is correctly being processed, but instead the audio gets set to 0 at the LC3DecodeSessionData().

[Einar Thorsrud]

Can you Is this data now encrypted before or after encoding? If you encrypt data before encoding I am not sure how the result will be, as the encoder is lossy and expects an audio signal. The encrypted signal is effectively white noise and impossible to compress in any measningfull way. It could perhaps also be that as there is no data erroc concelment will result in zero output (no noise)? If so, te return value will be 0 (LC3_RESULT_NO_ERROR) according to the API documentation for LC3DecodeSessionData.