nRFX based PDM to PCM implementation (I2S example)

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Tue, 15 Jun 2021 10:46:34 GMT

You could try making the printouts more efficient. For example:

		static char str[4000 * 4 + 1];
		size_t len = 0;

		for(int i=0; i<4000;i++)
		{
			int err = snprintf(&str[len], sizeof(str) - len, "%04X", (unsigned short) audio_block->buffer[i]);
			if (err < 0 || err > 4)
			{
				printk("Format error: %d\n", err);
				return;
			}
			len += err;
		}
		printk("%s\n", str);

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Mon, 14 Jun 2021 09:42:57 GMT

Hello,

I see a couple of problems here, some from my code reference in earlier reply:

1) You don't need both "K_MEM_SLAB_DEFINE" and "struct k_mem_slab audio_slab;". Same for FIFO

2) The mem slab needs to be at least 3 blocks long. The PDM driver will request 2 blocks right away, and request a third before you will have had time to process the first released one:

//Memory slab
K_MEM_SLAB_DEFINE(audio_slab, sizeof(struct data_item_t), 3, 4);

//FIFO to hold pointers to memory slab
K_FIFO_DEFINE(my_fifo);

3) No double-pointer to k_fifo_put. It should be like this:

k_fifo_put(&my_fifo, filled_block);

4) k_sleep() must be removed from Microphone_Start_Record(). Otherwise slab will be emptied while the main thread is sleeping. If you want to skip the first X seconds of audio, you need to free the first Y blocks in main() without printing them out first (k_mem_slab_free)

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Wed, 09 Jun 2021 12:35:24 GMT

Hello,

I suggest changing the data_item_t struct slightly to make it easier to get the block size right. Currently you have to manually make sure that the slab definition has block size that matches sizeof struct + *buffer:

struct data_item_t {
    void *fifo_reserved;   /* 1st word reserved for use by FIFO */
    int16_t buffer[16000]; /* 1 second of mono audio @ 16 kHz */
};

//Memory slab
struct k_mem_slab audio_slab;
K_MEM_SLAB_DEFINE(audio_slab, sizeof(struct data_item_t), 2, 4);

//FIFO to hold pointers to memory slab
struct k_fifo my_fifo;
K_FIFO_DEFINE(my_fifo);

Now you can request a buffer in the pdm handler, and put filled buffers into the fifo:

void nrfx_pdm_event_handler(nrfx_pdm_evt_t const * p_evt)
{
	if(p_evt->buffer_requested)
	{
		struct data_item_t *new_block;
		int err;

		err = k_mem_slab_alloc(&audio_slab, (void**) &new_block, K_NO_WAIT);
		if (err) {
			printk("alloc fail\n");
		} else {
			nrfx_err_t nrfx_err = nrfx_pdm_buffer_set(new_block->buffer, ARRAY_SIZE(new_block->buffer));
			if (nrfx_err != NRFX_SUCCESS) {
				printk("nrfx_pdm_buffer_set: %d\n", nrfx_err);
				// Free block if set buffer fails
				k_mem_slab_free(&audio_slab, (void**) &new_block);
			}
		}
		
	}
	if(p_evt->buffer_released != 0)
	{
		struct data_item_t *filled_block;

		// Get pointer to data_item_t which the released buffer is part of
		filled_block = CONTAINER_OF(p_evt->buffer_released, struct data_item_t, buffer);
		k_fifo_put(&my_fifo,&filled_block);
	}
}

Your main loop (or somewhere else that is not in the ISR) can then fetch data from the fifo and process it:

struct data_item_t *audio_block;

audio_block = k_fifo_get(&my_fifo, K_FOREVER);
if (audio_block == NULL) {
	printk("fifo error\n");
} else {
	// TODO: Process audio_block->buffer

	k_mem_slab_free(&audio_slab, (void**) &audio_block);
}

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/314331#314331"]In the python script you gave me why do we require extra processing after creating a PCM list of integers in first couple of lines. Is it just to generate .wav files or it have anything to do with PCM data directly. Also in PCM list we have converted negative int to positive int.[/quote]

The extra processing is indeed to make python wave library happy. The human-readable format would be signed 16-bit integers. That is, numbers in the range [-32767, 32767]. Your text output was as unsigned hex, which needed some massaging for python wave.

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/314331#314331"]So is it required to transform the data received from PDM interface in little endian format to train the model ? From the Nordic docs "If OPERATION=Mono, RAM will contain a succession of mono samples." i am assuming i can directly use the raw PCM data available in my int16_t buffer for training and testing. [/quote]

You should be able to use the raw PCM data (int16_t) directly. The questions is what format you use to transfer it from the nRF52840 to your training system. You can print it as text (like you did before), or in some binary format. Text is easier to read, but has more overhead than a binary format. The easiest is probably to match the expected input format in your training system, assuming the UART is fast enough to transfer this format.

If you transfer the data over UART as a binary format, you need to take the endianness into account in the training system. For example, lets say on the host side you read the audio as a 32000 byte buffer. To convert 32000 bytes into 16000 signed 16-bit integers, the proper endianness must be known.

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Mon, 07 Jun 2021 09:29:43 GMT

Btw, I realized I linked to and old version of the documentation, but these function names have not changed as far as I know. Also, note that in the PDM ISR/handler, you must used timeout value K_NO_WAIT

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Mon, 07 Jun 2021 08:47:35 GMT

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/313642#313642"]If you find any bug in above code kindly mention or have any another tricks than please help because i can't recognize voice patterns just on the basis of 2-3 sec audio.[/quote]

I think it would make things easier if you use some data structures to help organize the PCM buffers. Specifically, I recommend using a memory slab for allocating PCM buffers, and a queue/FIFO to enqueue PCM buffers that have been filled.

Let's say your algorithm runs on audio block sizes of 100 milliseconds, at 16 kHz sampling rate, this means each audio block is 3200 bytes. You would then define a memory slab of size X * 3200. X can be any number that fits within total memory. Note that it is probably best to define a struct for your audio data, which contains the int16_t audio buffer, plus the fifo_reserved data field, plus whatever else might be useful.

In the PDM event hander, when a new buffer is requested, you try to allocate a new buffer from the slab (k_mem_slab_alloc). If this fails, you can log the error.

In the PDM event handler, when a freshly filled buffer is released, you put the pointer to this buffer in the FIFO (k_fifo_put).

Your algorithm can run as soon as there is one or more items in the FIFO (k_fifo_get).

Once the algorithm has finished processing one audio buffer, it will release it back to the memory slab (k_mem_slab_free).

This way your algorithm will always process the audio in order, and it is easy to see if you have a bottleneck somewhere.

If you have multiple consumers of your audio data (e.g. local algorithm plus UART printout), this can be solved by adding a reference counter to the audio buffer struct.

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Fri, 04 Jun 2021 10:29:47 GMT

The UART baud rate might be an issue here. The default is 115200 baud, while your audio is 16-bit * 16 kHz = 256000. To test if this is the problem, you can try setting the board baud rate to 1000000 instead of 115200. The proper way to do this is to set up an overlay file in your sample, but as a quick test you can also update this config: https://github.com/nrfconnect/sdk-zephyr/blob/master/boards/arm/nrf52840dk_nrf52840/nrf52840dk_nrf52840.dts#L146

I also recommend checking out the USBD audio sample: https://github.com/nrfconnect/sdk-zephyr/tree/master/samples/subsys/usb/audio/headset This sample makes the nRF52840-DK appear as an audio device when connected to a USB host. With this you could copy the audio from the PDM directly to USB, and listen to the audio in real time on your computer without any additional scripts.

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Thu, 03 Jun 2021 13:03:44 GMT

This seems to work better. Tested using python3.

import sys
import wave
import re
import struct

with open('0317.test.txt', 'r') as pcmfile:
    # Convert comma-separated hex strings into list of integers
    pcm_list = list(map(lambda x: int(x, 16), re.findall(r'([\w\d]+)', pcmfile.read())))

# Convert list of integers into little endian-encoded byte array
pcmdata = bytes()
    
for sample in pcm_list:
    # Original sample is signed 16-bit. Format as uint16, and decode as int16.
    tmpdata = struct.pack('<H', sample)
    tmpdata = struct.unpack('<h', tmpdata)[0]
    pcmdata += struct.pack('<h', tmpdata)
    
with wave.open('0317.test.wav', 'wb') as wavfile:
    wavfile.setparams((1, 2, 16000, 0, 'NONE', 'NONE'))
    wavfile.writeframes(pcmdata)

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Thu, 03 Jun 2021 12:50:40 GMT

Hello,

there are a couple of problems with the script and data:
1) wavfile.writeframes() needs raw int16_t byte data, not a string. Your read the text file as a string, but to put it in writeframes each of the values in the comma-separated string needs to be extracted and formatted into a continuous array of formatted bytes.

2) The first parameter (number of channels) in wavfile.setparams is 2, but the data seems to be single channel? How is the PDM peripheral configured in the code?

3) I would recommend printing your buffer as int16_t (signed short), as this is the PCM format generated by the PDM peripheral. When I plot the data it looks somewhat good, but there also seems to be some formatting error going on in the printouts:

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Wed, 02 Jun 2021 10:42:18 GMT

Sorry, I have been thinking about this one for a bit, and I don't have any good ideas. I checked if it could be a VDD supply issue regarding nRF_USB, but I don't think that is a problem. VDD on the headers should be able to supply external circuitry regardless of which USB connection you have used.

I think it would be worthwhile to test another sensor, in case this particular unit is faulty.

To summarize: You've verified that VDD and CLK is within the sensor specification (albeit in the lower end for the clock, but that should not matter), but still the output does not look good. You've also verified that there is no external circuitry messing with the output signal, by disconnecting the data line. I think the next step would be to try another sensor unit.

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Thu, 27 May 2021 11:17:19 GMT

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/311948#311948"]Can it be a problem with my sensor ? It is outputting at PDM CLK rate but false data. [/quote]

It's possible. If you have another sensor at hand, it would be worth checking.

I re-read your description the setup. Am I understanding correctly that you don't have this USB connected?

Have you measured the voltage on the sensor VDD?

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Wed, 26 May 2021 14:22:00 GMT

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/311764#311764"]

L/R pin connection Information

When i see the nRF52840 PDM docs (Fig 2), it shows i have to connect L/R pin to VDD for left channel, but my sensor say opposite.

[/quote]

I wouldn't worry so much about the L/R selection or clock edge details at this stage. In my experience when only one microphone is connected to the PDM peripheral, the data signal will in effect end up in both channels as no other sensor is driving the data line for the other channel. Regardless, you can configure the nrfx pdm driver to sample stereo and sort out the exact channel details later.

Your hardware setup sounds sensible. The MP34DT05 supply voltage falls within VDD on the nRF52840-DK. I think VDD on nRF52840-DK defaults to 1.8V, but I'm not 100% sure.

[quote userid="104261" url="~/f/nordic-q-a/75483/nrfx-based-pdm-to-pcm-implementation-i2s-example/311764#311764"]Okay will also check that and will also use python script on the same. Any others suggestions or guidance is also appreciated.[/quote]

In terms of electrical setup, wire length could be an issue.

In the logic analyzer trace made at the sensor, I also noticed that the two signals seems very similar. It's hard to tell from just this snippet, but could there be a short between the clock and data lines?

This screenshot below was taken a while ago, but shows the kind of pattern I would expect to see on the PDM clock and data lines:

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Wed, 26 May 2021 11:24:30 GMT

Thanks for the Logic analyzer traces!

The logic analyzer trace from the sensor directly looks to be more in range of the expected frequency, but it's hard to say if it's good or not just by looking at this snippet. Did you try running the python script on this measurement?

It's possible that the issue is electrical. What does your test setup look like? Are you powering the sensor from one of the nRF52840-DK headers, or is there a separate supply? Are there other components connected to the CLK or DIN lines?

I would recommend connecting the CLK signal to the sensor, but keep the DIN line to nRF52840 disconnected. This way you can measure the data output on the sensor again, without getting potential interference from the nRF52840-DK. For example if the nRF52840 is configured with a pull resistor on P1.05, this could influence the signal level.

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Tue, 25 May 2021 13:11:56 GMT

Hello,

first off, I would recommend printing out the entire pdm_buf array and look at the contents. The microphone has a ~10 ms startup time (see below), so it is likely that the first sample in the first buffer will always be the same value.

Second, data on the DIN pin is normally in the same frequency range as the clock signal, but this will vary according to the acoustic input to the sensor.

If you want to validate the DIN signal you see in the logic analyzer, you can try the attached python script. The script parses and filters an exported .csv from Saleae Logic analyzer software. Just make sure the exported csv has DIN as the first exported value. For example:

Time [s],Channel 5,Channel 7
-0.000051150,0,0
0.000000000,1,0
0.000000030,1,1
0.000999980,0,1
0.001000018,0,0
0.001999976,1,0
0.002000010,1,1
0.002999954,0,1
0.002999998,0,0

Here, channel 5 is DIN, and Channel 7 is CLK.

Python script syntax, using 16 kHz as target PCM rate:

$ python pdm_parse.py my_exported_saleae_capture.csv 16000

devzone.nordicsemi.com/.../pdm_5F00_parse.py

RE: nRFX based PDM to PCM implementation (I2S example)

Audun — Tue, 25 May 2021 12:20:36 GMT

Hello,

instead of the NRFX I2S driver, it sounds like you should use the NRFX PDM driver: https://infocenter.nordicsemi.com/topic/sdk_nrf5_v17.0.2/group__nrf__pdm.html

The PDM driver interfaces with the PDM hardware on nRF52840: https://infocenter.nordicsemi.com/topic/ps_nrf52840/pdm.html?cp=4_0_0_5_14

The PDM hardware has the decimation filtering built into hardware, and outputs PCM samples. Two decimation factors are supported, 64 and 80. For example, if you want a 16 kHz PCM rate, you should choose a PDM clock rate of 1.28 MHz, and a decimation factor of 80.

Best regards,

Audun