Passing data from PDM callback to USB audio device

[Or]

Hi,

I'm trying to pass audio data from PDM callback function to a USB device and I'm kind of stuck.

Background:
I'm using the nRF52840-DK board and 2 MP34DT05-A microphones. Since there is no PDM example in the SDK I took the liberty of writing the code from scratch based on the non-legacy PDM driver. It is very similar to I2S in design so didnt take much to make it work. To verify that it works I used fstorage library to save the samples into memory nd then Segger J-Flash to dump contents to file on a windows computer and the WaveSurfer to make sure it is actual audio data. (basically up to date code and applications described here)

IT WORKED

Here is a snippet of the relevant code in my program:

static void nrfx_pdm_event_handler(nrfx_pdm_evt_t const *const p_evt) {
	/*NRF_LOG_INFO("nrfx_pdm_event_handler(buffer_released:%d buffer_requested:%d error:%d)",
			p_evt->buffer_released == NULL ? 0 : 1,
			p_evt->buffer_requested ? 1 : 0,
			p_evt->error);*/

	// we have an error ?
	if (p_evt->error != NRFX_PDM_NO_ERROR) {
		NRF_LOG_INFO("PDM error event: %d", p_evt->error);
		return;
	}

	// got a buffer with data ?
	if (p_evt->buffer_released != NULL) {
		// we have buffer ready
		audio_buffer_txed = 0;

		// do we have space to write ?
		if (flash_addr + NRFX_PDM_BUFFER_SIZE*2 < FSTORAGE_END_ADDR) {
			if (fstorage_write_done /*nrf_fstorage_is_busy(&fstorage_instance)*/) {
				// write to flash
				fstorage_write_done = 0;
				ret_code_t rc = nrf_fstorage_write(
					&fstorage_instance,     /* The instance to use. */
					flash_addr,             /* The address in flash where to store the data. */
					p_evt->buffer_released, /* A pointer to the data. */
					NRFX_PDM_BUFFER_SIZE*2, /* Lenght of the data, in bytes. */
					NULL                    /* Optional parameter, backend-dependent. */
				);

				if (rc != NRF_SUCCESS)
					NRF_LOG_ERROR("nrf_fstorage_write() = %d", rc);
			} else NRF_LOG_ERROR("fstorage busy -> dumping buffer");
		}
	}

	// expected to set a new buffer ?
	if (p_evt->buffer_requested) {
		NRF_LOG_INFO("PDM buffer requested");

		// update buffer index
		pdm_buffer_i = pdm_buffer_i == 0 ? 1 : 0;

		// set new buffer
		nrfx_pdm_buffer_set(pdm_buffer[pdm_buffer_i], NRFX_PDM_BUFFER_SIZE);
	}
}

int main(void)
{
    // .
    // .
    // .
    
    nrfx_pdm_config_t config = NRFX_PDM_DEFAULT_CONFIG(ARDUINO_SCL_PIN, ARDUINO_SDA_PIN);
    ret = nrfx_pdm_init(&config, nrfx_pdm_event_handler);
    APP_ERROR_CHECK(ret);
    
    // .
    // .
    // .
}

Now, the problem I'm facing is how to pass that data to a USB audio device.

I started with the USB audio device example in the SDK and stripped all the headphones related code since it is irrelevant.

When I looked at the code I could see that the function app_usbd_audio_class_tx_start() that actually passes data to the USB device is called inside the SOF handler of the headphones.
I needed to find a new home for it ..
My naive first attempt was to place it directly inside the PDM handler (where fstorage_write_done() is right now). To cut a long story short - it didnt work, it returns with code 8 (invalid state). So I went to the documentation and it says there that writing can only occur during an SOF event. (and it has a small example here that I tried to use).

I came up with this code (which doesnt work .. same old return code 8 - invalid state):

static void usbd_user_ev_handler(app_usbd_event_type_t event)
{
    switch (event)
    {
        case APP_USBD_EVT_DRV_SOF:
	    // make sure we're in proper state (configured)
            if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;

            // are we in the middle of a transfer ?
            if (!usb_tx_done) break;

            // do we have a buffer ready for transfer ?
            if (audio_buffer_txed) break;

        	// transfer the off-line buffer to the USB device now
            usb_tx_done = 0;
            ret_code_t ret = app_usbd_audio_class_tx_start(&m_app_audio_microphone.base,
        			pdm_buffer[pdm_buffer_i == 0 ? 1 : 0], NRFX_PDM_BUFFER_SIZE);

            // haven't started transferring -> wont reach TX done
            if (ret != NRF_SUCCESS) {
            	usb_tx_done = 1;
            	NRF_LOG_INFO("APP_USBD_EVT_DRV_SOF -> app_usbd_audio_class_tx_start = %d", ret);
            }

            // this buffer is being transferred
            else audio_buffer_txed = 1;

            break;
            
        // .
        // .
        // .
    }
    
    // .
    // .
    // .
}

My question is - how can I pass the data from the PDM callback to the USB device reliably with no errors ?

Remark #1:
In the code snippets I attached the "// ." signify mode code there that I didnt bother to attach.

Remark #2:
I didnt change anything in the definition of the USB microphone which might be a problem.
Especially the following line:

#define MIC_INTERFACES_CONFIG() APP_USBD_AUDIO_CONFIG_IN(2, 3)

When I connect the board to a Linux machine and look at dmesg output I get a warning/error that max interface should be 1. I guess the 2 and 3 values here are wrong.

Remark #3:
I read in the documentation that the isochroneous USB device can have a maximum buffer size of 1000 bytes, so I use a double buffer of 500 int16_t for the PDM:

// buffers for the PDM source
#define NRFX_PDM_BUFFER_SIZE 500
static volatile int8_t pdm_buffer_i = 0;
static int16_t pdm_buffer[2][NRFX_PDM_BUFFER_SIZE];

[Kenneth]

I assume that app_usbd_audio_class_tx_start() return NRF_ERROR_INVALID_STATE because the USB driver is not in active state (SUSTATE_ACTIVE). I would have started looking at why the driver haven't transitioned to active state (compared to default example for instance \nRF5_SDK_15.3.0_59ac345\examples\peripheral\usbd_audio).

[Or]

After debugging the SDK and checking why I get NRF_ERROR_INVALID_STATE when I call app_usbd_audio_class_tx_start() I found out that it is returned from inside the following if statement:

if (!nrf_drv_usbd_ep_enable_check(ep))
{
    return NRF_ERROR_INVALID_STATE;
}

inside the file "nRF5_SDK_15.3.0/components/libraries/usbd/app_usbd.c" inside function app_usbd_ep_transfer() (line 1820).

What can disable the endpoint ? why does it happen in the middle of using it ?

[Kenneth]

Adding the USB sniffer file, I do believe you should be able to open it by downloading Ellisys Visual USB. Seems as the application does not reply to the first 'get device descriptor' from the USB host, instead the intended data is sent on the next 'get configuration descriptor'.

Sniffer trace:

Case_236141.zip

Seems as the IN transaction from the nRF52840 which should be a response to 'get device descriptor', is instead replied on the next 'get configuration descriptors' from the USB host.

For comparison see the USB audio reference:

Case_236141_reference_usb_audio_sdk15.2.zip

[Or]

What am I doing wrong in the firmware code (which I attached) that causes this behavior ? is this a bug in the SDK ?

IS this related to the distorted audio I get ?

[Kenneth]

Maybe you are not handling the USB callbacks fast enough due to other application interrupts processing, for instance that the app_usbd_core_event_handler() give a response to late. Maybe try to change NRFX_USBD_CONFIG_IRQ_PRIORITY.

[Or]

I've changed both NRFX_USBD_CONFIG_IRQ_PRIORITY and USBD_CONFIG_IRQ_PRIORITY to 0 (highest, they were set to 6) to no avail.

Is there a chance one of the SDK developers can take a look at my code ? I'm pretty sure I follow exactly the code example and I still cant make it work (PDM audio through USB).

There is no code example on the internet that does just that and actually works. Making this one work will be a breakthrough that will be useful for other developers who are interested in using nrf52840 for USB audio.

[Kenneth]

I see your point, unfortunately I don't see this being prioritized at the moment.

Kenneth