Passing data from PDM callback to USB audio device

[Or]

Hi,

I'm trying to pass audio data from PDM callback function to a USB device and I'm kind of stuck.

Background:
I'm using the nRF52840-DK board and 2 MP34DT05-A microphones. Since there is no PDM example in the SDK I took the liberty of writing the code from scratch based on the non-legacy PDM driver. It is very similar to I2S in design so didnt take much to make it work. To verify that it works I used fstorage library to save the samples into memory nd then Segger J-Flash to dump contents to file on a windows computer and the WaveSurfer to make sure it is actual audio data. (basically up to date code and applications described here)

IT WORKED

Here is a snippet of the relevant code in my program:

static void nrfx_pdm_event_handler(nrfx_pdm_evt_t const *const p_evt) {
	/*NRF_LOG_INFO("nrfx_pdm_event_handler(buffer_released:%d buffer_requested:%d error:%d)",
			p_evt->buffer_released == NULL ? 0 : 1,
			p_evt->buffer_requested ? 1 : 0,
			p_evt->error);*/

	// we have an error ?
	if (p_evt->error != NRFX_PDM_NO_ERROR) {
		NRF_LOG_INFO("PDM error event: %d", p_evt->error);
		return;
	}

	// got a buffer with data ?
	if (p_evt->buffer_released != NULL) {
		// we have buffer ready
		audio_buffer_txed = 0;

		// do we have space to write ?
		if (flash_addr + NRFX_PDM_BUFFER_SIZE*2 < FSTORAGE_END_ADDR) {
			if (fstorage_write_done /*nrf_fstorage_is_busy(&fstorage_instance)*/) {
				// write to flash
				fstorage_write_done = 0;
				ret_code_t rc = nrf_fstorage_write(
					&fstorage_instance,     /* The instance to use. */
					flash_addr,             /* The address in flash where to store the data. */
					p_evt->buffer_released, /* A pointer to the data. */
					NRFX_PDM_BUFFER_SIZE*2, /* Lenght of the data, in bytes. */
					NULL                    /* Optional parameter, backend-dependent. */
				);

				if (rc != NRF_SUCCESS)
					NRF_LOG_ERROR("nrf_fstorage_write() = %d", rc);
			} else NRF_LOG_ERROR("fstorage busy -> dumping buffer");
		}
	}

	// expected to set a new buffer ?
	if (p_evt->buffer_requested) {
		NRF_LOG_INFO("PDM buffer requested");

		// update buffer index
		pdm_buffer_i = pdm_buffer_i == 0 ? 1 : 0;

		// set new buffer
		nrfx_pdm_buffer_set(pdm_buffer[pdm_buffer_i], NRFX_PDM_BUFFER_SIZE);
	}
}

int main(void)
{
    // .
    // .
    // .
    
    nrfx_pdm_config_t config = NRFX_PDM_DEFAULT_CONFIG(ARDUINO_SCL_PIN, ARDUINO_SDA_PIN);
    ret = nrfx_pdm_init(&config, nrfx_pdm_event_handler);
    APP_ERROR_CHECK(ret);
    
    // .
    // .
    // .
}

Now, the problem I'm facing is how to pass that data to a USB audio device.

I started with the USB audio device example in the SDK and stripped all the headphones related code since it is irrelevant.

When I looked at the code I could see that the function app_usbd_audio_class_tx_start() that actually passes data to the USB device is called inside the SOF handler of the headphones.
I needed to find a new home for it ..
My naive first attempt was to place it directly inside the PDM handler (where fstorage_write_done() is right now). To cut a long story short - it didnt work, it returns with code 8 (invalid state). So I went to the documentation and it says there that writing can only occur during an SOF event. (and it has a small example here that I tried to use).

I came up with this code (which doesnt work .. same old return code 8 - invalid state):

static void usbd_user_ev_handler(app_usbd_event_type_t event)
{
    switch (event)
    {
        case APP_USBD_EVT_DRV_SOF:
	    // make sure we're in proper state (configured)
            if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;

            // are we in the middle of a transfer ?
            if (!usb_tx_done) break;

            // do we have a buffer ready for transfer ?
            if (audio_buffer_txed) break;

        	// transfer the off-line buffer to the USB device now
            usb_tx_done = 0;
            ret_code_t ret = app_usbd_audio_class_tx_start(&m_app_audio_microphone.base,
        			pdm_buffer[pdm_buffer_i == 0 ? 1 : 0], NRFX_PDM_BUFFER_SIZE);

            // haven't started transferring -> wont reach TX done
            if (ret != NRF_SUCCESS) {
            	usb_tx_done = 1;
            	NRF_LOG_INFO("APP_USBD_EVT_DRV_SOF -> app_usbd_audio_class_tx_start = %d", ret);
            }

            // this buffer is being transferred
            else audio_buffer_txed = 1;

            break;
            
        // .
        // .
        // .
    }
    
    // .
    // .
    // .
}

My question is - how can I pass the data from the PDM callback to the USB device reliably with no errors ?

Remark #1:
In the code snippets I attached the "// ." signify mode code there that I didnt bother to attach.

Remark #2:
I didnt change anything in the definition of the USB microphone which might be a problem.
Especially the following line:

#define MIC_INTERFACES_CONFIG() APP_USBD_AUDIO_CONFIG_IN(2, 3)

When I connect the board to a Linux machine and look at dmesg output I get a warning/error that max interface should be 1. I guess the 2 and 3 values here are wrong.

Remark #3:
I read in the documentation that the isochroneous USB device can have a maximum buffer size of 1000 bytes, so I use a double buffer of 500 int16_t for the PDM:

// buffers for the PDM source
#define NRFX_PDM_BUFFER_SIZE 500
static volatile int8_t pdm_buffer_i = 0;
static int16_t pdm_buffer[2][NRFX_PDM_BUFFER_SIZE];

[Kenneth]

I assume that app_usbd_audio_class_tx_start() return NRF_ERROR_INVALID_STATE because the USB driver is not in active state (SUSTATE_ACTIVE). I would have started looking at why the driver haven't transitioned to active state (compared to default example for instance \nRF5_SDK_15.3.0_59ac345\examples\peripheral\usbd_audio).

[Or]

Notice where I placed app_usbd_audio_class_tx_start() ..

It is inside the usbd_user_ev_handler() in a switch case of APP_USBD_EVT_DRV_SOF.
Wouldnt that event be called only if a device is at least in activated state ?

Another thing - first thing I check is that the device is configured (if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;).
Doesnt a device reach configured state after it is activated ?

I attach here the entire main.c for you to take a look at (maybe you will notice something I have missed):

#include <stdint.h>
#include <stdbool.h>
#include <stddef.h>

#include "nrf.h"
#include "app_util.h"
#include "nrf_drv_usbd.h"
#include "nrf_drv_clock.h"
#include "nrf_gpio.h"
#include "nrf_delay.h"
#include "nrf_drv_power.h"
#include "nrf_drv_pdm.h"

#include "app_usbd.h"
#include "app_usbd_core.h"
#include "app_usbd_string_desc.h"
#include "app_usbd_audio.h"
#include "app_error.h"
#include "boards.h"

#include "nrf_drv_clock.h"
#include "nrf_fstorage_nvmc.h"

#include "nrf_log.h"
#include "nrf_log_ctrl.h"
#include "nrf_log_default_backends.h"

/**@file
 * @defgroup usbd_aduio_example main.c
 * @{
 * @ingroup usbd_audio_example
 * @brief USBD Audio class example
 *
 */

/* FSTORAGE
 */
void fstorage_callback(nrf_fstorage_evt_t * p_evt);

#define FSTORAGE_BASE_ADDR 0x60000
#define FSTORAGE_END_ADDR 0x80000

static volatile uint32_t flash_addr = FSTORAGE_BASE_ADDR;
static volatile uint8_t fstorage_write_done = 1;

NRF_FSTORAGE_DEF(nrf_fstorage_t fstorage_instance) =
{
    .evt_handler    = fstorage_callback,
    .start_addr     = FSTORAGE_BASE_ADDR,
    .end_addr       = FSTORAGE_END_ADDR,
};

// buffers for the PDM source
#define NRFX_PDM_BUFFER_SIZE 500
static volatile int8_t pdm_buffer_i = 0;
static int16_t pdm_buffer[2][NRFX_PDM_BUFFER_SIZE];
static volatile int8_t usb_tx_done = 1, audio_buffer_txed = 1;

#define LED_USB_RESUME (BSP_BOARD_LED_0)
#define LED_USB_START  (BSP_BOARD_LED_1)
#define LED_AUDIO_RX   (BSP_BOARD_LED_2)
#define LED_AUDIO_TX   (BSP_BOARD_LED_3)

/**
 * @brief USB audio samples size
 */
#define BUFFER_SIZE  (48)

/**
 * @brief Enable power USB detection
 *
 * Configure if example supports USB port connection
 */
#ifndef USBD_POWER_DETECTION
#define USBD_POWER_DETECTION true
#endif

/**
 * @brief Audio class user event handler
 */
static void mic_audio_user_ev_handler(app_usbd_class_inst_t const * p_inst,
                                      app_usbd_audio_user_event_t   event);

/* Channels and feature controls configuration */

/**
 * @brief   Input terminal channel configuration
 */
#define MIC_TERMINAL_CH_CONFIG()                                                                       \
        (APP_USBD_AUDIO_IN_TERM_CH_CONFIG_LEFT_FRONT | APP_USBD_AUDIO_IN_TERM_CH_CONFIG_RIGHT_FRONT)

/**
 * @brief   Feature controls
 *
 *      general
 *      channel 0
 *      channel 1
 */
#define MIC_FEATURE_CONTROLS()                                                                     \
        APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE),                         \
        APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE),                         \
        APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE)


/* Microphone descriptors */

/**
 * @brief   Audio class specific format descriptor
 */
APP_USBD_AUDIO_FORMAT_DESCRIPTOR(m_mic_form_desc, 
                                 APP_USBD_AUDIO_AS_FORMAT_I_DSC(    /* Format type 1 descriptor */
                                    2,                              /* Number of channels */
                                    2,                              /* Subframe size */
                                    16,                             /* Bit resolution */
                                    1,                              /* Frequency type */
                                    APP_USBD_U24_TO_RAW_DSC(16000)) /* Frequency */
                                );

/**
 * @brief   Audio class input terminal descriptor
 */
APP_USBD_AUDIO_INPUT_DESCRIPTOR(m_mic_inp_desc, 
                                APP_USBD_AUDIO_INPUT_TERMINAL_DSC(
                                    1,                                     /* Terminal ID */
                                    APP_USBD_AUDIO_TERMINAL_IN_MICROPHONE, /* Terminal type */
                                    2,                                     /* Number of channels */
                                    MIC_TERMINAL_CH_CONFIG())              /* Channels config */
                                );

/**
 * @brief   Audio class output terminal descriptor
 */
APP_USBD_AUDIO_OUTPUT_DESCRIPTOR(m_mic_out_desc, 
                                 APP_USBD_AUDIO_OUTPUT_TERMINAL_DSC(
                                    3,                                     /* Terminal ID */
                                    APP_USBD_AUDIO_TERMINAL_USB_STREAMING, /* Terminal type */
                                    2)                                     /* Source ID */
                                );

/**
 * @brief   Audio class feature unit descriptor
 */
APP_USBD_AUDIO_FEATURE_DESCRIPTOR(m_mic_fea_desc, 
                                  APP_USBD_AUDIO_FEATURE_UNIT_DSC(
                                    2,                      /* Unit ID */
                                    1,                      /* Source ID */
                                    MIC_FEATURE_CONTROLS()) /* List of controls */
                                 );

/* Interfaces lists */

/**
 * @brief Interfaces list passed to @ref APP_USBD_AUDIO_GLOBAL_DEF
 */
#define MIC_INTERFACES_CONFIG() APP_USBD_AUDIO_CONFIG_IN(2, 3)

/*lint -save -e26 -e64 -e123 -e505 -e651*/

/**
 * @brief Microphone Audio class instance
 */
APP_USBD_AUDIO_GLOBAL_DEF(m_app_audio_microphone,
                          MIC_INTERFACES_CONFIG(),
                          mic_audio_user_ev_handler,
                          &m_mic_form_desc,
                          &m_mic_inp_desc,
                          &m_mic_out_desc,
                          &m_mic_fea_desc,
                          0,
                          APP_USBD_AUDIO_AS_IFACE_FORMAT_PCM,
                          192,
                          APP_USBD_AUDIO_SUBCLASS_AUDIOSTREAMING
);


/*lint -restore*/

/**
 * @brief Internal audio temporary buffer
 */
//static int16_t  m_temp_buffer[2 * BUFFER_SIZE];



/**
 * @brief The size of last received block from the microphone
 */
//static size_t m_temp_buffer_size;

/**
 * @brief Actual microphone mute state
 */
static uint8_t  m_mute_mic;

/**
 * @brief Actual microphone sampling frequency
 */
static uint32_t m_freq_mic;

/**
 * @brief Audio class specific request handle (microphone)
 */
static void mic_audio_user_class_req(app_usbd_class_inst_t const * p_inst)
{
    app_usbd_audio_t const * p_audio = app_usbd_audio_class_get(p_inst);
    app_usbd_audio_req_t * p_req = app_usbd_audio_class_request_get(p_audio);

    UNUSED_VARIABLE(m_mute_mic);
    UNUSED_VARIABLE(m_freq_mic);

    switch (p_req->req_target)
    {
        case APP_USBD_AUDIO_CLASS_REQ_IN:

            if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
            {
                //Only mute control is defined
                p_req->payload[0] = m_mute_mic;
            }

            break;
        case APP_USBD_AUDIO_CLASS_REQ_OUT:

            if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
            {
                //Only mute control is defined
                m_mute_mic = p_req->payload[0];
            }

            break;
        case APP_USBD_AUDIO_EP_REQ_IN:
            break;
        case APP_USBD_AUDIO_EP_REQ_OUT:

            if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
            {
                //Only set frequency is supported
                m_freq_mic = uint24_decode(p_req->payload);
            }

            break;
        default:
            break;
    }
}

/**
 * @brief User event handler @ref app_usbd_audio_user_ev_handler_t (microphone)
 */
static void mic_audio_user_ev_handler(app_usbd_class_inst_t const * p_inst,
                                      app_usbd_audio_user_event_t   event)
{
    app_usbd_audio_t const * p_audio = app_usbd_audio_class_get(p_inst);
    UNUSED_VARIABLE(p_audio);

    switch (event)
    {
        case APP_USBD_AUDIO_USER_EVT_CLASS_REQ:
            mic_audio_user_class_req(p_inst);
            break;
        case APP_USBD_AUDIO_USER_EVT_TX_DONE:
        {
        	usb_tx_done = 1;
            bsp_board_led_invert(LED_AUDIO_TX);
            break;
        }
        default:
            break;
    }
}

/**
 * @brief USBD library specific event handler.
 *
 * @param event     USBD library event.
 */
static void usbd_user_ev_handler(app_usbd_event_type_t event)
{
    switch (event)
    {
        case APP_USBD_EVT_DRV_SOF:
            // make sure we're in proper state (configured)
            if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;

            // are we in the middle of a transfer ?
            if (!usb_tx_done) break;

            // do we have a buffer ready for transfer ?
            if (audio_buffer_txed) break;

        	// transfer the off-line buffer to the USB device now
            usb_tx_done = 0;
            ret_code_t ret = app_usbd_audio_class_tx_start(&m_app_audio_microphone.base,
        			pdm_buffer[pdm_buffer_i == 0 ? 1 : 0], NRFX_PDM_BUFFER_SIZE);

            // haven't started transferring -> wont reach TX done
            if (ret != NRF_SUCCESS) {
            	usb_tx_done = 1;
            	NRF_LOG_INFO("APP_USBD_EVT_DRV_SOF -> app_usbd_audio_class_tx_start = %d", ret);
            }

            // this buffer is being transferred
            else audio_buffer_txed = 1;

            break;
        case APP_USBD_EVT_DRV_SUSPEND:
            bsp_board_leds_off();
            break;
        case APP_USBD_EVT_DRV_RESUME:
            bsp_board_led_on(LED_USB_RESUME);
            break;
        case APP_USBD_EVT_STARTED:
            bsp_board_led_on(LED_USB_START);

            // start PDM
            APP_ERROR_CHECK(nrfx_pdm_start());

            break;
        case APP_USBD_EVT_STOPPED:
        	// start PDM
        	APP_ERROR_CHECK(nrfx_pdm_stop());

            app_usbd_disable();
            bsp_board_leds_off();
            break;
        case APP_USBD_EVT_POWER_DETECTED:
            NRF_LOG_INFO("USB power detected");

            if (!nrf_drv_usbd_is_enabled())
            {
                app_usbd_enable();
            }
            break;
        case APP_USBD_EVT_POWER_REMOVED:
            NRF_LOG_INFO("USB power removed");
            app_usbd_stop();
            break;
        case APP_USBD_EVT_POWER_READY:
            NRF_LOG_INFO("USB ready");
            app_usbd_start();
            break;
        default:
            break;
    }
}

static void nrfx_pdm_event_handler(nrfx_pdm_evt_t const *const p_evt) {
	/*NRF_LOG_INFO("nrfx_pdm_event_handler(buffer_released:%d buffer_requested:%d error:%d)",
			p_evt->buffer_released == NULL ? 0 : 1,
			p_evt->buffer_requested ? 1 : 0,
			p_evt->error);*/

	// we have an error ?
	if (p_evt->error != NRFX_PDM_NO_ERROR) {
		NRF_LOG_INFO("PDM error event: %d", p_evt->error);
		return;
	}

	// got a buffer with data ?
	if (p_evt->buffer_released != NULL) {
		// we have buffer ready
		audio_buffer_txed = 0;

		// do we have space to write ?
		if (flash_addr + NRFX_PDM_BUFFER_SIZE*2 < FSTORAGE_END_ADDR) {
			if (fstorage_write_done /*nrf_fstorage_is_busy(&fstorage_instance)*/) {
				// write to flash
				fstorage_write_done = 0;
				ret_code_t rc = nrf_fstorage_write(
					&fstorage_instance,     /* The instance to use. */
					flash_addr,             /* The address in flash where to store the data. */
					p_evt->buffer_released, /* A pointer to the data. */
					NRFX_PDM_BUFFER_SIZE*2, /* Lenght of the data, in bytes. */
					NULL                    /* Optional parameter, backend-dependent. */
				);

				if (rc != NRF_SUCCESS)
					NRF_LOG_ERROR("nrf_fstorage_write() = %d", rc);
			} else NRF_LOG_ERROR("fstorage busy -> dumping buffer");
		}
	}

	// expected to set a new buffer ?
	if (p_evt->buffer_requested) {
		NRF_LOG_INFO("PDM buffer requested");

		// update buffer index
		pdm_buffer_i = pdm_buffer_i == 0 ? 1 : 0;

		// set new buffer
		nrfx_pdm_buffer_set(pdm_buffer[pdm_buffer_i], NRFX_PDM_BUFFER_SIZE);
	}
}

void fstorage_callback(nrf_fstorage_evt_t * p_evt) {
	if (p_evt->result != NRF_SUCCESS) {
		NRF_LOG_INFO("--> Event received: ERROR while executing an fstorage operation.");
		return;
	}

	switch (p_evt->id) {
	case NRF_FSTORAGE_EVT_WRITE_RESULT:
		NRF_LOG_INFO("--> Event received: wrote %d bytes at address 0x%x.", p_evt->len, p_evt->addr);
		flash_addr += NRFX_PDM_BUFFER_SIZE*2;
		fstorage_write_done = 1;
		break;

	default:
		break;
	}
}

int main(void)
{
    ret_code_t ret;
    static const app_usbd_config_t usbd_config = {
        .ev_state_proc = usbd_user_ev_handler,
        .enable_sof = true
    };

    ret = NRF_LOG_INIT(NULL);
    APP_ERROR_CHECK(ret);
    NRF_LOG_DEFAULT_BACKENDS_INIT();

    ret = nrf_drv_clock_init();
    APP_ERROR_CHECK(ret);

    ret = nrf_fstorage_init(
		&fstorage_instance, /* You fstorage instance, previously defined. */
		&nrf_fstorage_nvmc, /* Name of the backend. */
		NULL                /* Optional parameter, backend-dependant. */
	);
    APP_ERROR_CHECK(ret);

    nrfx_pdm_config_t config = NRFX_PDM_DEFAULT_CONFIG(ARDUINO_SCL_PIN, ARDUINO_SDA_PIN);
    ret = nrfx_pdm_init(&config, nrfx_pdm_event_handler);
    APP_ERROR_CHECK(ret);

    NRF_LOG_INFO("USBD audio example started.");

    // Initialize LEDs and buttons
    bsp_board_init(BSP_INIT_LEDS | BSP_INIT_BUTTONS);

    ret = app_usbd_init(&usbd_config);
    APP_ERROR_CHECK(ret);

    app_usbd_class_inst_t const * class_inst_mic =
        app_usbd_audio_class_inst_get(&m_app_audio_microphone);
    ret = app_usbd_class_append(class_inst_mic);
    APP_ERROR_CHECK(ret);

    if (USBD_POWER_DETECTION)
    {
        ret = app_usbd_power_events_enable();
        APP_ERROR_CHECK(ret);
    }
    else
    {
        NRF_LOG_INFO("No USB power detection enabled\r\nStarting USB now");

        app_usbd_enable();
        app_usbd_start();
    }

    NRF_LOG_INFO("Entering main loop");

    while (true)
    {
        while (app_usbd_event_queue_process())
        {
            /* Nothing to do */
        }

        UNUSED_RETURN_VALUE(NRF_LOG_PROCESS());
        /* Sleep CPU only if there was no interrupt since last loop processing */
        __WFE();
    }
}

/** @} */

[Or]

After debugging the SDK and checking why I get NRF_ERROR_INVALID_STATE when I call app_usbd_audio_class_tx_start() I found out that it is returned from inside the following if statement:

if (!nrf_drv_usbd_ep_enable_check(ep))
{
    return NRF_ERROR_INVALID_STATE;
}

inside the file "nRF5_SDK_15.3.0/components/libraries/usbd/app_usbd.c" inside function app_usbd_ep_transfer() (line 1820).

What can disable the endpoint ? why does it happen in the middle of using it ?

[Kenneth]

If you can send me a hex file I may program to an nRF52840 DK here, then I can take an USB sniffer trace to check if I can see issues.

Kenneth

[Or]

I have embedded the whole main.c source file above 6 days ago.

You can comment out the fstorage parts since you wont need them and find 2 PDM mics for input source.
This was based on usb_audio code so same sdk_config (but you need to add PDM support).

Thanks for helping, something doesnt make sense in the USB functionality

[Kenneth]

Is there no way you can strip away the PDM for now so I may run this on a devkit without PDM?

Kenneth