This post is older than 2 years and might not be relevant anymore
More Info: Consider searching for newer posts

Passing data from PDM callback to USB audio device

Hi,

I'm trying to pass audio data from PDM callback function to a USB device and I'm kind of stuck.

Background:
I'm using the nRF52840-DK board and 2 MP34DT05-A microphones. Since there is no PDM example in the SDK I took the liberty of writing the code from scratch based on the non-legacy PDM driver. It is very similar to I2S in design so didnt take much to make it work. To verify that it works I used fstorage library to save the samples into memory nd then Segger J-Flash to dump contents to file on a windows computer and the WaveSurfer to make sure it is actual audio data. (basically up to date code and applications described here)

IT WORKED

Here is a snippet of the relevant code in my program:

static void nrfx_pdm_event_handler(nrfx_pdm_evt_t const *const p_evt) {
	/*NRF_LOG_INFO("nrfx_pdm_event_handler(buffer_released:%d buffer_requested:%d error:%d)",
			p_evt->buffer_released == NULL ? 0 : 1,
			p_evt->buffer_requested ? 1 : 0,
			p_evt->error);*/

	// we have an error ?
	if (p_evt->error != NRFX_PDM_NO_ERROR) {
		NRF_LOG_INFO("PDM error event: %d", p_evt->error);
		return;
	}

	// got a buffer with data ?
	if (p_evt->buffer_released != NULL) {
		// we have buffer ready
		audio_buffer_txed = 0;

		// do we have space to write ?
		if (flash_addr + NRFX_PDM_BUFFER_SIZE*2 < FSTORAGE_END_ADDR) {
			if (fstorage_write_done /*nrf_fstorage_is_busy(&fstorage_instance)*/) {
				// write to flash
				fstorage_write_done = 0;
				ret_code_t rc = nrf_fstorage_write(
					&fstorage_instance,     /* The instance to use. */
					flash_addr,             /* The address in flash where to store the data. */
					p_evt->buffer_released, /* A pointer to the data. */
					NRFX_PDM_BUFFER_SIZE*2, /* Lenght of the data, in bytes. */
					NULL                    /* Optional parameter, backend-dependent. */
				);

				if (rc != NRF_SUCCESS)
					NRF_LOG_ERROR("nrf_fstorage_write() = %d", rc);
			} else NRF_LOG_ERROR("fstorage busy -> dumping buffer");
		}
	}

	// expected to set a new buffer ?
	if (p_evt->buffer_requested) {
		NRF_LOG_INFO("PDM buffer requested");

		// update buffer index
		pdm_buffer_i = pdm_buffer_i == 0 ? 1 : 0;

		// set new buffer
		nrfx_pdm_buffer_set(pdm_buffer[pdm_buffer_i], NRFX_PDM_BUFFER_SIZE);
	}
}

int main(void)
{
    // .
    // .
    // .
    
    nrfx_pdm_config_t config = NRFX_PDM_DEFAULT_CONFIG(ARDUINO_SCL_PIN, ARDUINO_SDA_PIN);
    ret = nrfx_pdm_init(&config, nrfx_pdm_event_handler);
    APP_ERROR_CHECK(ret);
    
    // .
    // .
    // .
}

Now, the problem I'm facing is how to pass that data to a USB audio device.

I started with the USB audio device example in the SDK and stripped all the headphones related code since it is irrelevant.

When I looked at the code I could see that the function app_usbd_audio_class_tx_start() that actually passes data to the USB device is called inside the SOF handler of the headphones.
I needed to find a new home for it ..
My naive first attempt was to place it directly inside the PDM handler (where fstorage_write_done() is right now). To cut a long story short - it didnt work, it returns with code 8 (invalid state). So I went to the documentation and it says there that writing can only occur during an SOF event. (and it has a small example here that I tried to use).

I came up with this code (which doesnt work .. same old return code 8 - invalid state):

static void usbd_user_ev_handler(app_usbd_event_type_t event)
{
    switch (event)
    {
        case APP_USBD_EVT_DRV_SOF:
	    // make sure we're in proper state (configured)
            if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;

            // are we in the middle of a transfer ?
            if (!usb_tx_done) break;

            // do we have a buffer ready for transfer ?
            if (audio_buffer_txed) break;

        	// transfer the off-line buffer to the USB device now
            usb_tx_done = 0;
            ret_code_t ret = app_usbd_audio_class_tx_start(&m_app_audio_microphone.base,
        			pdm_buffer[pdm_buffer_i == 0 ? 1 : 0], NRFX_PDM_BUFFER_SIZE);

            // haven't started transferring -> wont reach TX done
            if (ret != NRF_SUCCESS) {
            	usb_tx_done = 1;
            	NRF_LOG_INFO("APP_USBD_EVT_DRV_SOF -> app_usbd_audio_class_tx_start = %d", ret);
            }

            // this buffer is being transferred
            else audio_buffer_txed = 1;

            break;
            
        // .
        // .
        // .
    }
    
    // .
    // .
    // .
}

My question is - how can I pass the data from the PDM callback to the USB device reliably with no errors ?

Remark #1:
In the code snippets I attached the "// ." signify mode code there that I didnt bother to attach.

Remark #2:
I didnt change anything in the definition of the USB microphone which might be a problem.
Especially the following line:

#define MIC_INTERFACES_CONFIG() APP_USBD_AUDIO_CONFIG_IN(2, 3)

When I connect the board to a Linux machine and look at dmesg output I get a warning/error that max interface should be 1. I guess the 2 and 3 values here are wrong.

Remark #3:
I read in the documentation that the isochroneous USB device can have a maximum buffer size of 1000 bytes, so I use a double buffer of 500 int16_t for the PDM:

// buffers for the PDM source
#define NRFX_PDM_BUFFER_SIZE 500
static volatile int8_t pdm_buffer_i = 0;
static int16_t pdm_buffer[2][NRFX_PDM_BUFFER_SIZE];

  • I assume that app_usbd_audio_class_tx_start() return NRF_ERROR_INVALID_STATE because the USB driver is not in active state (SUSTATE_ACTIVE). I would have started looking at why the driver haven't transitioned to active state (compared to default example for instance \nRF5_SDK_15.3.0_59ac345\examples\peripheral\usbd_audio).

  • Notice where I placed app_usbd_audio_class_tx_start() ..

    It is inside the usbd_user_ev_handler() in a switch case of APP_USBD_EVT_DRV_SOF.
    Wouldnt that event be called only if a device is at least in activated state ?

    Another thing - first thing I check is that the device is configured (if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;).
    Doesnt a device reach configured state after it is activated ?

    I attach here the entire main.c for you to take a look at (maybe you will notice something I have missed):

    #include <stdint.h>
    #include <stdbool.h>
    #include <stddef.h>
    
    #include "nrf.h"
    #include "app_util.h"
    #include "nrf_drv_usbd.h"
    #include "nrf_drv_clock.h"
    #include "nrf_gpio.h"
    #include "nrf_delay.h"
    #include "nrf_drv_power.h"
    #include "nrf_drv_pdm.h"
    
    #include "app_usbd.h"
    #include "app_usbd_core.h"
    #include "app_usbd_string_desc.h"
    #include "app_usbd_audio.h"
    #include "app_error.h"
    #include "boards.h"
    
    #include "nrf_drv_clock.h"
    #include "nrf_fstorage_nvmc.h"
    
    #include "nrf_log.h"
    #include "nrf_log_ctrl.h"
    #include "nrf_log_default_backends.h"
    
    /**@file
     * @defgroup usbd_aduio_example main.c
     * @{
     * @ingroup usbd_audio_example
     * @brief USBD Audio class example
     *
     */
    
    /* FSTORAGE
     */
    void fstorage_callback(nrf_fstorage_evt_t * p_evt);
    
    #define FSTORAGE_BASE_ADDR 0x60000
    #define FSTORAGE_END_ADDR 0x80000
    
    static volatile uint32_t flash_addr = FSTORAGE_BASE_ADDR;
    static volatile uint8_t fstorage_write_done = 1;
    
    NRF_FSTORAGE_DEF(nrf_fstorage_t fstorage_instance) =
    {
        .evt_handler    = fstorage_callback,
        .start_addr     = FSTORAGE_BASE_ADDR,
        .end_addr       = FSTORAGE_END_ADDR,
    };
    
    // buffers for the PDM source
    #define NRFX_PDM_BUFFER_SIZE 500
    static volatile int8_t pdm_buffer_i = 0;
    static int16_t pdm_buffer[2][NRFX_PDM_BUFFER_SIZE];
    static volatile int8_t usb_tx_done = 1, audio_buffer_txed = 1;
    
    #define LED_USB_RESUME (BSP_BOARD_LED_0)
    #define LED_USB_START  (BSP_BOARD_LED_1)
    #define LED_AUDIO_RX   (BSP_BOARD_LED_2)
    #define LED_AUDIO_TX   (BSP_BOARD_LED_3)
    
    /**
     * @brief USB audio samples size
     */
    #define BUFFER_SIZE  (48)
    
    /**
     * @brief Enable power USB detection
     *
     * Configure if example supports USB port connection
     */
    #ifndef USBD_POWER_DETECTION
    #define USBD_POWER_DETECTION true
    #endif
    
    /**
     * @brief Audio class user event handler
     */
    static void mic_audio_user_ev_handler(app_usbd_class_inst_t const * p_inst,
                                          app_usbd_audio_user_event_t   event);
    
    /* Channels and feature controls configuration */
    
    /**
     * @brief   Input terminal channel configuration
     */
    #define MIC_TERMINAL_CH_CONFIG()                                                                       \
            (APP_USBD_AUDIO_IN_TERM_CH_CONFIG_LEFT_FRONT | APP_USBD_AUDIO_IN_TERM_CH_CONFIG_RIGHT_FRONT)
    
    /**
     * @brief   Feature controls
     *
     *      general
     *      channel 0
     *      channel 1
     */
    #define MIC_FEATURE_CONTROLS()                                                                     \
            APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE),                         \
            APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE),                         \
            APP_USBD_U16_TO_RAW_DSC(APP_USBD_AUDIO_FEATURE_UNIT_CONTROL_MUTE)
    
    
    /* Microphone descriptors */
    
    /**
     * @brief   Audio class specific format descriptor
     */
    APP_USBD_AUDIO_FORMAT_DESCRIPTOR(m_mic_form_desc, 
                                     APP_USBD_AUDIO_AS_FORMAT_I_DSC(    /* Format type 1 descriptor */
                                        2,                              /* Number of channels */
                                        2,                              /* Subframe size */
                                        16,                             /* Bit resolution */
                                        1,                              /* Frequency type */
                                        APP_USBD_U24_TO_RAW_DSC(16000)) /* Frequency */
                                    );
    
    /**
     * @brief   Audio class input terminal descriptor
     */
    APP_USBD_AUDIO_INPUT_DESCRIPTOR(m_mic_inp_desc, 
                                    APP_USBD_AUDIO_INPUT_TERMINAL_DSC(
                                        1,                                     /* Terminal ID */
                                        APP_USBD_AUDIO_TERMINAL_IN_MICROPHONE, /* Terminal type */
                                        2,                                     /* Number of channels */
                                        MIC_TERMINAL_CH_CONFIG())              /* Channels config */
                                    );
    
    /**
     * @brief   Audio class output terminal descriptor
     */
    APP_USBD_AUDIO_OUTPUT_DESCRIPTOR(m_mic_out_desc, 
                                     APP_USBD_AUDIO_OUTPUT_TERMINAL_DSC(
                                        3,                                     /* Terminal ID */
                                        APP_USBD_AUDIO_TERMINAL_USB_STREAMING, /* Terminal type */
                                        2)                                     /* Source ID */
                                    );
    
    /**
     * @brief   Audio class feature unit descriptor
     */
    APP_USBD_AUDIO_FEATURE_DESCRIPTOR(m_mic_fea_desc, 
                                      APP_USBD_AUDIO_FEATURE_UNIT_DSC(
                                        2,                      /* Unit ID */
                                        1,                      /* Source ID */
                                        MIC_FEATURE_CONTROLS()) /* List of controls */
                                     );
    
    /* Interfaces lists */
    
    /**
     * @brief Interfaces list passed to @ref APP_USBD_AUDIO_GLOBAL_DEF
     */
    #define MIC_INTERFACES_CONFIG() APP_USBD_AUDIO_CONFIG_IN(2, 3)
    
    /*lint -save -e26 -e64 -e123 -e505 -e651*/
    
    /**
     * @brief Microphone Audio class instance
     */
    APP_USBD_AUDIO_GLOBAL_DEF(m_app_audio_microphone,
                              MIC_INTERFACES_CONFIG(),
                              mic_audio_user_ev_handler,
                              &m_mic_form_desc,
                              &m_mic_inp_desc,
                              &m_mic_out_desc,
                              &m_mic_fea_desc,
                              0,
                              APP_USBD_AUDIO_AS_IFACE_FORMAT_PCM,
                              192,
                              APP_USBD_AUDIO_SUBCLASS_AUDIOSTREAMING
    );
    
    
    /*lint -restore*/
    
    /**
     * @brief Internal audio temporary buffer
     */
    //static int16_t  m_temp_buffer[2 * BUFFER_SIZE];
    
    
    
    /**
     * @brief The size of last received block from the microphone
     */
    //static size_t m_temp_buffer_size;
    
    /**
     * @brief Actual microphone mute state
     */
    static uint8_t  m_mute_mic;
    
    /**
     * @brief Actual microphone sampling frequency
     */
    static uint32_t m_freq_mic;
    
    /**
     * @brief Audio class specific request handle (microphone)
     */
    static void mic_audio_user_class_req(app_usbd_class_inst_t const * p_inst)
    {
        app_usbd_audio_t const * p_audio = app_usbd_audio_class_get(p_inst);
        app_usbd_audio_req_t * p_req = app_usbd_audio_class_request_get(p_audio);
    
        UNUSED_VARIABLE(m_mute_mic);
        UNUSED_VARIABLE(m_freq_mic);
    
        switch (p_req->req_target)
        {
            case APP_USBD_AUDIO_CLASS_REQ_IN:
    
                if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
                {
                    //Only mute control is defined
                    p_req->payload[0] = m_mute_mic;
                }
    
                break;
            case APP_USBD_AUDIO_CLASS_REQ_OUT:
    
                if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
                {
                    //Only mute control is defined
                    m_mute_mic = p_req->payload[0];
                }
    
                break;
            case APP_USBD_AUDIO_EP_REQ_IN:
                break;
            case APP_USBD_AUDIO_EP_REQ_OUT:
    
                if (p_req->req_type == APP_USBD_AUDIO_REQ_SET_CUR)
                {
                    //Only set frequency is supported
                    m_freq_mic = uint24_decode(p_req->payload);
                }
    
                break;
            default:
                break;
        }
    }
    
    /**
     * @brief User event handler @ref app_usbd_audio_user_ev_handler_t (microphone)
     */
    static void mic_audio_user_ev_handler(app_usbd_class_inst_t const * p_inst,
                                          app_usbd_audio_user_event_t   event)
    {
        app_usbd_audio_t const * p_audio = app_usbd_audio_class_get(p_inst);
        UNUSED_VARIABLE(p_audio);
    
        switch (event)
        {
            case APP_USBD_AUDIO_USER_EVT_CLASS_REQ:
                mic_audio_user_class_req(p_inst);
                break;
            case APP_USBD_AUDIO_USER_EVT_TX_DONE:
            {
            	usb_tx_done = 1;
                bsp_board_led_invert(LED_AUDIO_TX);
                break;
            }
            default:
                break;
        }
    }
    
    /**
     * @brief USBD library specific event handler.
     *
     * @param event     USBD library event.
     */
    static void usbd_user_ev_handler(app_usbd_event_type_t event)
    {
        switch (event)
        {
            case APP_USBD_EVT_DRV_SOF:
                // make sure we're in proper state (configured)
                if (APP_USBD_STATE_Configured != app_usbd_core_state_get()) break;
    
                // are we in the middle of a transfer ?
                if (!usb_tx_done) break;
    
                // do we have a buffer ready for transfer ?
                if (audio_buffer_txed) break;
    
            	// transfer the off-line buffer to the USB device now
                usb_tx_done = 0;
                ret_code_t ret = app_usbd_audio_class_tx_start(&m_app_audio_microphone.base,
            			pdm_buffer[pdm_buffer_i == 0 ? 1 : 0], NRFX_PDM_BUFFER_SIZE);
    
                // haven't started transferring -> wont reach TX done
                if (ret != NRF_SUCCESS) {
                	usb_tx_done = 1;
                	NRF_LOG_INFO("APP_USBD_EVT_DRV_SOF -> app_usbd_audio_class_tx_start = %d", ret);
                }
    
                // this buffer is being transferred
                else audio_buffer_txed = 1;
    
                break;
            case APP_USBD_EVT_DRV_SUSPEND:
                bsp_board_leds_off();
                break;
            case APP_USBD_EVT_DRV_RESUME:
                bsp_board_led_on(LED_USB_RESUME);
                break;
            case APP_USBD_EVT_STARTED:
                bsp_board_led_on(LED_USB_START);
    
                // start PDM
                APP_ERROR_CHECK(nrfx_pdm_start());
    
                break;
            case APP_USBD_EVT_STOPPED:
            	// start PDM
            	APP_ERROR_CHECK(nrfx_pdm_stop());
    
                app_usbd_disable();
                bsp_board_leds_off();
                break;
            case APP_USBD_EVT_POWER_DETECTED:
                NRF_LOG_INFO("USB power detected");
    
                if (!nrf_drv_usbd_is_enabled())
                {
                    app_usbd_enable();
                }
                break;
            case APP_USBD_EVT_POWER_REMOVED:
                NRF_LOG_INFO("USB power removed");
                app_usbd_stop();
                break;
            case APP_USBD_EVT_POWER_READY:
                NRF_LOG_INFO("USB ready");
                app_usbd_start();
                break;
            default:
                break;
        }
    }
    
    static void nrfx_pdm_event_handler(nrfx_pdm_evt_t const *const p_evt) {
    	/*NRF_LOG_INFO("nrfx_pdm_event_handler(buffer_released:%d buffer_requested:%d error:%d)",
    			p_evt->buffer_released == NULL ? 0 : 1,
    			p_evt->buffer_requested ? 1 : 0,
    			p_evt->error);*/
    
    	// we have an error ?
    	if (p_evt->error != NRFX_PDM_NO_ERROR) {
    		NRF_LOG_INFO("PDM error event: %d", p_evt->error);
    		return;
    	}
    
    	// got a buffer with data ?
    	if (p_evt->buffer_released != NULL) {
    		// we have buffer ready
    		audio_buffer_txed = 0;
    
    		// do we have space to write ?
    		if (flash_addr + NRFX_PDM_BUFFER_SIZE*2 < FSTORAGE_END_ADDR) {
    			if (fstorage_write_done /*nrf_fstorage_is_busy(&fstorage_instance)*/) {
    				// write to flash
    				fstorage_write_done = 0;
    				ret_code_t rc = nrf_fstorage_write(
    					&fstorage_instance,     /* The instance to use. */
    					flash_addr,             /* The address in flash where to store the data. */
    					p_evt->buffer_released, /* A pointer to the data. */
    					NRFX_PDM_BUFFER_SIZE*2, /* Lenght of the data, in bytes. */
    					NULL                    /* Optional parameter, backend-dependent. */
    				);
    
    				if (rc != NRF_SUCCESS)
    					NRF_LOG_ERROR("nrf_fstorage_write() = %d", rc);
    			} else NRF_LOG_ERROR("fstorage busy -> dumping buffer");
    		}
    	}
    
    	// expected to set a new buffer ?
    	if (p_evt->buffer_requested) {
    		NRF_LOG_INFO("PDM buffer requested");
    
    		// update buffer index
    		pdm_buffer_i = pdm_buffer_i == 0 ? 1 : 0;
    
    		// set new buffer
    		nrfx_pdm_buffer_set(pdm_buffer[pdm_buffer_i], NRFX_PDM_BUFFER_SIZE);
    	}
    }
    
    void fstorage_callback(nrf_fstorage_evt_t * p_evt) {
    	if (p_evt->result != NRF_SUCCESS) {
    		NRF_LOG_INFO("--> Event received: ERROR while executing an fstorage operation.");
    		return;
    	}
    
    	switch (p_evt->id) {
    	case NRF_FSTORAGE_EVT_WRITE_RESULT:
    		NRF_LOG_INFO("--> Event received: wrote %d bytes at address 0x%x.", p_evt->len, p_evt->addr);
    		flash_addr += NRFX_PDM_BUFFER_SIZE*2;
    		fstorage_write_done = 1;
    		break;
    
    	default:
    		break;
    	}
    }
    
    int main(void)
    {
        ret_code_t ret;
        static const app_usbd_config_t usbd_config = {
            .ev_state_proc = usbd_user_ev_handler,
            .enable_sof = true
        };
    
        ret = NRF_LOG_INIT(NULL);
        APP_ERROR_CHECK(ret);
        NRF_LOG_DEFAULT_BACKENDS_INIT();
    
        ret = nrf_drv_clock_init();
        APP_ERROR_CHECK(ret);
    
        ret = nrf_fstorage_init(
    		&fstorage_instance, /* You fstorage instance, previously defined. */
    		&nrf_fstorage_nvmc, /* Name of the backend. */
    		NULL                /* Optional parameter, backend-dependant. */
    	);
        APP_ERROR_CHECK(ret);
    
        nrfx_pdm_config_t config = NRFX_PDM_DEFAULT_CONFIG(ARDUINO_SCL_PIN, ARDUINO_SDA_PIN);
        ret = nrfx_pdm_init(&config, nrfx_pdm_event_handler);
        APP_ERROR_CHECK(ret);
    
        NRF_LOG_INFO("USBD audio example started.");
    
        // Initialize LEDs and buttons
        bsp_board_init(BSP_INIT_LEDS | BSP_INIT_BUTTONS);
    
        ret = app_usbd_init(&usbd_config);
        APP_ERROR_CHECK(ret);
    
        app_usbd_class_inst_t const * class_inst_mic =
            app_usbd_audio_class_inst_get(&m_app_audio_microphone);
        ret = app_usbd_class_append(class_inst_mic);
        APP_ERROR_CHECK(ret);
    
        if (USBD_POWER_DETECTION)
        {
            ret = app_usbd_power_events_enable();
            APP_ERROR_CHECK(ret);
        }
        else
        {
            NRF_LOG_INFO("No USB power detection enabled\r\nStarting USB now");
    
            app_usbd_enable();
            app_usbd_start();
        }
    
        NRF_LOG_INFO("Entering main loop");
    
        while (true)
        {
            while (app_usbd_event_queue_process())
            {
                /* Nothing to do */
            }
    
            UNUSED_RETURN_VALUE(NRF_LOG_PROCESS());
            /* Sleep CPU only if there was no interrupt since last loop processing */
            __WFE();
        }
    }
    
    /** @} */

  • After debugging the SDK and checking why I get NRF_ERROR_INVALID_STATE when I call app_usbd_audio_class_tx_start() I found out that it is returned from inside the following if statement:

    if (!nrf_drv_usbd_ep_enable_check(ep))
    {
        return NRF_ERROR_INVALID_STATE;
    }

    inside the file "nRF5_SDK_15.3.0/components/libraries/usbd/app_usbd.c" inside function app_usbd_ep_transfer() (line 1820).

    What can disable the endpoint ? why does it happen in the middle of using it ?

  • If you can send me a hex file I may program to an nRF52840 DK here, then I can take an USB sniffer trace to check if I can see issues.

    Kenneth

  • I have embedded the whole main.c source file above 6 days ago.

    You can comment out the fstorage parts since you wont need them and find 2 PDM mics for input source.
    This was based on usb_audio code so same sdk_config (but you need to add PDM support).

    Thanks for helping, something doesnt make sense in the USB functionality

Related