USBD endpoint transfer completion event.

[Ronald Hoogenboom]

In my NRF52840 based device using NRF5 SDK 17.1.0, I notice that sometimes this loop in the nrfx usbd driver code of the SDK doesn't finish:

[modules/nrfx/drivers/src/nrfx_usbd.c: 1444]

/* There is a lot of USBD registers that cannot be accessed during EasyDMA transfer.
 * This is quick fix to maintain stability of the stack.
 * It cost some performance but makes stack stable. */
while (!nrf_usbd_event_check(nrfx_usbd_ep_to_endevent(ep)) &&
       !nrf_usbd_event_check(NRF_USBD_EVENT_USBRESET))
{
    /* Empty */
}

It seems the transfer-end EasyDMA hardware-event isn't generated reliably in all situations. This happens mostly with small transfers on interrupt-in-endpoints and only with some usb hosts (notably raspberry pi). This is a big problem, because this function is called from interrupt context (and therefore should actually not do such busy waiting).

Is this a known issue and is there a workaround? I tried to enter the loop conditionally only when transfer size is bigger than 4 or 8 bytes, which seems to fix this, but that causes other problems.  I tried to pad the 2-byte transfer to 8 bytes, but that doesn't change anything. I think it could work to skip the loop for interrupt endpoints, but this routine handles interrupt and bulk endpoints the same; there is no USBD register that keeps track of bulk vs. interrupt endpoints.

I know that NRF5 sdk is out of support, but my project was built with it and it isn't easy to move to NRF-Connect SDK.

[Ronald Hoogenboom]

I have found the reason for the issue. Look at the comments on the function usbd_dma_pending_set:

/**
 * @brief Mark that EasyDMA is working.
 *
 * Internal function to set the flag informing about EasyDMA transfer pending.
 * This function is called always just after the EasyDMA transfer is started.
 */
static inline void usbd_dma_pending_set(void)
{
    if (nrfx_usbd_errata_199())
    {
        *((volatile uint32_t *)0x40027C1C) = 0x00000082;
    }
    m_dma_pending = true;
}

It says that this function is (to be) called just AFTER the DMA is started, but in the function usbd_dmareq_process, it is called a little BEFORE the DMA is started. This isn't a problem for setting the value of m_dma_pending (of course) but it is a problem for the errata 199 workaround.

I moved the call to the pending_set function from line 1429 to right after usbd_dma_start(ep), line 1443 and it solves the issue with raspberry pi. It also still works fine with the windows laptop.

I do think this should be rectified in the NRF5 SDK and in the NRF Connect SDK (sdk-hal_nordic/nrfx/drivers/src/nrfx_usbd.c).

[Hung Bui]

Hi Ronald, 
Thanks for the report. I have forwarded your finding internally. I will keep you updated when I get a feedback. 

[Ronald Hoogenboom]

I now have the real root cause of the problem. In some situations, the host doesn't do any IN request on the interrupt endpoint after configuring the usb device. When it doesn't, and I send an interrupt transfer to the host, this phenomenon in the dmareq_process function occurs and it still does after the change I described above. I'm not sure why I initially saw that the change solved the issue.

I don't see a way to check whether the host 'activated' the interrupt endpoint (by sending IN requests) before doing the endpoint tranfer. I think a valid way to handle such situations is to just ignore the transfer, possibly returning an error, not hang in an interrupt routine in the driver code.

Maybe polling a few times for the 'STARTED' event before setting dma_pending?

[Hung Bui]

Hi Ronald, 
So from my understanding, it's the issue with the host that it doesn't send IN requests. But I still don't understand why that would cause the driver to end in a deadloop as in your initial question. Would that because the host is unresponsive and leading to END event never come ? 
Maybe we can safeguard this loop by adding a way to exit it if it stuck there for a while ?

[Ronald Hoogenboom]

Hi Hung Bui,

There are just no IN tokens coming from the host, because it didn't listen for the interrupts from the endpoint yet. This apparently results in the DMA for the triggered endpoint task to never 'capture' (no STARTED event) and thus no END event either (which is what the loop in the driver is waiting for).

I think it's better to check a few times for the STARTED event (which should come quickly, as all DMA requests are serialized by the driver software already) than have a longer timeout on the END event. ISRs are timing critical, because they block all other parts of the code. I have seen that the STARTED event always comes within 5 loops, so timing out for the STARTED event can be done with as little as 10 loops or so. Then when the STARTED loop times out, it's useless to wait for END. I'm not sure if the triggered endpoint task needs to be canceled in the timeout case. I didn't do that and it seems to not harm anything.

The fragment in the nrfx_usbd.c around line 1440 looks like this now: 

    /* Start transfer to the endpoint buffer */
    nrf_usbd_ep_easydma_set(ep, transfer.p_data.addr, (uint32_t)transfer.size);

    usbd_dma_start(ep);
    for (int i=0; ; i++)
    {
        if (nrf_usbd_event_check(NRF_USBD_EVENT_STARTED)) break;
        if (i>8)
        {
            if (NRFX_USBD_DMAREQ_PROCESS_DEBUG)
            {
                NRFX_LOG_DEBUG("USB DMA process - not started");
            }
            /* Transfer won't capture - abort */
            return;
        }
    }
    usbd_dma_pending_set();

    /* There is a lot of USBD registers that cannot be accessed during EasyDMA transfer.
     * This is quick fix to maintain stability of the stack.

This seems to make the behavior stable for both when the host sends IN tokens to the interrupt endpoint and when it doesn't.

What do you think?

[Hung Bui]

Hi Ronald, 

I would need to check internally to see if this can cause any potential issue. I will get back when I have a reply.