SPI/USB will drop large multiple of packets in a row. Can't keep up with 1 Mbps on nrf52840 dk?

[ryerye120]

Hi,

I'm using an nrf52840 dk to query another device over spi and then pipe the response back USB to a host PC. Ideally, we're piping data at 1 Mbps (which should be very much doable for USB) and for a little bit, it will work perfectly. Then randomly we'll miss 4 kB worth of packets. After plotting the data stream it's become apparent that it's quite bad. We consistently lose at least a 3rd of our packets. Furthermore, the lost data is lost in chunks. Here's an example of a 'noisy' sine wave that we sent through the serial link.

I don't believe it's an issue with the spi because I've tied an oscope to the spi pins and observed perfect spi packets being transmitted at a rate of 1 Mbps (but I could only observe maybe 20 packets at a time and didn't decode them so ...). Given that we're missing consecutive packets, and it seems to happen more or less regularly, I think this might have to do with a USB overhead issue.  

Is there something I'm missing about USB? Is there an ideal packet size that I should send over the USB at a given moment? I've played with the following settings but they've seem to make no difference:

• APP_USBD_CONFIG_SOF_HANDLING_MODE (tried both normal and compress queue)
• APP_USBD_CONFIG_EVENT_QUEUE_SIZE (Tried setting this to the max, 64, in case I was trying to send packets too frequently and they needed a larger queue)

I'm pretty new with USB communication so any help would be amazing! 

EDIT: I've started saving the data to a buffer on the dev kit and reading it all out at once (ideally this decouples the SPI and USB). While I'm missing significantly fewer packets, I am still missing a few. Now I'm getting 75%-85% of all my packets. So I guess this means there is some function that's interrupting the SPI's interrupt, possibly? I've tried bumping the SPI interrupts priority but it doesn't seem to have changed anything. I'm completely out of ideas so any pointers would be a huge help.

Thanks,

Ryan

[Håkon Alseth]

Hi,

 

Is the SPIM or SPIS running on the nRF?

If logging is an issue, you could use RTT instead of UART as the logger backend (see sdk_config.h::NRF_LOG_BACKEND_RTT_ENABLED and clearing NRF_LOG_BACKEND_UART_ENABLED) to see if something is caught and printed.

 

How large are the USB packets and how often do you send? How does your device enumerate, HID, CDC, libusb, winusb?

Note that HID is limited to 64 byte per 1 ms frame.

 

Kind regards,

Håkon

[ryerye120]

The SPIM is running on the nRF.

I can try with RTT to see if the chip select issue continues (you're right in that I was using UART debugging) - and if not, I can look into debugging with that.

I send 520 bits every 500 us (so throughput is slightly higher than 1 Mbps). The software will send/receive 520 bits over SPI, then after a flag (spi_xfr_complete flag) is tripped in the main() body. After this flag is tripped, the data is processed (minorly) and then sent via USB to the host pc.

I've built this off the CDC example so it enumerates as CDC with whatever settings were in the usb_CDC example. Does CDC have a similar limit?

Thank you for the help!

[Mojo]

What is your SPI speed settings ?

[ryerye120]

The SPI clock? I set it to 4MHz because I was getting set up errors at 8MHz. 

I should note that I found a bug in my code. Now if I save all the SPI codes and send them over USB *after* the fact, I don't drop any packets. So is there a recommended way of transferring data from SPI to USB on a packet-per-packet basis? 

Earlier I had a naive nested conditional:

app_usbd_event_queue_process();
if (spi_xfer_done)
{
    spis_xfer_done = 0;
    if (usb_xfer_done)
    {
        size = sprintf(some data);
        app_usbd_cdc_acm_write(some data, size);
        usb_xfer_done = 0;
    }
}

Where spi_xfer_done is set to 1 by the spi_event_handler & usb_xfer_done is set to 1 by a handler looking out for the APP_USBD_CDC_ACM_USER_EVT_TX_DONE event.

This assumes that the USB driver is much faster than the SPI driver and will finish before the next SPI transaction is fired off/complete. Unfortunately, I must be missing something because experimentally speaking, this doesn't work. 

[Håkon Alseth]

Did you call app_usbd_cdc_acm_write() directly from the SPI interrupt callback prior to this? By moving it to main, with a flag "usb_xfer_done", you are essentially forcing the usbd process to occur in main priority instead of interrupt priority, which is a good thing, as you want interrupts to finish processing as quick as possible.

Do you clear the usb_xfer_done flag somewhere else in that loop? Right now, your code will run the inner if-sentence regardless of if the usb is finished or not.

 

Kind regards,

Håkon

[ryerye120]

Hi Hakon!

Håkon Alseth said:

Did you call app_usbd_cdc_acm_write() directly from the SPI interrupt callback prior to this?

I tried this a long time ago but ultimately moved everything to the main specifically because of what you were saying. I worried that if I put the data processing and the usb_write() in the SPI interrupt handler, then I'd run into issues down the line. Thanks for confirming this because I'm kind of going off the cuff here.

Håkon Alseth said:

Do you clear the usb_xfer_done flag somewhere else in that loop?

Yea sorry, in my actual code, I actually clear usb_xfer_done in that inner if-sentence. I've updated the above blurb to reflect this. 

Am I right to assume that the USB driver finishes really fast? Or is there overhead that I'm not accounting for.

Right now I'm considering putting a FIFO  in that the spi will push data into and the USB driver will pop from? Unfortunately, I don't know if there's enough space to make this FIFO worth it since it would need to be 520 bits x 1024 at least...

Also thanks Hakon and everyone who's chimed in. I really appreciate all of your help. Otherwise, I'd be on my lonesome and be a sad boy.

[Håkon Alseth]

Hi Ryan,

 

ryerye120 said:

Am I right to assume that the USB driver finishes really fast? Or is there overhead that I'm not accounting for.

The call to app_usbd_cdc_acm_write() will not be a blocking call (ie: will return before the transaction is done), and eventually you will get a callback in your event handler with event "APP_USBD_CDC_ACM_USER_EVT_TX_DONE". At this point, the transaction is done and fully transferred to the USB host.

It is therefore very important that you check the return code from app_usbd_cdc_acm_write(). If your timing on SPI / USB is asynchronous to each other, then you can get into a scenario where you push several packets through USB, and this function returning NRF_ERROR_BUSY (0x11, 17 decimal). At this point, you should wait, then try to call the _acm_write() function again. Its simplest form for testing purposes is: while (app_usbd_cdc_acm_write(...) != NRF_SUCCESS);

These types of while-loops should not be called from interrupts (including callback handlers)

ryerye120 said:

Right now I'm considering putting a FIFO  in that the spi will push data into and the USB driver will pop from? Unfortunately, I don't know if there's enough space to make this FIFO worth it since it would need to be 520 bits x 1024 at least...

If you need to buffer up 1024 entries, this indicates a very large delay in USB communications. Having a buffer that is between 2-10 entries deep should be sufficient, unless you plan to buffer while the USB is inactive (not plugged in)

 

ryerye120 said:

Also thanks Hakon and everyone who's chimed in. I really appreciate all of your help. Otherwise, I'd be on my lonesome and be a sad boy.

Happy to help! No one wants you to be sad and lonesome.

 

Cheers,

Håkon 

  

[ryerye120]

Håkon Alseth said:

It is therefore very important that you check the return code from app_usbd_cdc_acm_write().

Of course! I didn't appreciate that app_usbd_cdc_acm_write() was non-blocking and wasn't checking the return value. This is definitely a major issue that I've now fixed in my code.

Unfortunately, that alone didn't seem to fix the issue - nor did the FIFO - nor did the combination.

I played with my FIFO depth and in the end, putting a 2000 entry FIFO between the SPI and USB seemed to do the trick (while of course checking to make sure the usb_write() returned NRF_SUCCESS).

Unfortunately, I worry that I haven't solved the root issue and that putting such a deep FIFO has just pushed the error further down the road.

My code currently looks like:

app_usbd_event_queue_process();

if (spis_xfer_done)
{
  spis_xfer_done = 0;
  data = process_data();
  fifo_status = fifoWrite(data); //if FIFO full, doesn't do anything
  if (fifo_status == 0) 
  {
    NRF_LOG_INFO("ERROR: FIFO FULL");
  }
}

if (usb_xfer_done)
{
  if (fifoRead()) //if FIFO empty, function returns 0 and skips if block
  {
    // fifo updates pointers so that usb_data points to the right place
    size_t size = sprintf(usb_data)
    while (app_usbd_cdc_acm_write(&m_app_cdc_acm, m_tx_buffer, size) != NRF_SUCCESS);    
    usb_xfer_done = 0;
  }
  else
  {
    NRF_LOG_INFO("FIFO Empty, nothing to send");
  }
}

I only call app_usbd_event_queue_process() once because I was scared of putting a blocking while loop before the spi_processing block. Again, I'm assuming this loop will be called enough times in between interrupts that it should be fine. Do you think this may cause a problem down the line?

[Håkon Alseth]

Hi,

 

ryerye120 said:

I only call app_usbd_event_queue_process() once because I was scared of putting a blocking while loop before the spi_processing block. Again, I'm assuming this loop will be called enough times in between interrupts that it should be fine. Do you think this may cause a problem down the line?

 Sorry, I didn't think of the queue handler. try catching the "retry function" in a non-blocking fashion and setup the logic to run the _queue_process().

Has there been any progress compared to the original race-condition/missing packets? Its a bit hard to follow by looking solely on code snippets.

 

Kind regards,

Håkon

[ryerye120]

Hi Hakon,

So at first I thought it was fixed, but ultimately no. Something is slowing down the USB transfer such that the FIFO fills up and then I start missing packets.

Do you think these missed packets are related to how often I run _queue_process()?

Also what did you mean by "retry function". I interpreted as checking the usb_write() return code. So now, my code will try to such that whenever a USB transfer returns something NOT(NRF_SUCCESS) then I run _queue_process() instead and then wait till the next iteration of my main loop before trying another USB transfer. 

This way my USB_transfer doesn't block me from pushing SPI data into my FIFO.

Regardless ... I still can't empty out my FIFO fast enough.

[Håkon Alseth]

Hi,

 

ryerye120 said:

Do you think these missed packets are related to how often I run _queue_process()?

It could be, but without seeing the whole picture, I would only be guessing.

  

ryerye120 said:

Also what did you mean by "retry function". I interpreted as checking the usb_write() return code. So now, my code will try to such that whenever a USB transfer returns something NOT(NRF_SUCCESS) then I run _queue_process() instead and then wait till the next iteration of my main loop before trying another USB transfer.

Since you're using the queue, events and handling related to usbd is performed in iterations, as per how often the _queue_process() function is called. If there's something blocking the function from executing (like a delay in your main loop or similar), handling of the USBD traffic would also be delayed.

Could you share more of your code so I can have a look, or even better; be able to reproduce the problem?

 

Do you have a log file or similar which has timestamped data on the host side, to see which transfers are missing? Is the data missing, as in you see "skipped packets", or is the data delayed?

 

Kind regards,

Håkon