nRF51422 SDK11 app_uart_fifo race condition and fix

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

shadowncs — Sun, 17 Dec 2017 16:48:52 GMT

SDK12 has this issue fixed. As Hung Bui pointed out, there is still a race condition in SDK12 however it is benign if using a queue with length more than 1. If you want SDK12 to only call app handler when the first character is received, as opposed to when every character is received, you can modify app_uart_fifo.c like this

// Notify that there are data available.
else if (FIFO_LENGTH(m_rx_fifo) == 1)

== 1 instead of != 0

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

shadowncs — Fri, 15 Dec 2017 23:49:26 GMT

Unrelated question but I'n a n00b and you probably know this. How can I use SDK12 with uVision 5.24? edit: nevermind, I figured it out.

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

Hung Bui — Fri, 15 Dec 2017 12:51:27 GMT

I'm not 100% certain on why we changed the event handling. I assume, in some application we don't want to handle every time there is a byte in the buffer but rather wait til there is a number of byte in the buffer to extract. Then having an event for each of the byte would be useful.

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

shadowncs — Thu, 14 Dec 2017 20:01:28 GMT

Thanks. I'll take a closer look at SDK12 implementation. I didn't like that SDK12 calls the application uart callback for every byte received. That is IMHO not needed so will also look if that can be improved to only call the callback if the FIFO is empty, on first received byte.

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

Hung Bui — Thu, 14 Dec 2017 15:27:34 GMT

Thanks for the explanation. I agree on the part that we should read the byte out of the buffer before we start RX again. This was the change in the newer SDK as I mentioned above. In the newer SDK we also do not read the number of bytes in the buffer inside app_uart_get() but we use a global variable that being set inside uart_event_handler(NRF_DRV_UART_EVT_RX_DONE) and the fifo is full (m_rx_ovf).

Doing this we avoid scheduling RX twice when we have only one slot as you described.

However we may still have a race condition situation when m_rx_ovf is not set when we read it, but then it 's set when we are getting the byte from fifo out. This will result in that we don't schedule RX when we should. But, this can be solved if we keep calling app_uart_get() until the buffer is empty.

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

shadowncs — Thu, 14 Dec 2017 00:13:34 GMT

Bui, sorry if what follows is not super-clear as I said it took me a few days to go thru this.

I have source and examples to demonstrate each point if that helps.

First, the condition to start a read is wrong in app_uart_get. I will prove with an example: Let's have a FIFO with N-2 elements. Another character arrives and uart_event_handler(NRF_DRV_UART_EVT_RX_DONE) gets called. Since it adds one element, FIFO now has N-1 elements, the ISR calls nrf_drv_uart_rx.

app_uart_get gets called, FIFO_LENGTH(m_rx_fifo) == m_rx_fifo.buf_size_mask is true so app_uart_get also calls nrf_drv_uart_rx. Now, if the timing of next byte is just right, we will receive 2 characters and error out with APP_UART_FIFO_ERROR. This is pretty easy to reproduce with a short 32B RX queue and say 100 chars pasted at once into the serial terminal.

I tried fixing app_uart_get by only starting a new RX when FIFO_LENGTH(m_rx_fifo) > m_rx_fifo.buf_size_mask but of course then we're scheduling a RX before reading the full FIFO, potentially overflowing it before we get to read from it. There is no getting around the fact: we first must read the FIFO before we schedule RX.

So I modified app_uart_get to read FIFO first and then check for length and schedule RX if length is N-1 (we just read the last element).

However this does not fix the issue, as next example proves:

Having a FIFO with N-1 elements. RX is active waiting for the next byte. app_uart_get gets called, dequeues 1B (we're at N-2 bytes in FIFO) and execution is interrupted by ISR delivering the N-1'th byte. ISR also schedules the next RX. Execution is resumed however now FIFO_LENGTH = N-1 which makes it look like we just read the last element and app_uart_get schedules RX. Therefore we again have 2 RX'es scheduled and we may receive 2 bytes, errorin out on the FIFO overflow.

Proposed solution is to make fifo read-and-test-if-it-was-full an atomic operation.

Since reading now involves a critical section, I just made a get array for ehnhanced performance.

PS this html editor is horrible - no newlines!

RE: nRF51422 SDK11 app_uart_fifo race condition and fix

Hung Bui — Wed, 13 Dec 2017 15:01:54 GMT

Hi shadow,

Could you elaborate a little bit more on how you detect the error ? Please correct me if I'm wrong but in app_uart_get() we basically pop one byte out from the fifo and in uart_event_handler(NRF_DRV_UART_EVT_RX_DONE) we add one byte into the fifo. So even if the event handler interrupt the app_uart_get() it shouldn't cause a byte missing or duplicating bytes ? Not 100% sure there is a race condition here.

One thing we tried to fix in the later SDK is to pop the byte out before checking if it's full or not as we may end up receiving byte before we get the byte out at the end of the function.