problems with transferring large datasets from computer to nRF over UARTE

[ahjr]

I am trying to use the nRF as an intermediary bootloader for another microcontroller. All that I need the nRF to do is accept a 64kB binary file from my computer (at the moment I do this over UART). Occasionally it works, but most of the time, I lose bytes, and I'm not entirely sure why.

Here is my UARTE0 initialization function:

void uarts_init(void) {
    // configure
    NRF_UARTE0->RXD.PTR = (uint32_t)app_vars.uart_buf_DK_RX;
    NRF_UARTE0->RXD.MAXCNT = UART_BUF_SIZE;
    NRF_UARTE0->TXD.PTR = (uint32_t)app_vars.uart_buf_DK_TX;
    NRF_UARTE0->TXD.MAXCNT = UART_BUF_SIZE;
    NRF_UARTE0->PSEL.TXD = 0x00000006; // 0x00000006==P0.6
    NRF_UARTE0->PSEL.RXD = 0x00000008; // 0x00000008==P0.8
    NRF_UARTE0->CONFIG = 0x00000000; // 0x00000000==no flow control, no parity bits, 1 stop bit
    NRF_UARTE0->BAUDRATE = 0x04000000; // 0x004EA000==19200 baud (actual rate: 19208)
    NRF_UARTE0->TASKS_STARTRX = 0x00000001; // 0x00000001==start RX state machine; read received byte from RXD register
    NRF_UARTE0->SHORTS = 0x00000020; // short end RX to start RX

    NRF_UARTE0->INTENSET = 0x00000010;
    NRF_UARTE0->ENABLE = 0x00000008; // 0x00000008==enable

    // enable interrupts
    NVIC_SetPriority(UARTE0_UART0_IRQn, 1);
    NVIC_ClearPendingIRQ(UARTE0_UART0_IRQn);
    NVIC_EnableIRQ(UARTE0_UART0_IRQn);
}

My UARTE0 interrupt handler is a bit complicated to look at, sorry about that. Fundamentally, all that it does is upon receiving a character, it puts that character into the next index of a buffer in memory. Then it has a simple state machine: upon receiving a "\n" the nRF checks if the buffer is a specific string (this works as expected every time). If it receives that string, the next 64kB that the nRF receives from the computer should go to a separate location in memory. That's really it (for now, eventually I would like to add other strings in the ISR but I'm stuck on this one). Here is the ISR:

void UARTE0_UART0_IRQHandler(void) {

    uint8_t uart_rx_byte;

    // debug
    app_dbg.num_ISR_UARTE0_UART0_IRQHandler++;

    if (NRF_UARTE0->EVENTS_ERROR == 0x00000001) {
        // clear the error and continue?
        NRF_UARTE0->EVENTS_ERROR = 0x00000000;
        NRF_UARTE0->TASKS_FLUSHRX = 0x00000001;
        NRF_UARTE0->TASKS_STARTRX = 0x00000001;
    }

    if (NRF_UARTE0->EVENTS_ENDRX == 0x00000001) {
        // byte received from computer

        // clear
        NRF_UARTE0->EVENTS_ENDRX = 0x00000000;

        // debug
        app_dbg.num_ISR_UARTE0_UART0_IRQHandler_ENDRX++;

        if(app_vars.programmer_state == PROGRAMMER_WAIT_4_CMD_ST) {
            uart_rx_byte = app_vars.uart_buf_DK_RX[0];
            app_vars.uart_RX_command_buf[app_vars.uart_RX_command_idx++] = uart_rx_byte;

            if((uart_rx_byte=='\n')||(uart_rx_byte=='\r')) { // \r for debugging w/ putty, \n for the python scripts
                app_vars.uart_RX_command_idx = 0; // reset index to receive the next command
                if(memcmp(app_vars.uart_RX_command_buf,UART_TRANSFERSRAM,sizeof(UART_TRANSFERSRAM))==0) { // enter transfer SRAM state
                    app_vars.programmer_state = PROGRAMMER_SRAM_LD_ST;
                    app_vars.uart_RX_command_idx = 0;
                    print_sram_started_msg();
                }
                // else - erroneous command, clear the buffer and reset to default state
                else {
                    memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
                    app_vars.uart_RX_command_idx = 0;
                    app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
                }
            }
            else if (app_vars.uart_RX_command_idx > MAX_COMMAND_LEN) { // max length exceeded w/out return character, reset buffer
                memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
                app_vars.uart_RX_command_idx = 0;
                app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
            }
        }
        else if (app_vars.programmer_state == PROGRAMMER_SRAM_LD_ST) {
            uart_rx_byte = app_vars.uart_buf_DK_RX[0];
            app_vars.instruction_memory[app_vars.uart_RX_command_idx++] = uart_rx_byte;
            if(app_vars.uart_RX_command_idx == MEM_SIZE) { // finished w/ the 64kB memory
                // after loading memory - reset state, index, and command buffer
                app_vars.programmer_state = PROGRAMMER_SRAM_LD_DONE;
                print_sram_done_msg();
                app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
                app_vars.uart_RX_command_idx = 0;
                memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
            }
        }
    }
}

I am running into quite a few problems, all of which involve some problem with the 64kB. First, if I use an nRF UART baud that is not exactly equal to the baud set in pyserial on the computer, the UARTE0 register indicates a buffer overflow error, which really shouldn't be happening; I think UART has a ~10% tolerance in clock rate error. This does not occur if I set the pyserial baud to exactly what the nRF wants (for instance, 19208 instead of 19200). Second, even when the baud is exact, sometimes I lose bytes... somewhere, regardless of baud. No UARTE errors are triggered (I put a breakpoint inside of an if(NRF_UARTE0->EVENTS_ERROR==1UL) statement that never fires). And when I look at the output from my computer with an FTDI chip and a logic analyzer, all of the bytes do indeed leave the computer. The third is, when I run in debug mode (F5 in Segger) it will periodically succeed and transfer the full 64kB. But, when I run continuously (ctrl+T, L), I always lose bytes. Not sure what this indicates, but it is worrisome.

It is also possible that my problem is on the pyserial side. Here is the relevant code snippet:

# Open COM port to nRF
nRF_ser = serial.Serial(
 port=nRF_port,
 baudrate=250000,
 parity=serial.PARITY_NONE,
 stopbits=serial.STOPBITS_ONE,
 bytesize=serial.EIGHTBITS)

# Open binary file from Keil
with open(binary_image, 'rb') as f:
 bindata = bytearray(f.read())

# Need to know how long the binary payload to pad to 64kB
code_length = len(bindata) - 1
pad_length = 65536 - code_length - 1

for i in range(pad_length):
 bindata.append(0)

# Transfer payload to nRF
nRF_ser.write(b'transfersram\n')
print(nRF_ser.read_until())
# Send the binary data over uart
nRF_ser.write(bindata)
# and wait for response that writing is complete
print(nRF_ser.read_until())

# Execute 3-wire bus bootloader on nRF
nRF_ser.write(b'boot3wb\n')

# Display confirmation message from nRF
print(nRF_ser.readline())

On my nRF 52840-DK, the crystal is on and HF clock is on.

I thought that it might be a hardware issue, so I changed out cables, computer, nRF board, and I had no luck. I would prefer not to use the suite of uarte_app functions because I would like the code to be as standalone as possible and because I tried it and I actually had the same problem (it's harder to debug because I don't understand the functions). One other thing I should note: I configured the nRF to run at 1.8V. That might be relevant, but I didn't notice any difference running at 2.4V or 3.3V. It fails just as often. If anyone has any further debugging suggestions or an alternative idea for loading 64kB into the nRF's RAM (UART is fairly slow) I would appreciate it.

Thanks!

[Susheel Nuguru]

Hi Ahjr,

Thanks for clear description. Will definitely save us time as it is faster for me to understand the problem. Few thoughts while I am reading your information.

• Not using flow control is ok as long as you have a solid error handling mechanism. It looks like all your ISR is doing at any error (EVENTS_ERROR) is that it is clearing that event and flushing RX. This flush will make you lose received bytes unless you have a mechanism to tell to the peer to retransmit those missed bytes. Also it would be interesting to see what error occurred (if any) before you flush out all bytes so that we might attempt to solve the source of this error.
• You are using UARTE (EasyDMA), so unless you have UART_BUF_SIZE set to 1, your handling of data in the ISR is wrong. When you get EVENTS_ENDRX, it means that you have received UART_BUF_SIZE of new data in the receive buffer (which you set to uart_buf_DK_RX). So you need to extract and process that many bytes for every EVENTS_ENDRX. So your ISR event processing could be something like below

void UARTE0_UART0_IRQHandler(void) {

    uint8_t uart_rx_byte;
    uint8_t count;

    // debug
    app_dbg.num_ISR_UARTE0_UART0_IRQHandler++;

    if (NRF_UARTE0->EVENTS_ERROR == 0x00000001) {
        // clear the error and continue? (Susheel: This below code means loss of data)
        NRF_UARTE0->EVENTS_ERROR = 0x00000000;
        NRF_UARTE0->TASKS_FLUSHRX = 0x00000001;
        NRF_UARTE0->TASKS_STARTRX = 0x00000001;
    }

    if (NRF_UARTE0->EVENTS_ENDRX == 0x00000001) {
        // byte received from computer

        // clear
        NRF_UARTE0->EVENTS_ENDRX = 0x00000000;

        // debug
        app_dbg.num_ISR_UARTE0_UART0_IRQHandler_ENDRX++;

        count = UART_BUF_SIZE;
        
        while(count-- != 0)
        {
        if(app_vars.programmer_state == PROGRAMMER_WAIT_4_CMD_ST) {
                uart_rx_byte = app_vars.uart_buf_DK_RX[0];
                app_vars.uart_RX_command_buf[app_vars.uart_RX_command_idx++] = uart_rx_byte;

                if((uart_rx_byte=='\n')||(uart_rx_byte=='\r')) { // \r for debugging w/ putty, \n for the python scripts
                    app_vars.uart_RX_command_idx = 0; // reset index to receive the next command
                    if(memcmp(app_vars.uart_RX_command_buf,UART_TRANSFERSRAM,sizeof(UART_TRANSFERSRAM))==0) { // enter transfer SRAM state
                        app_vars.programmer_state = PROGRAMMER_SRAM_LD_ST;
                        app_vars.uart_RX_command_idx = 0;
                        print_sram_started_msg();
                    }
                    // else - erroneous command, clear the buffer and reset to default state
                    else {
                        memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
                        app_vars.uart_RX_command_idx = 0;
                        app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
                    }
                }
                else if (app_vars.uart_RX_command_idx > MAX_COMMAND_LEN) { // max length exceeded w/out return character, reset buffer
                    memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
                    app_vars.uart_RX_command_idx = 0;
                    app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
                }
            }
            else if (app_vars.programmer_state == PROGRAMMER_SRAM_LD_ST) {
                uart_rx_byte = app_vars.uart_buf_DK_RX[0];
                app_vars.instruction_memory[app_vars.uart_RX_command_idx++] = uart_rx_byte;
                if(app_vars.uart_RX_command_idx == MEM_SIZE) { // finished w/ the 64kB memory
                    // after loading memory - reset state, index, and command buffer
                    app_vars.programmer_state = PROGRAMMER_SRAM_LD_DONE;
                    print_sram_done_msg();
                    app_vars.programmer_state = PROGRAMMER_WAIT_4_CMD_ST;
                    app_vars.uart_RX_command_idx = 0;
                    memset(app_vars.uart_RX_command_buf,0,sizeof(app_vars.uart_RX_command_buf));
                }
            }
        }
    }
}

[ahjr]

Hi Susheel,

Thanks for responding. UART_BUF_SIZE is 1 in my code. Sorry about the confusion, if it were anything else your addition would be necessary. For UARTE errors: when I used certain baud rates (like 19200) I was seeing a 1 in the error register. Once I changed the baud I no longer saw any errors, even when bytes were missing.

I tried to simplify the code to try to isolate the problem. I removed the UART-based state machine and simply defaulted to the "load 64kB into RAM" state. Once it receives 64kB, it resets the index to 0 and waits for 64kB new bytes. I tried it around 250 times and I missed zero bytes (CRC checked as well). It's a temporary fix. I will continue to investigate the UART-based state machine. I know that the nRF can reliably receive 64kB over UART, and I know that the state machine works, but when I combine the two it breaks in ways that make no sense.

I will also try flow control. I have fairly robust error handling with CRC, but flow control can't hurt.

[Susheel Nuguru]

seems you have a good control on this ahjr, Please do care to come back here to update me with your findings. I am confident that the UART should function correctly (especially when you are seeing no errors) . Could be a race condition somewhere in the logic.