SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 03 May 2019 11:48:42 GMT

The NRF_UARTE -> ERRORSCR register indicates OVERRUN, FRAMING and BREAK error when going into error, so I'm having trouble determining the orignal issue that provoked it.

Ok well if there is no other way with the SDK 15.3 bootloader I'll do that then.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Vidar Berg — Fri, 03 May 2019 10:44:31 GMT

Have you tried to check what the error source is in NRF_UARTE->ERRORSCR? I think it's a bit strange that it happens when sending the CRC since you use flow control.

Yes, you don't want any padding. Maybe you can re-use some of the buffer handling used in SDK 12.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 03 May 2019 09:55:33 GMT

I am using flow control yes. The GSM chip waits for the CRC reception and validation before proceeding with the next step (execute command) also yes, that is how I first detected this issue.

Ok I will try that and update the results, though I am not sure how to implement this without a zero-padding like solution which would mess with the data.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Vidar Berg — Fri, 03 May 2019 08:17:36 GMT

[quote userid="77752" url="~/f/nordic-q-a/45627/sdk-15-3-secure-serial-uart-bootloader-does-not-handle-nrf_dfu_op_object_write-data/185066"]Did it work on your side with RX_BUF_SIZE = 128 ? Because in this case I have a NRF_DRV_UART_EVT_ERROR when trying to send CRC via UART after the reception of the first 4096 bytes of data.[/quote]

Yes, 128 worked for me. I'm using flow control, are you? When waiting for the CRC, does the GSM controller wait for the CRC and validate it before proceeding with the execute command? NRF_DRV_UART_EVT_ERROR indicate that you received invalid data (framing error,etc)

[quote userid="77752" url="~/f/nordic-q-a/45627/sdk-15-3-secure-serial-uart-bootloader-does-not-handle-nrf_dfu_op_object_write-data/185066"]Is there any way to modify the write function so it can work with non-multiple of 4 values for RX_BUF_SIZE and so data packets size (such as 127) ? It was the case in the SDK 12.1 example bootloader.[/quote]

Not the write function itself. The length must be a multiple of 4. But you could create a new buffer for immediate storage of the data, then make sure the data length passed to nrf_fstorage_write is always a multiple of 4.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 03 May 2019 07:55:58 GMT

Hi,

Thank you for your answer, I did find out about RX_BUF_SIZE having to be a multiple of 4 (UPDATE of my previous post), and then I also found this PARAM_CHECK line you are talking about, and I was surprised that it does not do anything when RX_BUF_SIZE = 127, because as you said it should have failed there...

I have two questions from there:

- Did it work on your side with RX_BUF_SIZE = 128 ? Because in this case I have a NRF_DRV_UART_EVT_ERROR when trying to send CRC via UART after the reception of the first 4096 bytes of data.

- Is there any way to modify the write function so it can work with non-multiple of 4 values for RX_BUF_SIZE and so data packets size (such as 127) ? It was the case in the SDK 12.1 example bootloader.

Thank you !

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Vidar Berg — Fri, 03 May 2019 07:20:58 GMT

Hi,

I have tested Softdevice and app update with different MTU sizes but not been able to replicate the hardfault:

Setup:

- nrfutil 5.2.0

- sdk 15.3.0

- UART Serial bootloader - debug version

I did, however, observe that the RX_BUF_SIZE is passed as length to nrf_fstorage_write() so the size must be a multiple of 4, the NRF_FSTORAGE_PARAM_CHECK(!(len % p_fs->p_flash_info->program_unit), NRF_ERROR_INVALID_LENGTH); check will fail otherwise. Didn't this check fail on your side when you set the RX_BUF_SIZE to 127?

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Thu, 25 Apr 2019 11:45:18 GMT

Hi,

All right thank you, still working on it on my side also.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Vidar Berg — Thu, 25 Apr 2019 11:30:55 GMT

Hi,

Thanks for the update. This stack frame looks more correct. I will try to replicate then debug this on my side to see if I can find the root cause. I'll keep you posted.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Wed, 24 Apr 2019 08:02:01 GMT

Hi Vidar, thank you for your help.

No progress in the meantime no, I used 100 bytes data packet to continue my project, but it can't be a final solution...

By default, a delay of 100 ms is present between every data packet sent by the GSM chip to the NRF52, and I tried increasing this delay (up to 1000ms), but it doesn't change anything.

Here are the results shown by the SDK 15.2 Hardfault Handler (I am still going into HardFault when handling, so writing, the second 127 bytes data packet received whith RX_BUF_SIZE = 127):

<error> hardfault: HARD FAULT at 0x00076F90
<error> hardfault:   R0:  0x0263DB00  R1:  0x4001E504  R2:  0x0000001F  R3:  0x00000001
<error> hardfault:   R12: 0x0007AD89  LR:  0x0007ADB9  PSR: 0x81000000
<error> hardfault: Cause: Data bus error (PC value stacked for the exception return points to the instruction that caused the fault).
<error> hardfault: Bus Fault Address: 0x0002607F

The PC value points to this line in the code:

The LR value points to this line in the code:

And BFAR value, which I'm guessing represents the address where the value should be written:

More precisely, the HardFault PC points to the "STR" assembly instruction, that is supposed to store (at the address r5+(r4<<2)) the data and that causes the fault here:

I checked the assembly code for the same C instruction in my SDK 12.1 Serial Bootloader, that works with 127 bytes packets, and I see no difference in the assembly code, except for the registers used:

UPDATE:

After running some tests, it seems that this issue may have to do with the size of sent data packets, in bytes, being a multiple of 4. It actually works for 124 bytes data packets with RX_BUF_SIZE >= 124. It also works with 128 bytes data packets with RX_BUF_SIZE = 128, but for this precise case I get a NRF_DRV_UART_EVT_ERROR when trying to send CRC via UART after the reception of the first 4096 bytes of data.

Any reason for a non divisible per 4 size not working ? Could it come from the fact that the packet length is at some point divided by 4 to equal the number of words ?

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Vidar Berg — Wed, 24 Apr 2019 05:59:24 GMT

Hi,

Hung asked me to follow up on this case as he's out of office this week. There has been a while since the last update, have you had any progress in the meantime?

As a test, I think it would be interesting if you could try adding a delay between every packet on the controller side and see if that makes a difference (w 127 byte MTU). Also, the stack frame from the hardfault handler appears to be incorrect. The LR value is not pointing to a valid address but is a typical value for xPSR. It might be worth trying the hardfault handler included in the SDK, I'm not familiar with the ARM version, but this implementation will detect stack overflow as well.

Do the following to use the SDK Hardfault handler:

Add "\nRF5_SDK_15.2.0_9412b96\components\libraries\hardfault\nrf52\handler\hardfault_handler_keil.c" and "\nRF5_SDK_15.2.0_9412b96\components\libraries\hardfault\hardfault_implementation.c" to the project and build with HARDFAULT_HANDLER_ENABLED=1

Best regards,

Vidar

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 12 Apr 2019 08:44:56 GMT

Hi,

I agree, hence my update. Serial bootloader + MBR gives me the same hard fault at the same point (second data packet received).

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Fri, 12 Apr 2019 08:38:13 GMT

Hi Pierre,

Shouldn't we focus on solving the 127 bytes byte issue first ? Could you try to test just updating the APP without the SD ?

I'm don't think we support updating SD+APP from a non-softdevice bootloader. You can merge the softdevice and app and then updating it as an app. But we can solve this after we solve the 127 bytes packet issue.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 12 Apr 2019 08:02:01 GMT

I tried to do that, problem is I will have an error NRF_DFU_EXT_ERROR_SD_VERSION_FAILURE as the package generated that contains APP+SD (6.1.0) needs a --sd-req 0xXX parameter, even though there will be no SoftDevice programmed on my flash, only the MBR. So when comparing the required SD version and the one (not) installed, it throws this error.

Is there a way to do a APP+SD DFU with only bootloader and MBR programmed ?

UPDATE:

It does work if my package only contains SD (6.1.0). So I programmed bootloader settings and MBR hex files on the flash memory using nrfjprog, than launched debug of the bootloader, and I still have the exact same Hard Fault with bootloader + MBR only.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Thu, 11 Apr 2019 15:27:04 GMT

That hardfault is more likely an softdevice assert.

Please try to test using the stock serial bootloader + MBR not serial bootloader + softdevice.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Thu, 11 Apr 2019 09:14:13 GMT

Ok yes thank you debugging works with the SDK 15.2 bootloader.

So with RX_BUF_SIZE = 127 and the GSM whip sending 127 bytes data packet (so the configuration for which I have this issue), the code get stuck in an infinite loop in an hardfault_handler in the startup file arm_startup_nrf52.s :

I'll try to obtain some more information as to the source that provokes this hardfault.

UPDATE: I used a code given by ARM in order to get the registers that provoked this HardFault:

#include "RTE_Components.h"             // Component selection
#include CMSIS_device_header            // Include device header file from
#include "stdio.h"                      // include stdio library for printf output
#include "project_configuration.h"      // Header file for test case configuration

#if HARDFAULT_HANDLER == 1

void HardFault_Handler_C(unsigned long * svc_args, unsigned int lr_value);
 
// HardFault handler wrapper in assembly language.
// It extracts the location of stack frame and passes it to the handler written
// in C as a pointer. We also extract the LR value as second parameter.
__asm void HardFault_Handler(void)
{
	TST    LR, #4
	ITE    EQ
	MRSEQ  R0, MSP
	MRSNE  R0, PSP
	MOV    R1, LR
	B      __cpp(HardFault_Handler_C)
}	
 
 
// HardFault handler in C, with stack frame location and LR value extracted
// from the assembly wrapper as input parameters
void HardFault_Handler_C(unsigned long * hardfault_args, unsigned int lr_value)
{
  unsigned long stacked_r0;
  unsigned long stacked_r1;
  unsigned long stacked_r2;
  unsigned long stacked_r3;
  unsigned long stacked_r12;
  unsigned long stacked_lr;
  unsigned long stacked_pc;
  unsigned long stacked_psr;
  unsigned long cfsr;
  unsigned long bus_fault_address;
  unsigned long memmanage_fault_address;
  
  bus_fault_address       = SCB->BFAR;
  memmanage_fault_address = SCB->MMFAR;
  cfsr                    = SCB->CFSR;
 
  stacked_r0  = ((unsigned long) hardfault_args[0]);
  stacked_r1  = ((unsigned long) hardfault_args[1]);
  stacked_r2  = ((unsigned long) hardfault_args[2]);
  stacked_r3  = ((unsigned long) hardfault_args[3]);
  stacked_r12 = ((unsigned long) hardfault_args[4]);
  stacked_lr  = ((unsigned long) hardfault_args[5]);
  stacked_pc  = ((unsigned long) hardfault_args[6]);
  stacked_psr = ((unsigned long) hardfault_args[7]);
 
  printf ("[HardFault]\n");
  printf ("- Stack frame:\n"); 
  printf (" R0  = %x\n", stacked_r0);
  printf (" R1  = %x\n", stacked_r1);
  printf (" R2  = %x\n", stacked_r2);
  printf (" R3  = %x\n", stacked_r3);
  printf (" R12 = %x\n", stacked_r12);
  printf (" LR  = %x\n", stacked_lr);
  printf (" PC  = %x\n", stacked_pc);
  printf (" PSR = %x\n", stacked_psr);
  printf ("- FSR/FAR:\n");  
  printf (" CFSR = %x\n", cfsr);
  printf (" HFSR = %x\n", SCB->HFSR);
  printf (" DFSR = %x\n", SCB->DFSR);
  printf (" AFSR = %x\n", SCB->AFSR);
	if (cfsr & 0x0080) printf (" MMFAR = %x\n", memmanage_fault_address);
	if (cfsr & 0x8000) printf (" BFAR = %x\n", bus_fault_address);
  printf ("- Misc\n"); 
  printf (" LR/EXC_RETURN= %x\n", lr_value);
 	
  while(1); // endless loop
}
#endif

Results:

SCB -> CFSR = 0x00008200 ("cfsr")

PC = 0x00070240 ("stacked_pc")

SCB -> BFAR = 0x00074031 ("bus_fault_address")

So apparently this Hard Fault is forced and actually comes from a Bus Fault, generated at the address BFAR and where at this point PC was 0x0007024.

This is precisely the moment the code enters write() function from nrf_fstorage_nvmc.c file:

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Thu, 11 Apr 2019 08:51:02 GMT

The CPU has to be stalled at some place , right ?

If you want to debug I would suggest to start with SDK v15.2 instead of 15.3. In v15.3 we changed the location of the bootloader start address making it slightly harder to re-flash the bootloader. Have a look here.

To be able to debug you would need to make sure the optimization level is set to 0. If the size of the flash dedicated for bootloader is not big enough you would need to move the bootloader down in the flash address to make more space for it.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Tue, 09 Apr 2019 15:01:46 GMT

Thank you for your answer, I tried debug mode indeed in order to see if and where the CPU would be stuck, but for some reason I can't seem to be able to go into debug mode. I created a ticket for this issue (https://devzone.nordicsemi.com/f/nordic-q-a/45910/debugging-serial-uart-bootloader-sdk-15-3).

What would be this timing problem you mentioned that could explain the maximum of a 100 bytes size for a data packet ?

I understand, thank you for your time ! I will also try to work on this debugger issue.

Though I do think that what my GSM chip is doing as DFU master is exactly the same thing that your Python script does, if you were to use it as DFU master.

UPDATE: after test, it does not seem that CPU is going to any of the app error handler function.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Tue, 09 Apr 2019 13:49:09 GMT

Yes an MBR is needed for the bootloader to do DFU update itself as well as to receive the interrupt vector forwarding.

If you don't initiate the softdevice (calling sd_mbr_command() with SD_MBR_COMMAND_INIT_SD) , it shouldn't be a problem to access the flash directly using NVMC.

Note that if you use an GPIO to detect if you can get to point 1 or point 2, you should also check if there is any assertion that interrupt the process. Please check if the CPU get to any of the app error handler function.

Have you tried to test using debug mode ? And the check where the CPU is at when it stuck ?

It's a little bit harder for us to reproduce the issue here to test what you described (due to that we need to customize the DFU master). We will try to find time to test but it's a bit busy here.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Mon, 08 Apr 2019 14:53:29 GMT

Well, SoftDevice 6.1.1 is programmed on the chip before downloading bootloader, because without it bootloader alone doesn't seem to do anything, but the purpose of this bootloader will be at some point to download and write into memory this SoftDevice 6.1.1 in order to replace an older one. So I guess it is supposed to be able to work without it, and the fact that it can not seem to for now is also an issue.

Plus my SDK 12.1 serial bootloader was using nrf_nvmc_write_words() also.

What would be this timing problem ?

UPDATE: Bootloader works without SoftDevice if I add MBR hex file to bootloader (nRF5_SDK_15.3.0_59ac345\components\softdevice\mbr\nrf52832\hex).

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Mon, 08 Apr 2019 14:35:25 GMT

Oh, I was not aware that you have the softdevice. Was the softdevice initialized when you call nrf_nvmc_write_words() ?

If the softdevice is initialized you would need to nrf_fstorage_sd.c instead of nrf_fstorage_nvmc.c. There could be a chance that the timing needed to write 64 bytes was OK with the softdevice but not 100 bytes.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 05 Apr 2019 15:22:55 GMT

With RX_BUF_SIZE = 127 and GSM chip sending 100 bytes data packets, m_slip.current_index
= 101 bytes, so 100 bytes payload + 1 command byte.

I agree printing out the log would be preferable but one of my issue previously stated is that I can not: the log is printed with quite an important delay, and as the MCU seems to crash after receiving the 2nd data packet, I can't access debug logs after a certain point.

I detect it because my custom board has a buzzer that I can use to know when the code reaches certain points, and I used it on point 1 and point 2. That is kow I know that as some point the code reaches point 1 but never point 2.

I found the issue is from nrf_nvmc_write_words () using the method previously stated.

Could it be a problem related to the presence of a SoftDevice with the bootloader ?

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Hung Bui — Fri, 05 Apr 2019 15:10:09 GMT

sizeof(m_slip.p_buffer) is only the size of the pointer, not the size of the array, you should find the size of the packet in m_slip.current_index.

How do you detect that it stuck in point 1 can can't get to point 2 ? I don't think it would stuck in wait_for_flash_ready(). It only checks the hardware register and we haven't seen any case the hardware (writing to flash) wouldn't work.

I would suggest to print out log instead of line stepping .

How do you find that the issue is from nrf_nvmc_write_words () ?

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Fri, 05 Apr 2019 08:12:53 GMT

UPDATE: With RX_BUF_SIZE = 127 and GSM chip sending 100 bytes data packets, on_rx_complete() pass to nrf_dfu_serial_on_packet_received() a length of 101 bytes (m_slip.current_index), and the length of m_slip.p_buffer is 4 (sizeof(m_slip.p_buffer)).

It also appears that the issue come specifically from nrf_nvmc_write_words() in nrf_nvmc.c file.

static inline void wait_for_flash_ready(void)
{
    while (NRF_NVMC->READY == NVMC_READY_READY_Busy) {}
}

...

void nrf_nvmc_write_words(uint32_t address, const uint32_t * src, uint32_t num_words)
{
    uint32_t i;

    // Enable write.
    NRF_NVMC->CONFIG = NVMC_CONFIG_WEN_Wen;
    __ISB();
    __DSB();
    
    //Point 1
    
    for (i = 0; i < num_words; i++)
    {
        ((uint32_t*)address)[i] = src[i];
        wait_for_flash_ready();
    }

    //Point 2
    
    NRF_NVMC->CONFIG = NVMC_CONFIG_WEN_Ren;
    __ISB();
    __DSB();
}

The first time nRF52 receives a 127 bytes data packet with WRITE command (line 30 of the GSM log), code gets to //Point 1 and to //Point 2 without any visible issue. But for the second packet (line 32 of the GSM log), code gets to //Point 1 but never to //Point2. It would seem that it is stuck waiting for flash to be ready this time.

Any possible reason for that ?

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Thu, 04 Apr 2019 15:31:52 GMT

So apparently, whatever the value of RX_BUF_SIZE (127 or higher), the max length in bytes of data packet I can send is between 100 and 110 per packet. It works with 100 bytes data packets, but not with 110 bytes data packets.

RE: SDK 15.3 Secure Serial UART Bootloader does not handle NRF_DFU_OP_OBJECT_WRITE (data)

Pierre B. — Thu, 04 Apr 2019 13:49:42 GMT

When RX_BUF_SIZE = 127 and the GSM chip sends 127 bytes data packets, just like in the log attached to my original post, the code goes into on_rx_complete() only for the first two chunks of data (lines 30 and 32 of the log), it doesn't anymore after that.

I can not have any more details as I can't have access to the nRF52 log from this part.