Serial DFU Flash erase stuck

RE: Serial DFU Flash erase stuck

Harry — Thu, 25 Jun 2020 22:28:37 GMT

Thanks Vidar. Btw, I'm running it on a custom board, not dev kit.

Also, I found that it's been running fine from debugger. So, my breakpoints are pointless. Now I have to go back to debugging using debug messages.

RE: Serial DFU Flash erase stuck

Vidar Berg — Thu, 25 Jun 2020 08:03:48 GMT

I'm glad to hear that you found it helpful, thanks. I can try to debug this on my side as well if the project can be run on a Nordic dev kit without external dependencies.

RE: Serial DFU Flash erase stuck

Harry — Wed, 24 Jun 2020 23:28:19 GMT

Hi Vidar,

Thank you so much for all your help. I'll mark your last reply as the answer. I hope I'll find the root cause soon.

RE: Serial DFU Flash erase stuck

Vidar Berg — Wed, 24 Jun 2020 13:58:10 GMT

Hi,

[quote user="Serene_Guy"]I am not quite sure regarding the difference. Btw, I run it using the debugger, on both register readouts. I captured the screenshot first actually, from Keil IDE. And then I used nrfjprog.[/quote]

There is no valid application to boot according to the RTT log, which is why I'm not expecting the CPU to execute code from the application region as the register readout shows. The registers must be read while the bootloader is stuck. Any unintentional reset before doing the readout will likely be enough to recover the device back into a normal state. Anyway, I think we can move on from this troubleshooting step now as you have managed to resolve the issue with the breakpoints.

[quote user="Serene_Guy"]Thanks for your suggestion, I added HardFault library, now I saw it stopped at the Bootloader, not in MBR anymore. Please see the register readout below:[/quote]

To be sure the correct hard fault handler is included, please make sure hardfault_handler_keil.c and hardfault_implementation.c are both included in the build and also enabled in sdk_config ( the handler should not appear greyed out in the Keil editor). Once this is done you can place a breakpoint at the end of HardFault_c_handler() and see if you can replicate the problem where the bootloader gets stuck after a flash erase. I'm still suspecting it's a hard fault. I hope is that the HardFault_c_handler() will log more information about what's triggering the fault exception.

[quote user="Serene_Guy"]Are you saying if it's pre-compiled, then I won't be able to set breakpoint?[/quote]

I haven't tried it myself, but I think it should be possible. I guess one potential problem is that a pre-compile library may use different source paths, though it should be possible to edit those in Keil.

RE: Serial DFU Flash erase stuck

Harry — Wed, 24 Jun 2020 04:58:13 GMT

Hi Vidar,

[quote userid="4240" url="~/f/nordic-q-a/62855/serial-dfu-flash-erase-stuck/256370"]Maybe the device has gone into a boot loop, and the program just happened to be inside the MBR when you ran --readregs? Or does the device seem to recover if you perform a reset after getting stuck?[/quote]

I am not quite sure regarding the difference. Btw, I run it using the debugger, on both register readouts. I captured the screenshot first actually, from Keil IDE. And then I used nrfjprog.

[quote userid="4240" url="~/f/nordic-q-a/62855/serial-dfu-flash-erase-stuck/256370"]I'm also wondering if you have defined your own Hardfault handler in your bootloader? The weakly declared hardfault in arm_startup_nrf52840.s will make the program hang until it's reset. Either way, it can be very helpful to use the SDK's HardFault handling library to find out what triggered the exception.[/quote]

Thanks for your suggestion, I added HardFault library, now I saw it stopped at the Bootloader, not in MBR anymore. Please see the register readout below:

R0:   0x00008DF4
R1:   0x40002000
R2:   0x00000008
R3:   0x2000A85A
R4:   0x00000001
R5:   0x000003E8
R6:   0x0000FA00
R7:   0x000E3FF1
R8:   0x00000000
R9:   0x00000000
R10:  0x000E4D5C
R11:  0x00000000
R12:  0x80000000
SP:   0x2002B7E0
LR:   0x000DC3EF
PC:   0x000E3FF0
xPSR: 0x21000000
MSP:  0x2002B7E0
PSP:  0x00000000

[quote userid="4240" url="~/f/nordic-q-a/62855/serial-dfu-flash-erase-stuck/256370"]Does the main project enable debug information for the output file as shown below? This is one thing that could explain why you're unable to place breakpoints[/quote]

Yes, I already checked that. Btw, I found that after merging the Keil uVision project (from the Bootloader example and mine), I can set breakpoints now. The other problem I have is, the error is different when I run it from debugger vs. freerun. I still need to rely on debug messages to find the root cause.

[quote userid="4240" url="~/f/nordic-q-a/62855/serial-dfu-flash-erase-stuck/256370"]So just for my understanding, and please correct me if I'm wrong, but it sounds like in your case that the bootloader components are being pre-compiled into a library and then imported to another existing Keil project?[/quote]

You are absolutely correct. I have a thin layer of application (handshaking) and then it calls the Bootloader SDK. SO, the Bootloader SDK is part of the project.

Regarding pre-compiled, I don't think so. I inherited this project from another team member, and as far as I can tell, the make file includes the source code (*.c files) as well as the library path, and refer to those libraries as *.ext files. Does it mean it's pre-compiled? My understanding is if it's pre-compiled, then they will be in *.a files format. (I'm not super familiar with Nordic yet).

Are you saying if it's pre-compiled, then I won't be able to set breakpoint?

P.S.: The MBR I used is version 2.4.1. The SDK is 15.3.0.

RE: Serial DFU Flash erase stuck

Vidar Berg — Tue, 23 Jun 2020 09:58:26 GMT

Hi,

Thanks for the updates. The first register readout shows the program being in the Hardfault interrupt context somewhere inside the MBR. So something is causing a fault exception to occur during DFU. Although I'm a bit surprised to see that the program counter (PC) is inside the MBRs memory range as all interrupts shall be forwarded to the bootlaoder. Maybe the device has gone into a boot loop, and the program just happened to be inside the MBR when you ran --readregs? Or does the device seem to recover if you perform a reset after getting stuck?

I'm also wondering if you have defined your own Hardfault handler in your bootloader? The weakly declared hardfault in arm_startup_nrf52840.s will make the program hang until it's reset. Either way, it can be very helpful to use the SDK's HardFault handling library to find out what triggered the exception.

As for the second readout from Keil, it appears to be taken while the CPU executes code from the main application. It's not inside any interrupt context either. Could it be that it's different because Keil, by default, re-loads the app image resets the target when attaching the debugger?

[quote userid="90336" url="~/f/nordic-q-a/62855/serial-dfu-flash-erase-stuck/256273"]A separate update, regarding setting the breakpoints on Bootloader components from SDK, I added the reference to the libraries, I can put breakpoint but only before I start debugging. It seems to hit them, but they're not visible (no red dot - only yellow arrow). Please see the screenshot below from Keil uVision:[/quote]

Does the main project enable debug information for the output file as shown below? This is one thing that could explain why you're unable to place breakpoints.

The bootloader is usually built using the project files provided by the SDK. So just for my understanding, and please correct me if I'm wrong, but it sounds like in your case that the bootloader components are being pre-compiled into a library and then imported to another existing Keil project?

RE: Serial DFU Flash erase stuck

Harry — Mon, 22 Jun 2020 17:12:22 GMT

Hi Vidar, the IDE/toolchain that I used is Keil uVision.

Btw, a side note, I put the Bootloader at address 0xD6000, and I have MBR.

I noticed that some registers are different, comparing the values I got from nrfjprog and from Keil IDE. I run it via J-Link. Or perhaps do I need to run it without J-Link?

Please find below is the CPU register readout, as well as the screenshot from:

R0:   0x00000000
R1:   0x20002774
R2:   0x00000040
R3:   0x0000B635
R4:   0x200257D4
R5:   0x00000040
R6:   0x0000006E
R7:   0x00000000
R8:   0x20002774
R9:   0x00000000
R10:  0x200257D4
R11:  0x00000000
R12:  0x00000000
SP:   0x20025670
LR:   0xFFFFFFE9
PC:   0x000003CE
xPSR: 0x61000003
MSP:  0x20025670
PSP:  0x00000000

A separate update, regarding setting the breakpoints on Bootloader components from SDK, I added the reference to the libraries, I can put breakpoint but only before I start debugging. It seems to hit them, but they're not visible (no red dot - only yellow arrow). Please see the screenshot below from Keil uVision:

RE: Serial DFU Flash erase stuck

Vidar Berg — Mon, 22 Jun 2020 07:46:07 GMT

Hello,

This is not a known issue, but I suspect it may have something to do with the integration or implementation of the bootloader since it doesn't fail consistently every time. As a start, please run 'nrfjprog --readregs' to get a CPU register readout when it's stuck and post the result here. Or look up the PC value to find out where in the program it gets stuck.

[quote user=""]Background: Currently I am unable to put breakpoint in the Bootloader component that I imported from SDK 15.3.0, but it's a separate issue that I posted in a separate post. So, I can only do black-box debugging with debug messages for now.[/quote]

This is often a sign of missing debug symbols in the output executable (*.elf, *.out,..). Which IDE/toolchain are you using to build the bootloader?