NRF52840 issue getting from bootloader to application

[jspear]

Hello all,

Currently using an nRF52840-DK, GCC Makefile build tools including nrfjprog and nrfutil, Nordic SDK v15.0, SoftDevice s140 v6.0.0

I am currently trying to get a grasp on how the MBR, SoftDevice, DFU Bootloader, and Application all work together and I'm having a hard time combining some of the examples together cohesively. What I have done so far is the following:

• Installed the SoftDevice (which includes the MBR)
• Set the following debugging sdk_config.h elements for the bootloader example
  • NRF_LOG_BACKEND_UART_ENABLED to 1
  • NRF_LOG_ENABLED to 1
  • NRF_LOG_DEFERRED to 1
  • NRF_LOG_DEFAULT_LEVEL
• Installed a slightly customized open_bootloader (pca10056_usb_debug) example (I've simply added a few extra debug messages)  
• Installed the ble_app_blinky example application
• Installed bootloader_settings generated using the nrfutil program

I am having an issue where the device gets stuck somewhere in the bootloader process. Based on some DEBUG messages that I've placed, it appears that it is having an issue in the nrf_dfu_mbr_irq_forward_address_set() function, called by the nrf_bootloader_app_start(). I am unable to get any DEBUG messages come out of the nrf_dfu_mbr_irq_forward_address_set() function, but right before this function is called is where I continue to get my last DEBUG message. The device gets here whenever it turns on from a power off/reset, and I want it to get past here and go to the blinky application which it cannot.

On the DK, I've got it set to where holding down Button 4 and turning the device on brings the device into DFU mode, where it gets stuck in the loop_forever() function found in nrf_bootloader.c and then I am unable to get to the application from here as well. Setting the NRF_BL_DFU_INACTIVITY_TIMEOUT_MS to 10000 will eventually cause it to exit loop_forever() and revert to the previously mentioned case, but again, not getting me to the application.

So I have two scenarios in which I am trying to get to the application from the bootloader:

1. Whenever I turn the device on/restart via reset pin it gets stuck in the nrf_bootloader_app_start(), most likely the nrf_dfu_mbr_irq_forward_address_set() function.
2. Whenever the device enters the DFU, I get stuck in the loop_forever() function and can't get to the application.

This is becoming a pretty big headache, so I appreciate any and all advice on the matter.

[jspear]

Interesting piece of information... Originally I had the sdk_config.h values NRF_LOG_BACKEND_RTT_ENABLED set to 0 and the NRF_LOG_BACKEND_UART_ENABLED set to 1 which would allow me to see the output during the bootloader process. This was when I was getting the pause right before the nrf_dfu_mbr_irq_forward_address_set() function. Trying to test some things out, I turned off the NRF_LOG_BACKEND_UART_ENABLED and it now appears to be working as expected (when powered on/reset, it begins the blinky application, and when the Button 4 is held while reset it goes into DFU mode). Now I can't look at debug messages in the bootloader though.

This is good, however, I am apparently misunderstanding the proper way to look at debug messages in the bootloader. If someone can explain to me why having NRF_LOG_BACKEND_UART_ENABLED set to 1 gives the originally questioned behavior, that would help me feel more confident in this whole process and I'd be very thankful.

[Edvin]

Hello,

What bootloader project did you start with before you started modifying it? Can you please share the path to your makefile (relative from the top SDK folder). 

Did you try to set NRF_LOG_BACKEND_RTT_ENABLED instead of the UART? Does it behave the same, or does it work with RTT, only not with UART?

When you compile your bootloader. What does the bootloader .hex file look like? What I really want to know is how large it is. Try to drag-n-drop it into the nRF Connect for Desktop -> Programmer application, and hover your mouse over the  Bootloader section (on top). What addresses does it occupy? 

For comparison, this is what it looks like on the unmodified SDK17.1.0\examples\dfu\secure_bootloader\pca10056_uart\armgcc:

You can see that it occupies the addresses: 0x000F8000 - 0x000FD81F

What addresses does your occupy?

Best regards,

Edvin

[jspear]

Hi Edvin,

Thanks for getting back to me.

Edvin said:

Can you please share the path to your makefile (relative from the top SDK folder).

The path to the bootloader makefile project is:

<SDK_15_0_Root>/examples/dfu/open_bootloader/pca10056_usb_debug/armgcc/

Edvin said:

Did you try to set NRF_LOG_BACKEND_RTT_ENABLED instead of the UART? Does it behave the same, or does it work with RTT, only not with UART?

I have tried this and I am able to get output via the JLinkRTTViewer tool without any breaking like I saw when I was using the UART. I am running into a problem I think related to this post, but otherwise, RTT is working fine.

Edvin said:

When you compile your bootloader. What does the bootloader .hex file look like?

Here is a corresponding image of when I drag the bootloader.hex file into the programmer tool:

This is not at all what I would have expected. The bootloader application is split into two regions:

Lower Region: 0x000E0000 - 0x000EDEAF 

Upper Region: 0x10001014 - 0x1000101B

I usually try to perform all of my uploading using the nrfjprog tool, so I have not looked at this screen yet, nor have I explicitly looked at any memory regions in any files before. This seems a little concerning in comparison to the screenshot which you posted. What would have caused this large discrepancy?

To be honest, I am very new to the idea of memory region manipulation and linker scripts, so if you have any good resources on where to learn this, that would be very helpful!

Thanks again for your response!

[Edvin]

jspear said:

I usually try to perform all of my uploading using the nrfjprog tool, so I have not looked at this screen yet,

Me too, but it is a simple way to see the application size. I just wanted to check that there was enough pages that were free above the bootloader. You need two pages (0x000FE000 and 0x000FF000), and that is fine in this case. I was not sure whether you used a _debug project or a non- _debug project. The difference is that the _debug projects are moved a bit down, to give space to the logging module. 

Since the bootloader is close to the "top" of the flash, it needs to "grow downwards", i.e. move the start address as the bootloader becomes bigger, unlike a normal application that starts at the bottom, and can grow quite a lot upwards before it starts being an issue. 

That is all I wanted to see. the memory regions are not the issue here.

Further investigation:

Are you using a custom board or a DK to reproduce this issue?

I see you are using SDK15.0.0. I don't think there are any known issues on that version.

Can you please show me how you generate your bootloader settings, and how you generate and upload your dfu package, if you get that far?

Is there some way for me to reproduce what you are seeing on an nRF52840DK? Is it possible to zip a project folder that I can unzip in an unmodified SDK15.0.0 to see the same behavior that you see?

Best regards,

Edvin

[jspear]

Edvin said:

Since the bootloader is close to the "top" of the flash, it needs to "grow downwards", i.e. move the start address as the bootloader becomes bigger

Is this typically something that is taken into account with each projects Makefile and Linker Script?

Edvin said:

Are you using a custom board or a DK to reproduce this issue?

I am currently using the DK at the moment. We are going to be moving this over to a custom board eventually, but wanted to start simple with the DK.

Edvin said:

Can you please show me how you generate your bootloader settings, and how you generate and upload your dfu package, if you get that far?

To generate my bootloader settings, I use the following command:

nrfutil nrf5sdk-tools settings generate --family NRF52840 --application _build/nrf52840_xxaa.hex --bootloader-version 0 --bl-settings-version 1 --application-version 3 bootloader_settings.hex

This is the BLE blinky application from the <SDK_15_0_Root>/examples/ble_peripheral/ble_app_blinky/pca10056/s140/armgcc/ project directory. 

In terms of the dfu package, I am still working on getting kinks worked out with this, as I have an issue open here. I'm having problems uploading .zip examples via the nrfutil dfu procedure, so I haven't actually been able to test that feature yet unfortunately (though it is of high importance that I get it working soon). 

Edvin said:

Is there some way for me to reproduce what you are seeing on an nRF52840DK?

I can definitely send over a .zip file. What files would be most helpful? Do you want my entire bootloader project from <SDK_15_0_Root>/examples/dfu/open_bootloader/ which includes the main.c file, or do you just want the .hex file located in the _build directory? Same question for the BLE blinky app from <SDK_15_0_Root>/examples/ble_peripheral/ble_app_blinky/ .

Joseph

[Edvin]

jspear said:

Is this typically something that is taken into account with each projects Makefile and Linker Script?

All linker scripts (if you use armgcc. In an IDE it is usually a project setting) will set the start address for that application. But please note that this is only something you need to consider if it is a bootloader project or if you add/remove the softdevice from an application.

jspear said:

To generate my bootloader settings, I use the following command:

Why do you have the option nrf5sdk-tools? Is this a custom version of nrfutil? that is not something I have seen before. What does it print if you use the command "nrfutil version"

jspear said:

In terms of the dfu package, I am still working on getting kinks worked out with this, as I have an issue open here.

I see. I see that in your other ticket you are using the UART bootloader with the blinky_mbr example, which is fine. Just make sure that you use the correct USB ports on the DK. The USB port on the short end is for the UART (through the debugger), while the USB port on the long side of the DK is for the nRF's USB port. 

For a flying start, you can paste this file into the SDK15.0.0\examples\dfu\secure_bootloader\pca10056_usb_debug\armgcc folder:

Fullscreen 3821.test.bat Download

make -j9
make -j9 -C ..\..\..\..\ble_peripheral\ble_app_buttonless_dfu\pca10056\s140\armgcc

rm -rf files
mkdir files

copy ..\..\..\..\ble_peripheral\ble_app_buttonless_dfu\pca10056\s140\armgcc\_build\nrf52840_xxaa.hex files\app.hex
nrfutil settings generate --family NRF52840 --application files\app.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 files\settings.hex
nrfutil pkg generate --application files\app.hex --application-version 2 --hw-version 52 --sd-req 0x0100 --key-file ..\..\..\private.key files\dfu_img.zip

nrfjprog -e
nrfjprog --program ..\..\..\..\..\components\softdevice\s140\hex\s140_nrf52_7.2.0_softdevice.hex --verify
nrfjprog --program _build\nrf52840_xxaa_s140.hex --verify
nrfjprog --program files\app.hex --verify
nrfjprog --program files\settings.hex --verify
nrfjprog --reset

It looks like you are using Linux or MacOS, so you have to convert it to an sh script, but you probably know how to do that. You also need to change the COM port to match the USB port of your device. Also, when you see the "timeout 10", this is where you need to reboot your DK while holding button 4 to put it in DFU mode. 

As to why the application is hanging in serial_tx() -> while (m_async_mode && (m_xfer_done == false)), I believe this is a mismatch between the log backend (UART) and the bootloader. 

The bootloader will go to: main() (which will log something) -> nrf_bootloader_init() -> nrf_bootloader_app_start() -> NRF_LOG_FLUSH(), but right before the NRF_LOG_FLUSH, it will forward all interrupts to the application, using nrf_dfu_mbr_irq_forward_address_set(). So when NRF_LOG_FLUSH is called, the UART will wait for the TX complete event (which will set m_xfer_done = true), but since this is forwarded, this will never happen. 

To work around this, you can either just keep using the RTT backend (which doesn't require the UART peripheral), or you can set NRF_LOG_DEFERRED 0 in sdk_config.h.
When NRF_LOG_DEFERRED is set to 1, all instances of NRF_LOG_...() will just queue the log, and the log will be processed at a later time. Typically in your main() loop. However, in the bootloader project, the log will not be processed before the NRF_LOG_FLUSH(), right before nrf_bootloader_app_start_final(), and at this point, the UART interrupts are forwarded to the application's start address (really the softdevice's/mbr's start address, which later will forward the to the application's start address), which is why you see it hanging waiting for m_xfer_done to be true.

I hope this will clear up things a bit.

Best regards,

Edvin

[jspear]

Edvin said:

Why do you have the option nrf5sdk-tools?

So I've followed this guide in order to install the latest nrfutil program (it was last updates only a few weeks ago). The very first step was the one that caused me to open up the other ticket I mentioned. 

I have two things I can output. When I do the command nrfutil -V I get: 

nrfutil 7.0.1 (19248e3 2022-11-01)
commit-hash: 19248e3210eb89a3e9f0c007cd4ba3080d3f392c
commit-date: 2022-11-01
host: x86_64-unknown-linux-gnu
build-timestamp: 2022-11-01T07:41:25.668667032+00:00

and when I do nrfutil nrf5sdk-tools -V I get: 

nrfutil-nrf5sdk-tools 1.0.1 (c86ddc2 2022-11-29)
commit-hash: c86ddc246ce42f669e82748f7eb2542caf0bf0fd
commit-date: 2022-11-29
host: x86_64-unknown-linux-gnu
build-timestamp: 2022-11-29T09:27:03.011122071+00:00
inner-executable-version: nrfutil version 6.1.7

Edvin said:

The USB port on the short end is for the UART (through the debugger), while the USB port on the long side of the DK is for the nRF's USB port.

I actually did not know this, so this is going to be very helpful in preparation for our custom board, thank you!

Edvin said:

For a flying start, you can paste this file

Thanks for this script, I was planning on making one myself, so it's nice to have one that was made by an expert. Gives me more confidence on this part of the building. 

Edvin said:

To work around this, you can either just keep using the RTT backend (which doesn't require the UART peripheral), or you can set NRF_LOG_DEFERRED 0 in sdk_config.h.

I suppose I should ask then, which debugging methodology is better, RTT or UART? It seems that there are less issues coming from the RTT implementation, so it appears that this would be the better choice for debugging. If it's a subjective answer, what would be your recommendation? 

Edvin said:

I hope this will clear up things a bit.

While the details behind this seem to be fairly complex, I believe I understand the general procedure going on and therefore the issue/solution, so thanks for the clarity!

Joseph

[Edvin]

jspear said:

So I've followed this guide in order to install the latest nrfutil program

Ah, thanks for sharing. I was not aware of this. I just tried uninstalling and re-installing using "pip uninstall nrfutil" and "pip install nrfutil", and it didn't allow me to use the command "nrfutil nrf5sdk-tools ...". I will look into this later.

jspear said:

I suppose I should ask then, which debugging methodology is better, RTT or UART? It seems that there are less issues coming from the RTT implementation, so it appears that this would be the better choice for debugging. If it's a subjective answer, what would be your recommendation? 

It is indeed very much a subjective answer, and both have their pros and cons.

I actually prefer UART as long as I don't need to output too much data too fast, because UART is slower, but I can open a UART terminal, and it will remain connected as long as the DK is powered, because the UART terminal is actually connecting to the debugger, and not the nRF chip itself (on the DK at least). RTT is a bit more flaky, because I usually need to reconnect it every time after flashing. At least if I am using Segger RTT Viewer. If I am using Segger Embedded Studio, however, it will do so automatically, so if you are using SES, the RTT backend is very (!) neat.

Again, it depends on the situation. The UART backend actually requires some resources on the nRF. It requires a UART instance, so if you need to use your UART for something else, it is a blocker. The RTT is not a peripheral on the nRF in the same way. It is just the debugger reading some memory areas on the nRF, which doesn't affect the performance of the chip. It does require a debugger, though, which the UART strictly doesn't (if you have an external UART -> USB bridge). 

So my go to is:

1: If I use Segger Embedded Studio -> RTT.

2: If I don't need too much output, my application doesn't use UART, and the nRF CPU is not throttled -> UART

3: RTT.

In addition, when you are working on a bootloader, remember that the pin configuration is carried over from the bootloader to the application, so when using GPIOs for UART or buttons, those will keep their configuration (input/output) in your application, unless you actively disable it before entering the application. Worth knowing for current consumption. This is something to keep in mind for UART, but not RTT, as it doesn't use any GPIOs.

jspear said:

While the details behind this seem to be fairly complex, I believe I understand the general procedure going on and therefore the issue/solution, so thanks for the clarity!

Glad to help. Let me know if anything other nrfutil or bootloader questions pops up. (At least in the nRF5 SDK. I am still wrapping my head around the subject in the nRF Connect SDK).

Best regards,

Edvin