Issue with Application loading after migration from SDK 14.2 to SDK 15.3

[Adam Gordon]

I am running custom firmware on a custom board using an nRF52832 chip. I have encountered an issue after migrating my project from SDK 14.2 to SDK 15.3. The issue is that my application is not starting and it occurs on two different occasions:

1. After successfully performing an OTA DFU using a secure ble bootloader, my application doesn't run. The DFU is performed successfully but after nrf_bootloader_app_start() is called, is does not load my application.
2. If my application is loaded and running on my device and the device is removed from power and then plugged back in, the application does not startup.

With that being said, if I merge the .hex files of the BL, BL settings, SD, and APP and flash them onto my device using nrfjprog, my application is loaded and runs properly. If I then remove my device from power and plug it back in, the application no longer starts. Loading the same bootloader and softdevice and then performing a DFU with my application does not successfully start my application either.

I tried both above scenarios with an example app from the SDK (blinky) which I modified to run on my custom board, and everything worked correctly. I was able to perform a DFU and have the application start, and if I power cycled the device, the app restarted.

This functionality used to work fine with my app when it was using SDK 14.2, it has only broken since migrating to SDK 15.3. Is there something that has changed that I need to be aware of?

Any help with the issue would be greatly appreciated.

[Hung Bui]

Hi Adam, 

You may want to re-compile the bootloader with optimization to level 0 (so you can debug) and step into the bootloader code to see why it doesn't jump to your application after a reset. I suspect your application may change (due to a flash operation ? ) causing CRC32 check doesn't match ?  

Do you have any special function in your application related to UICR ? flash ? 

I assume your application works fine if you only flash the softdevice and the application ? 

[Adam Gordon]

Hi Hung,

Thank you for getting back to me! Unfortunately it's not just after a reset that my application does not start, it is also after a DFU.

We have no special function related to UICR. We use FDS in our application, but that is it.

Do you mean from SES? If so then yes, our application works fine if flashed from Segger with just the application and softdevice. It does not work if just the softdevice and application are flashed using nrfjprog (and then the device reset).

However, what is odd, is that if we take all of the .hex files of the bootloader, bootloader settings, softdevice, and application and merge them using mergehex and then flash them onto the device using nrfjprog and reset, the application runs and works fine. Even if I flash the .hex files each individually everything runs fine.

But if I flash just the bootloader and softdevice and then successfully perform a DFU, the bootloader does not jump to my application code. Is there a reason why it would work when I flash all the components together and not when I perform a DFU?

[Adam Gordon]

Hi Hung,

Good point, I have my own set of public/private keys that I will use with the examples to test the flash_fds app. 

As for the BOOTLOADER_ADDRESS macro, unfortunately I’m currently not in the office, but I looked into it a lot yesterday so will try to respond from memory to see if we can resolve this issue today. 

Yes we do have that macro inside app_util.h. I believe MBR_BOOTLOADER_ADDR was 0xFF8, pointing to a value of 0xFFFFFFFF. flash_end_addr() ended up returning 0x80000 because one of the lines of code has something that resembled two _sz being multiplied together. 

I’m sorry if that’s not 100% detailed or clear, I’m going off memory, but I wanted to be able to respond before you left for the day. I’m not in tomorrow and would like to get this resolved before the weekend, if possible. 

Thanks. 

[Adam Gordon]

Hi Hung,

I was able to be in touch with one of your colleagues in Anaheim today, and they helped me work through some of the details of this issue. Essentially the MBR_BOOTLOADER_ADDR loaded into memory location 0xFF8 was correctly set at 0x78000 when the softdevice and bootloader were loaded, but for some reason, when using Segger to load the application, the value in 0xFF8 was being overwritten to 0xFFFFFFFF.

However, that did not happen when the app was loaded via nrfjprog or via DFU (the value remained 0x78000). So that was significantly throwing off our troubleshooting. I am now working through the issue with that knowledge in mind to better isolate the main problem.

[Hung Bui]

Hi Adam, 

It's really strange that the application would touch the MBR area. Do you have any code that may set the attribute to the MBR address (0-0x1000) ? 

If you can provide the hex file that reproduce the problem I can have a look. 

[Daniel Veilleux]

Hello Hung, it wasn't that the application was modifying the MBR -- SES was erasing the MBR page and reflashing the SoftDevice when Adam started the debugger. Using Target -> Attach prevents SES from modifying the flash.

[Adam Gordon]

Hi Hung,

I'm sorry I wasn't more specific in my last reply. When I used Target -> Attach debugger, I no longer ran into the fds_init() issue that we were previously discussing (the BOOTLOADER_ADDRESS macro was returning the correct value). Unfortunately the issue of our application not starting after a successful DFU or after a power cycle persisted.

But now that the FDS issue was no longer clouding our troubleshooting, I continued trying to isolate the underlying issue further. I discovered some odd behaviour that I was hoping you might be able to help with.

I was trying to figure out why our application can load and run without issue when programmed using nrfjprog, but runs into issues when loaded via DFU. I will write out the steps I took and their results:

Build and Generate:

1. Build full application (nothing commented out) in debug build.

2. Generate DFU package using the following command (names changed for clarity):

nrfutil pkg generate --hw-version 52 --sd-req 0xB7 --application-version 1 --application application.hex --key-file private.key dfu_package.zip

3. Generate bootloader settings using the following command:

nrfutil settings generate --no-backup --family NRF52 --application application.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 bootloader_settings.hex

Load and Run Using NRFJPROG

1. Load and run the application using nrfjprog with the following commands:

nrfjprog --recover
nrfjprog --program s132_nrf52_6.1.1_softdevice.hex
nrfjprog --program ble_bootloader.hex
nrfjprog --program bootloader_settings.hex
nrfjprog --program application.hex
nrfjprog --reset

2. The application then starts and runs without issue. ️

 

Load and Run Using DFU

1. Load and run the bootloader using nrfjprog with the following commands:

nrfjprog --recover
nrfjprog --program s132_nrf52_6.1.1_softdevice.hex
nrfjprog --program ble_bootloader.hex

nrfjprog --reset

2. Open dfu_package.zip in nRF Toolbox iOS app and perform DFU on "DfuTarg".

3. The DFU progresses and completes without issue, but application's LED pattern never shows. 

4. Attach debugger in Segger and restart application from Main. Fails right away at NRF_LOG_INIT()

 

Test Again Using NRFJPROG

1. Repeat step 1 from "Load and Run Using NRFJPROG"

2. The application then starts and runs without issue. ️

3. Attach debugger in Segger and restart application from Main. Application runs without issue. ️

4. Unplug device from power.

5. Plug device back into power.

6. The application's LED pattern never shows

7. Attach debugger in Segger and restart application from Main. Fails right away at NRF_LOG_INIT()

 

Check to See if it is a Reset Issue

1. Repeat step 1 from "Load and Run Using NRFJPROG"

2. The application then starts and runs without issue. ️

3. Reset the device using nrfjprog with the following command

nrfjprog --reset

4. The application then starts and runs without issue. ️

 

It's worth mentioning that the issue isn't specific to NRF_LOG_INIT() since I've tried removing it and it just fails at the next init() function I call anyway. It's also worth mentioning that I tried resetting the device using nrfjprog --pinreset and it showed the same behaviour as a power cycle.

So I guess my question would be, is there a reason why the same application that loads and runs without issue using nrfjprog is not able to run properly when loaded and started via DFU? And is it the same reason why a properly loaded and running program no longer runs without issue after being power cycled?

 

The issue seems to present in a very similar fashion, but what would cause it?

 

Once again I really appreciate the help.

[Hung Bui]

Hi Adam, 

I assume when you do debugging you have followed the suggestion from Daniel so that Segger wouldn't erase MBR, Softdevice ? 

I am not sure what could be wrong here.

In the step "Test Again Using NRFJPROG" do you mean that if you connect and use SEGGER to debug, you have trouble after that ? What you can do is to make a hex dump (nrfjprog --readcode) before and after connecting Segger and then compare to see what changed. You would also need to read UICR (nrfjprog --readuicr)

For further debugging I would suggest: 

- Adding some more LED pattern inside the bootloader, so we know if the bootloader is executed, and if it forward to the application

- Strip down the application until it would work with DFU update. (for example only do LED blinking). To do that you can just add an infinite loop in your code. Then you can gradually more this loop down until you can reproduce the issue. 

If you can provide an minimal code that we can test here with the NRF52 DK it would be easier to find the root cause.

[Adam Gordon]

Hi Hung,

Yes, when I now do debugging, I program and reset my device using nrfjprog and then use Target -> Attach debugger to do debugging.

Hung Bui said:

In the step "Test Again Using NRFJPROG" do you mean that if you connect and use SEGGER to debug, you have trouble after that ?

No, the trouble occurs before connecting and using Segger to debug. Essentially what I'm trying to say is that if I have the application running without issue and then power cycle my device, my application will not restart and run. Adding the addition comments about attaching the debugger was to shown when the application was running without issue and when it wasn't.

Hung Bui said:

- Adding some more LED pattern inside the bootloader, so we know if the bootloader is executed, and if it forward to the application

I added an LED pattern inside the bootloader just before nrf_bootloader_init(), and in every scenario the bootloader was executed. However I did start doing a little bit of debugging and noticed something that may or may not be worth mentioning.

I performed two separate DFUs, one with an application that just flashes some LEDs (and is a functioning app without any over the above issues), and one with our application that does not start after a DFU.

After performing a DFU with the LED app, the bootloader LED sequence executed twice (I'm assuming that means the bootloader was restarted twice). The LED app was then started.

After performing a DFU with our application, the bootloader LED sequence executed three times. I followed the bootloader execution in Segger and noticed that after bootloader restarted twice, it called app_start() and had the address of 0x1000, but that is when the bootloader restarted for the third time.

I feel like that behaviour wasn't supposed to occur. Not sure if that gives you any addition information to work with.

Hung Bui said:

If you can provide an minimal code that we can test here with the NRF52 DK it would be easier to find the root cause.

As was recommended, I've opened a private ticket that I will send you the link to in a private message. Thanks.

[Hung Bui]

OK, we can continue on the private case.