Issue with Application loading after migration from SDK 14.2 to SDK 15.3

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Thu, 29 Aug 2019 11:14:08 GMT

OK, we can continue on the private case.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Wed, 28 Aug 2019 22:11:25 GMT

Hi Hung,

Yes, when I now do debugging, I program and reset my device using nrfjprog and then use Target -> Attach debugger to do debugging.

[quote userid="2121" url="~/f/nordic-q-a/50954/issue-with-application-loading-after-migration-from-sdk-14-2-to-sdk-15-3/206681"]In the step "Test Again Using NRFJPROG" do you mean that if you connect and use SEGGER to debug, you have trouble after that ?[/quote]

No, the trouble occurs before connecting and using Segger to debug. Essentially what I'm trying to say is that if I have the application running without issue and then power cycle my device, my application will not restart and run. Adding the addition comments about attaching the debugger was to shown when the application was running without issue and when it wasn't.

[quote userid="2121" url="~/f/nordic-q-a/50954/issue-with-application-loading-after-migration-from-sdk-14-2-to-sdk-15-3/206681"]- Adding some more LED pattern inside the bootloader, so we know if the bootloader is executed, and if it forward to the application[/quote]

I added an LED pattern inside the bootloader just before nrf_bootloader_init(), and in every scenario the bootloader was executed. However I did start doing a little bit of debugging and noticed something that may or may not be worth mentioning.

I performed two separate DFUs, one with an application that just flashes some LEDs (and is a functioning app without any over the above issues), and one with our application that does not start after a DFU.

After performing a DFU with the LED app, the bootloader LED sequence executed twice (I'm assuming that means the bootloader was restarted twice). The LED app was then started.

After performing a DFU with our application, the bootloader LED sequence executed three times. I followed the bootloader execution in Segger and noticed that after bootloader restarted twice, it called app_start() and had the address of 0x1000, but that is when the bootloader restarted for the third time.

I feel like that behaviour wasn't supposed to occur. Not sure if that gives you any addition information to work with.

[quote userid="2121" url="~/f/nordic-q-a/50954/issue-with-application-loading-after-migration-from-sdk-14-2-to-sdk-15-3/206681"]If you can provide an minimal code that we can test here with the NRF52 DK it would be easier to find the root cause.[/quote]

As was recommended, I've opened a private ticket that I will send you the link to in a private message. Thanks.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Wed, 28 Aug 2019 14:29:25 GMT

Hi Adam,

I assume when you do debugging you have followed the suggestion from Daniel so that Segger wouldn't erase MBR, Softdevice ?

I am not sure what could be wrong here.

In the step "Test Again Using NRFJPROG" do you mean that if you connect and use SEGGER to debug, you have trouble after that ? What you can do is to make a hex dump (nrfjprog --readcode) before and after connecting Segger and then compare to see what changed. You would also need to read UICR (nrfjprog --readuicr)

For further debugging I would suggest:

- Adding some more LED pattern inside the bootloader, so we know if the bootloader is executed, and if it forward to the application

- Strip down the application until it would work with DFU update. (for example only do LED blinking). To do that you can just add an infinite loop in your code. Then you can gradually more this loop down until you can reproduce the issue.

If you can provide an minimal code that we can test here with the NRF52 DK it would be easier to find the root cause.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Tue, 27 Aug 2019 22:41:20 GMT

Hi Hung,

I'm sorry I wasn't more specific in my last reply. When I used Target -> Attach debugger, I no longer ran into the fds_init() issue that we were previously discussing (the BOOTLOADER_ADDRESS macro was returning the correct value). Unfortunately the issue of our application not starting after a successful DFU or after a power cycle persisted.

But now that the FDS issue was no longer clouding our troubleshooting, I continued trying to isolate the underlying issue further. I discovered some odd behaviour that I was hoping you might be able to help with.

I was trying to figure out why our application can load and run without issue when programmed using nrfjprog, but runs into issues when loaded via DFU. I will write out the steps I took and their results:

Build and Generate:

1. Build full application (nothing commented out) in debug build.

2. Generate DFU package using the following command (names changed for clarity):

nrfutil pkg generate --hw-version 52 --sd-req 0xB7 --application-version 1 --application application.hex --key-file private.key dfu_package.zip

3. Generate bootloader settings using the following command:

nrfutil settings generate --no-backup --family NRF52 --application application.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 bootloader_settings.hex

Load and Run Using NRFJPROG

1. Load and run the application using nrfjprog with the following commands:

nrfjprog --recover
nrfjprog --program s132_nrf52_6.1.1_softdevice.hex
nrfjprog --program ble_bootloader.hex
nrfjprog --program bootloader_settings.hex
nrfjprog --program application.hex
nrfjprog --reset

2. The application then starts and runs without issue. ✔️

Load and Run Using DFU

1. Load and run the bootloader using nrfjprog with the following commands:

nrfjprog --recover
nrfjprog --program s132_nrf52_6.1.1_softdevice.hex
nrfjprog --program ble_bootloader.hex

nrfjprog --reset

2. Open dfu_package.zip in nRF Toolbox iOS app and perform DFU on "DfuTarg".

3. The DFU progresses and completes without issue, but application's LED pattern never shows. ❌

4. Attach debugger in Segger and restart application from Main. Fails right away at NRF_LOG_INIT() ❌

Test Again Using NRFJPROG

1. Repeat step 1 from "Load and Run Using NRFJPROG"

2. The application then starts and runs without issue. ✔️

3. Attach debugger in Segger and restart application from Main. Application runs without issue. ✔️

4. Unplug device from power.

5. Plug device back into power.

6. The application's LED pattern never shows ❌

7. Attach debugger in Segger and restart application from Main. Fails right away at NRF_LOG_INIT() ❌

Check to See if it is a Reset Issue

1. Repeat step 1 from "Load and Run Using NRFJPROG"

2. The application then starts and runs without issue. ✔️

3. Reset the device using nrfjprog with the following command

nrfjprog --reset

4. The application then starts and runs without issue. ✔️

It's worth mentioning that the issue isn't specific to NRF_LOG_INIT() since I've tried removing it and it just fails at the next init() function I call anyway. It's also worth mentioning that I tried resetting the device using nrfjprog --pinreset and it showed the same behaviour as a power cycle.

So I guess my question would be, is there a reason why the same application that loads and runs without issue using nrfjprog is not able to run properly when loaded and started via DFU? And is it the same reason why a properly loaded and running program no longer runs without issue after being power cycled?

The issue seems to present in a very similar fashion, but what would cause it?

Once again I really appreciate the help.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Daniel Veilleux — Tue, 27 Aug 2019 21:27:15 GMT

Hello Hung, it wasn't that the application was modifying the MBR -- SES was erasing the MBR page and reflashing the SoftDevice when Adam started the debugger. Using Target -> Attach prevents SES from modifying the flash.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Mon, 26 Aug 2019 11:56:54 GMT

Hi Adam,

It's really strange that the application would touch the MBR area. Do you have any code that may set the attribute to the MBR address (0-0x1000) ?

If you can provide the hex file that reproduce the problem I can have a look.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Fri, 23 Aug 2019 02:21:30 GMT

Hi Hung,

I was able to be in touch with one of your colleagues in Anaheim today, and they helped me work through some of the details of this issue. Essentially the MBR_BOOTLOADER_ADDR loaded into memory location 0xFF8 was correctly set at 0x78000 when the softdevice and bootloader were loaded, but for some reason, when using Segger to load the application, the value in 0xFF8 was being overwritten to 0xFFFFFFFF.

However, that did not happen when the app was loaded via nrfjprog or via DFU (the value remained 0x78000). So that was significantly throwing off our troubleshooting. I am now working through the issue with that knowledge in mind to better isolate the main problem.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Thu, 22 Aug 2019 15:15:03 GMT

Hi Hung,

Good point, I have my own set of public/private keys that I will use with the examples to test the flash_fds app.

As for the BOOTLOADER_ADDRESS macro, unfortunately I’m currently not in the office, but I looked into it a lot yesterday so will try to respond from memory to see if we can resolve this issue today.

Yes we do have that macro inside app_util.h. I believe MBR_BOOTLOADER_ADDR was 0xFF8, pointing to a value of 0xFFFFFFFF. flash_end_addr() ended up returning 0x80000 because one of the lines of code has something that resembled two _sz being multiplied together.

I’m sorry if that’s not 100% detailed or clear, I’m going off memory, but I wanted to be able to respond before you left for the day. I’m not in tomorrow and would like to get this resolved before the weekend, if possible.

Thanks.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Thu, 22 Aug 2019 14:43:04 GMT

Hi Adam,

I assume you have your own private key use with your bootloader ? You mentioned that you can do OTA DFU with your application ?

If you want to test the flash_fds you should either generate a .zip file of the hex or generate at bootloader setting. I attached here the .zip file and the bootloader setting I used. Saw no issue with the example.

Could you please double check what exactly return in flash_end_addr() ? when you have the bootloader installed and the application installed.

Especially check if you have this inside app_util.h

#define BOOTLOADER_ADDRESS ((*(uint32_t *)MBR_BOOTLOADER_ADDR) == 0xFFFFFFFF ? *MBR_UICR_BOOTLOADER_ADDR : *(uint32_t *)MBR_BOOTLOADER_ADDR)

The bootloader address should point to the start address of your bootloader.

I suspect that since we don't write to MBR_UICR_BOOTLOADER_ADDR any more it could cause an issue if you don't have correct BOOTLOADER_ADDRESS macro .

devzone.nordicsemi.com/.../flash_5F00_fds.hex devzone.nordicsemi.com/.../bootloader_5F00_settings.hex

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Wed, 21 Aug 2019 15:47:24 GMT

Hi Hung,

So I ran my application without loading the bootloader and the start and end addresses were 0x7D000 and 0x80000. I then ran the application with the bootloader loaded and the start and end addresses were the same. I'm assuming this is the issue since it overlaps with the bootloader's location in memory?

Also, I checked for data stored using Segger and there doesn't seem to be anything stored right before the bootloader (see image below).

As for reproducing the issue with the flash_fds example, I wish I could have tested it by performing a DFU, however a DFU package for the flash_fds example was not provided in the test images in the SDK, and since I do not have the private key of the SDK, I could not generate a DFU package on my own.

I actually tested it two ways with the same results. The first way I did it was load the bootloader project file into Segger and build and run. And then load the flash_fds project file into Segger and build and run. Also, I tried using nrfjprog to program the softdevice (with --chiperase) and then program the bootloader right after (with --reset). And then once that ran I used Segger to build and run the flash_fds example.

EDIT: I tried hardcoding the end_addr value in flash_bounds_set() to 0x78000 to see if that would resolve the issue and I got some interesting results. If I load and run the bootloader (with softdevice loaded) using nrfjprog and then use Segger to build and run my application, it now loads and runs properly! Also, if I use nrfjprog to load the softdevice, bootloader, and bootloader settings and then reset the chip, allow the bootloader to run, and then load my application and reset the chip, the application runs properly. HOWEVER, I tried only loading and running the bootloader+softdevice to then load the application using OTA DFU from the bootloader and the application did NOT start. It's also worth mentioning that in the instances where I got the application to start running properly (with the hardcoded value), the application did not restart after a power cycle.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Wed, 21 Aug 2019 11:25:39 GMT

Hi Adam,

I think we are getting closer to the problem.

I'm suspecting the change in the start address of the bootloader can cause the issue on the new SDK. The reason is that fds would detect where the bootloader is (or if the bootloader exist) and then allocate the data space accordingly.

We need to check if it allocated the area correctly. Could you add a breakpoint inside flash_subsystem_init() and check what's the end_addr and the start_addr inside flash_bounds_set() function ?

Also could you do a hex dump (nrfjprog --readcode) and check if there is any data stored in the pages right before the bootloader (0x78000 or 0x72000 if you use bootloader debug version)

Regarding reproducing the issue with flash_fds example, could you let me know how you tested it ?

Did you flash softdevice and bootloader. And then you did DFU update using a .zip file of the flash_fds example ?

Or you flash the flash_fds using the programmer and change the bootloader setting to accept that image ?

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Tue, 20 Aug 2019 17:26:53 GMT

[quote userid="2121" url="~/f/nordic-q-a/50954/issue-with-application-loading-after-migration-from-sdk-14-2-to-sdk-15-3/204965"]Strip down your application to a minimal function, for example blinking an LED. If it works after an DFU, you can start testing with more feature until it stop.[/quote]

The error occurs when calling fds_init(). Specifically, pages_init() returns with a value of NO_PAGES.

This, however, only occurs after bootloader code was run. If I erase all and then run only the application, pages_init() returns FRESH_INSTALL and things work normally.

From this, I have come to understand that because the bootloader writes to flash storage, my application is no longer able to initialize fds properly. How can I get around this issue?

EDIT: To further isolate the issue, and to hopefully allow you to reproduce it on your end, I programmed and ran the secure ble bootloader example from the SDK (with softdevice loaded) on a PCA10040 DK and then ran the flash_fds example from the SDK and encountered the same issue. If I then erased all on the device and ran the flash_fds example on it's own, it functioned properly. If I loaded and ran the bootloader and then ran the flash_fds example, I would get the issue with CLI output of:

<info> app: FDS example started.
<info> app: Initializing fds...
<error> app: Fatal error
<warning> app: System reset

Debugging through the flash_fds code, I see the same issue where fds_init() returns FDS_ERR_NO_PAGES.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Tue, 20 Aug 2019 09:08:36 GMT

Hi Adam,

It's normal to have app_start () jumping to 0x1000. It's where the vector table of the softdevice is and the softdevice will then forward the vector table to the application address which is at 0x26000.

One thing you would need to make sure is that your application is configured to start at 0x26000 (IROM1 start address at 0x26000). Note that, the application is always located at 0x26000 because of the bootloader. But if it's not configured to start at 0x26000, it may not work properly (the vector table offset could be wrong)

I would suggest to do the following:

- Clarify that the application (both release and debug version) can work without the bootloader (you may need to turn off buttonless DFU feature)

- Strip down your application to a minimal function, for example blinking an LED. If it works after an DFU, you can start testing with more feature until it stop. If the application can start with LED blinking, you can start doing debugging with it because this mean the program counter is jumping to the application's reset handler. You can remove CRC checking inside the bootloader, so you can test the new image without doing DFU, just flash it normally.

Do you have any flash or UICR operation in your application ?

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Mon, 19 Aug 2019 20:49:00 GMT

Hi Hung,

Unfortunately commenting out the functions related to ble_dfu_buttonless did not resolve the issue.

I think I may have narrowed down the issue even further. As previously mentioned, our custom firmware is having the issue with the app being loaded correctly after an OTA DFU and on power on. However, I took one of the example apps from the SDK and changed it to work with our hardware and that app runs correctly after an OTA DFU and after a power cycle. So I tried to look into differences in how they are loaded/handled by the bootloader.

As far as I can tell, they are handled in exactly the same way. By using breakpoints and stepping through the bootloader code, after the DFU is successfully complete, the bootloader then starts the application by calling app_start(vector_table_addr) from within nrf_bootloader_app_start_final(). In both cases, the address being passed to app_start is 0x1000, which corresponds to the start address of the softdevice.

In the case of the modified example app, after app_start() is called the app starts running successfully. In the case of our custom firmware, once app_start() is called, the app does not start running, Segger is "Stopped by vector cache" and has the following call stack:

I then wanted to make sure that the application was being loaded into the right address (should be 0x26000). So I double checked 0x26000 in memory before and after the DFU was performed, and sure enough it was being loaded correctly.

Something else I tried was adding our application's .hex file under the loader options of the bootloader project in Segger. So if I build and run the bootloader in Segger, it loads the softdevice and our app and then runs the bootloader. Doing this successfully started our application. Again, stepping through the bootloader code, it called app_start() with the argument address being 0x1000, except this time it successfully started the application (I double checked that it was loaded at 0x26000)

So what is the difference between Segger loading the app into memory and the DFU process loading the app into memory before jumping into the softdevice code? Why does one work and the other not?

And even when the app is loaded in Segger and the bootloader starts it correctly, why is it that if it is power cycled it does not start up successfully again once plugged in?

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Fri, 16 Aug 2019 08:38:36 GMT

Hi Adam,

Could you try to comment out the functions related to ble_dfu_buttonless and check if you still having trouble ? (you can still test DFU without buttonless)

Regarding the change in SDK v15.3, there is only one thing I can think of is the change of the place the start address of the bootloader is stored. It's now no longer stored in the UICR but by default stored in the MBR (but it still backward compatible). You can have a look at my answer here.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Thu, 15 Aug 2019 21:28:08 GMT

Hi Hung,

Thanks again for getting back to me. Unfortunately I realized that I may have had a piece of misleading information in my last reply without realizing. I mentioned that when I use nrfjprog my application runs fine but when I use Segger it does not. I realize that the difference is that in Segger I was flashing the debug build and with nrfjprog I was flashing the release build. If I flash the debug build with nrfjprog, my applications runs properly. Also, if I flash the release build using Segger, it does not work. All that to say, I am seeing the same behaviour when flashing with nrfjprog as I am when flashing with Segger.

Also, to clarify, when flashing the debug build, the application runs properly but does not start again after a power cycle. The application never starts after a power cycle.

Also, unfortunately I don't be able to provide you with a .hex file to test on your nRF52 DK since we are using a custom board and have different pin definitions.

Lastly, I never had any of these issues for the year+ that our firmware was running on the same board but with SDK 14.2. Can you think of what might have changed in SDK 15.3 that would have cause this functionality to break after the migration?

Edit: It's worth mentioning that I also tried performing a DFU with a package generated from the debug build of our application and it had the same result as with the release mode package. Either way, I did some further investigating and realized that the reason the release build threw an error is because it was calling ble_dfu_buttonless_async_svci_init(), which assumes the presence of a bootloader, which there wasn't.

Are there any new bootloader-related configurations or flags we need to set in our application that are new to SDK 15.3?

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Thu, 15 Aug 2019 10:17:18 GMT

Hi Adam,

To isolate the issue, I would suggest we focus into the observation that if you flash just the softdevice and the application with nrfjprog, the application won't start after a reset. (So it does not involve the bootloader)

We need to find out why it works with Segger but not with nrfjprog. You can do a nrfjprog --readcode and compare the hex file between flashing with nrfjprog and Segger.

What you can do is to trim down the functionality of the application (until it only blink an LED for example) we can narrow down where the problem could be. You can simply put an infinite loop into your code to limit the functionality.

Then you can send us the minimal hex file that we can try to test here on a nRF52 DK.

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Adam Gordon — Wed, 14 Aug 2019 20:15:13 GMT

Hi Hung,

Thank you for getting back to me! Unfortunately it's not just after a reset that my application does not start, it is also after a DFU.

We have no special function related to UICR. We use FDS in our application, but that is it.

Do you mean from SES? If so then yes, our application works fine if flashed from Segger with just the application and softdevice. It does not work if just the softdevice and application are flashed using nrfjprog (and then the device reset).

However, what is odd, is that if we take all of the .hex files of the bootloader, bootloader settings, softdevice, and application and merge them using mergehex and then flash them onto the device using nrfjprog and reset, the application runs and works fine. Even if I flash the .hex files each individually everything runs fine.

But if I flash just the bootloader and softdevice and then successfully perform a DFU, the bootloader does not jump to my application code. Is there a reason why it would work when I flash all the components together and not when I perform a DFU?

RE: Issue with Application loading after migration from SDK 14.2 to SDK 15.3

Hung Bui — Wed, 14 Aug 2019 15:36:02 GMT

Hi Adam,

You may want to re-compile the bootloader with optimization to level 0 (so you can debug) and step into the bootloader code to see why it doesn't jump to your application after a reset. I suspect your application may change (due to a flash operation ? ) causing CRC32 check doesn't match ?

Do you have any special function in your application related to UICR ? flash ?

I assume your application works fine if you only flash the softdevice and the application ?