Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Wed, 19 Oct 2022 12:19:47 GMT

Ah, so the switch to LDO when attaching the debugger explains why the firmware works after the attach.

BTW: The peripheral_lbs build does fail with my custom dts. I built it for the nRF52840-DK and ran that on my custom board as it is somewhat compatible, at least for that simple example.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Wed, 19 Oct 2022 09:02:25 GMT

Thank you for the summary. I'm glad to hear that you found the problem!

[quote userid="66147" url="~/f/nordic-q-a/92774/hang-with-nrf5-sdk-17-1-0-bootloader-and-nrf-connect-sdk-2-1-0-application/391296"]Thank you for all the help! Obviously I should have seen this when setting up the custom dts. What I still don't understand is how that manifested in all these other odd things happening, working from the debugger, etc...[/quote]

It must be because of the noise generated by the DCDC. Although, I'm not able to explain exactly how this would interfere with the clock startup. This does however explain why you didn't experience the problem when the chip was in Debug Interface mode as this mode forces the chip to switch to the LDO regulator.

[quote userid="66147" url="~/f/nordic-q-a/92774/hang-with-nrf5-sdk-17-1-0-bootloader-and-nrf-connect-sdk-2-1-0-application/391296"]I don't have a storage partition defined in my custom board dts that my app uses.[/quote]

The build should fail at compile time if there is no storage partition defined. This is also what I observe here if I remove the storage_partition node before building the LBS sample.

I would suggest that you check the generated zephyr.dts file in <build directory>/zephyr/ before adding the overlay to see if the storage partition is indeed missing.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 20:00:04 GMT

Sorry, I missed that. I see, the change is in the peripheral_lbs/nrf52840dk_nrf52840.overlay. Using the nRF52840-DK board I can now DFU with that change and the app starts up. And if I remove the config changes to not use the DC/DC and to use the RC the app also starts up.

I then tried the same boot loader and peripheral_lbs dfu on my custom board without the RC changes. And it does hang. Until I attach debugger and hit continue.

So the behavior on the nRF52840-DK and our custom board is different.

I then checked the schematics and sure enough the inductors are not in this design for the DC/DC converter. Adding CONFIG_BOARD_ENABLE_DCDC=n to my app fixed the issue. The app starts up properly now.

Thank you for all the help! Obviously I should have seen this when setting up the custom dts. What I still don't understand is how that manifested in all these other odd things happening, working from the debugger, etc...

BTW: I don't have a storage partition defined in my custom board dts that my app uses.

CONFIG_BT_SETTINGS=y
CONFIG_SETTINGS_RUNTIME=y
CONFIG_SETTINGS=y
CONFIG_SETTINGS_NONE=y

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 19:46:34 GMT

Yes, the DK starts advertiding when using my version. But it enters a reset loop if I DFU your app. Writing the settings page is a quicker way to test this. And it’s the only way for me if I’m going to run my app with your bootloader.

Did you move the settings partition in your lbs project. If not, it will go in a boot loop like I mentioned in my first reply here.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 19:26:08 GMT

By "works on the DK" do you mean that peripheral_lbs starts up without hanging and you can connect to it?

I'm not programming in any settings.hex (or ncs_app.hex). Just programming in the boot loader, then the soft device, then the boot loader hex. Then using nrf connect app to run the DFU process. At the end of that is says "Your DFU update was completed successfully". At that point I do not see peripheral_lbs advertisements.

When I attach the vscode debugger, the call stack is the boot loader crc32_compute. When I continue from there I still don't see advertisements. Break in the debugger again and it is in the application in CC_PalWaitInterruptRND. Then tried break and continue a few more times and it is usually in one of those, but sometimes in other parts of the startup code. Sounds like a reset loop...

So seeing three different things:

1) My app. Doesn't startup properly. Connect debugger and continue and it then starts up fine.

2) The hello_world app. With the clock crystal in the startup it hangs waiting for the crystal startup. With the synthetic option, it starts fine.

3) The peripheral_lbs app. Connect debugger and continue and it appears to be stuck in a reset loop.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 18:19:31 GMT

[quote userid="66147" url="~/f/nordic-q-a/92774/hang-with-nrf5-sdk-17-1-0-bootloader-and-nrf-connect-sdk-2-1-0-application/391251"]I built peripheral_lbs with these additions to the prj.conf and reproduced the hang:[/quote]

Can you confirm that it hangs at the same location inside the cryptocell runtime library, and not in the fault handler because the settings partition overlap with the bootloader?

[quote userid="66147" url="~/f/nordic-q-a/92774/hang-with-nrf5-sdk-17-1-0-bootloader-and-nrf-connect-sdk-2-1-0-application/391254"]lbs-1.0.0.zip bootloader-neo.hex[/quote]

I can't get this bootloader to exit DFU mode. I guess it may be because of your NRF_BL_DFU_ENTER_METHOD_BUTTON_PIN setting in sdk_config, or were you able to run the same hex on your DK?

Turns out I programmed the wrong settings page. Programming your bootloader with my ncs_app.hex works on the DK:

nrfutil settings generate --family NRF52840 --application ncs_app.hex --application-version 1 --bootloader-version 1 --bl-settings-version 2 settings.hex

nrfjprog -e
nrfjprog --program bootloader-neo.hex --verify
nrfjprog --program s140_nrf52_7.2.0_softdevice.hex --verify
nrfjprog --program ncs_app.hex --verify
nrfjprog --program settings.hex --verify
nrfjprog -r

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 15:10:22 GMT

I ran the above binaries on an nRF52840-DK and am seeing the hang here.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 15:00:21 GMT

devzone.nordicsemi.com/.../35464.sdk_5F00_config.h

This is the config for the SDK boot loader.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 14:59:16 GMT

Thanks, I will try and see if I can reproduce it with your bootloader (hoping it will run on a DK).

nrfutil pkg display lbs-1.0.0.zip gives this:

So, the bootloader is not using the cc310 for boot validation, at least.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:58:00 GMT

Program boot loader:

nrfjprog -f NRF52 --recover

nrfjprog -f NRF52 --program

sdk_nordic/components/softdevice/s140/hex/s140_nrf52_7.2.0_softdevice.hex –-chiperase

nrfjprog -f NRF52 --program bin/bootloader-neo/bootloader-neo.hex

nrfjprog -f NRF52 --reset

Then use nrf connect on iOS to DFU the zip.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:56:17 GMT

devzone.nordicsemi.com/.../lbs_2D00_1.0.0.zip devzone.nordicsemi.com/.../bootloader_2D00_neo.hex

sdk_nordic/components/softdevice/s140/hex/s140_nrf52_7.2.0_softdevice.hex

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 14:55:49 GMT

In the CC_PalWaitInterruptRND loop? Can you check what the boot validation type is set to in your bootloader?

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:53:02 GMT

I built peripheral_lbs with these additions to the prj.conf and reproduced the hang:

CONFIG_NO_OPTIMIZATIONS=y

CONFIG_FLASH_LOAD_OFFSET=0x27000

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 14:41:28 GMT

I wish I could reproduce this here. Could you try the peripheral_lbs app I included in my first reply here to see if it results in the same problem? I included a pre-built hex for it (ncs_app.hex).

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:37:38 GMT

It is in the application address range.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 14:35:45 GMT

Could you check the program counter value to see if it is in the application's or bootloader's address range? The bt stack in NCS will initialize the cc310 RNG on startup, but I don't remember if it does it before main() or not.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:32:13 GMT

This is happening before app main is called.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Tue, 18 Oct 2022 14:24:01 GMT

I can't think of any obvious reasons for this. The peripherals used by the bootloader should be reset before the bootloader starts the application. Do you use the cc310 for boot validation or just do a CRC check (Boot validation modes)?

[quote userid="66147" url="~/f/nordic-q-a/92774/hang-with-nrf5-sdk-17-1-0-bootloader-and-nrf-connect-sdk-2-1-0-application/391229"]

So I put that config into my application and it gets past that point, but now hangs on:

CC_PalWaitInterruptRND

[/quote]

Is this in the application code?

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Tue, 18 Oct 2022 14:16:06 GMT

I added this to the hello_world config and now it starts up without the hang above:

CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH=y

So I put that config into my application and it gets past that point, but now hangs on:

CC_PalWaitInterruptRND

It looks like in both cases there is a wait for interrupt and the CPU is never woken up. I wonder if maybe these things are already on in the boot loader. Then when the boot loader starts the app, the app startup sequence assumes they are not on and waits for an interrupt that will never happen. Then attaching the debugger wakes the MCU and the code continues.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Fri, 14 Oct 2022 11:05:20 GMT

It is starting to look like a problem with the 32KHz clock now (the OS scheduler needs this clock to run). I didn't think this could be the issue earlier considering how your application ran fine when programmed without the bootloader.

Do you have the 32K crystal mounted on your board and have you tried to use these configurations from the peripheral_lbs.zip example I uploaded:

# Config settings to allow the FW to run on boards without the
# optional 32KHz crystal or DCDC indctors mounted
CONFIG_BOARD_ENABLE_DCDC=n
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Thu, 13 Oct 2022 16:19:39 GMT

I am trying to use a simpler app to see if it will start properly, so I am now using the sample hello_world. It is showing the same issue. However, this time when I attach in the debugger I end up in the same place each time - inside lfclk_spinwait loop:

while (!(nrfx_clock_is_running(d, (void *)&type)

... k_cpu_atomic_idle(key);

If I hit continue in the debugger then hello_world starts up. So maybe some issue starting up the low frequency clock or getting the interrupt that it has started?

This issue does not happen when an SDK 17 app is DFU'ed. So maybe something different in the Zephyr startup sequence with respect to the low frequency clock startup?

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Thu, 13 Oct 2022 15:35:08 GMT

I change some other config options and have RTT logging enabled. I added a print to the top of my main. When I start the app from the debugger (no boot loader) I see a boot message from zephyr and the message from my main:

*** Booting Zephyr OS build v3.1.99-ncs1 ***

starting vulcan

However, if I have RTT logging connected then the code works fine (no hang or reset loop). Same as when debugger is attached.

So I created a logging backend that just store the log text in RAM. Ran from debugger and the log messages are showing up in RAM.

Now, when I run the boot loader and then DFU my app and have hit the point where things look locked up or in a reset loop and then attach the debugger, I don't see any logging messages. So seems the issue happens before zephyr prints the boot message.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Wed, 12 Oct 2022 21:49:26 GMT

I added that config to the app, erased the part, programmed the boot loader, attached RTT, then did a DFU. I don't see any log output. And the app started normally (did not hang). I assume because the debug probe is connected for RTT.

How do you turn of ACL protection? Is that a config option in the SDK?

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

Vidar Berg — Wed, 12 Oct 2022 18:11:51 GMT

Yes, it's probably reaching the error handler which will reset the device (soft reset). I suggest you add the following kconfig settings to your prj.conf file:

CONFIG_LOG=y
CONFIG_LOG_BACKEND_UART=n
CONFIG_USE_SEGGER_RTT=y
CONFIG_RESET_ON_FATAL_ERROR=n
CONFIG_THREAD_NAME=y

Then when the app hangs, open Segger RTTViewer and see if you can get a crashlog from the error handler.

The reason I recommend disabling the ACL protection earlier is that it is one of the few configurations that will carry over from the bootloader to the application. But only as a test to help narrow down the problem. I don't recommend leaving ACL off in production.

RE: Hang with nRF5 SDK 17.1.0 Bootloader and nRF Connect SDK 2.1.0 application

denis — Wed, 12 Oct 2022 15:45:04 GMT

Yes, this is the nRF52840 chip.

It does look like some kind of reset loop. And I'm not seeing the lockup bit set now. Just the soft reset bit.

Problem is still that if I attach the debugger then things work normally, so unsure how to catch the problem.

Possibly something in the ncs sequence is doing a soft reset (except when running from the debugger).