DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 10 Feb 2022 08:34:48 GMT

Good to hear that Vidar, Both the findings are in my favour. Thanks for the update...

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Thu, 10 Feb 2022 08:30:43 GMT

Hi Alankar,

We have been able to find a root cause through simulation of the HW, and so far, we have determined that the condition for this error is not met when you have the WDT enabled. We still have some work remaining before we can publish the results.

Also, nRF52832 is not affected.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 10 Feb 2022 07:42:27 GMT

Hi Vidar,

Any update on this yet?

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Fri, 03 Dec 2021 10:27:49 GMT

Okay Vidar, I think it's better to keep this open, so that we can have eye on it...

Understood, with minimal example you can always narrow down to probable causes, good luck with your experiments. Please let me know if I could help at any point...

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Fri, 03 Dec 2021 10:00:11 GMT

It's fine to keep it open. I will make sure to update this thread either way. I'm currently trying to find a reliable way to reproduce the issue with a minimal example (no Softdevice), but it is proving to be quite difficult.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Wed, 01 Dec 2021 16:42:07 GMT

No problem, Vidar, do you want to keep this thread open till then? or we can mark WDT solution as a workaround and say, "Verify Answer". Please let me know what's your preference.

Thanks,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Wed, 01 Dec 2021 12:38:28 GMT

Hi Alankar,

I will definitely keep you updated. But I'm afraid it is going to take some time to get to the bottom of this as it seems to be more than just a SW bug.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Tue, 30 Nov 2021 07:56:43 GMT

Hi Vidar,

It's good to hear that, you could reproduce the issue now, which will help to debug further and find the solution. I am not sure about the timing and delay in my case!

Also, good to hear that it does not occur after WD reset so my current work around should help in field and again thanks for confirmation on 52832, so I do not need to worry about it at this point...

Thanks for updates, also please let me know if you find a solution on the issue with 52833/s140

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Mon, 29 Nov 2021 15:49:08 GMT

Hi Alankar,

Just an update: I have managed to reproduce the problem now by running a sd_enable() -> sd_disable() -> NVIC_Systemreset() cycle (hangs in sd_disable() on subsequent boot) . There appears to be some timing involved as well because the problem only occurs when I insert a delay between sd_disable() and reset. I think the timing here may help explain why the problem is so intermittent in your case.

The good news is that the issue does not occur after a WD reset. Also, the 52832 does not get stuck when I run the same test code indicating that either the 52832 or the s132 softdevice is not affected by this issue.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Fri, 26 Nov 2021 12:34:06 GMT

Hi Alankar,

That answers my question, thanks. As we are struggling to reproduce the issue on our side I wanted to make sure it didn't have anything to do with normal chip to chip variation.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Fri, 26 Nov 2021 12:28:32 GMT

Hi Vidar,

We haven't kept the track/history of which board has failed as it was very random, but I can guarantee that there is no board which never failed. Because last time I needed to power cycle 21 devices out of 23 and I am sure remaining 2 were failed previously.

Also, I can say that every device failed in similar number of counts as others, this is for sure.

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Fri, 26 Nov 2021 12:12:19 GMT

Hi Alankar,

Have you kept track of which board has failed and which hasn't? I'm basically wondering if there are boards that have never failed, and also if there are boards that seem to fail more frequently.

Thanks,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Fri, 26 Nov 2021 07:46:13 GMT

Hi Alankar,

Agree, there is no extreme temperature change here so I don't think it can be related to the issue either. I need to discuss with the team to figure out what the next step is.

Regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Fri, 26 Nov 2021 07:03:16 GMT

Hi Vidar, here are the results from this morning...

It shows only 2 degrees of rise. which is normal as the first readings were just after power on the device. I don't think temperature could be an issue...

Thanks

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 25 Nov 2021 13:20:10 GMT

Hi Vidar,

Setup ready, following image shows current die temperature readings, let's see what happens tomorrow

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 25 Nov 2021 12:48:14 GMT

Hi Vidar,

Yes, you are right, WDT reset helps to recover from this scenario and we have already tested it in our lab and deployed it to beta test site, so far, no issues...

Sure, I can do this trial, it won't need many efforts; I will try to setup a device today and keep it running overnight, so that we can monitor temperature tomorrow morning. I will update you on the results of this tomorrow morning...

Regards

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Thu, 25 Nov 2021 12:37:17 GMT

Hi Alankar,

Thanks!

[quote userid="93246" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340701#340701"]Obviously, this is for testing only, and we do not expect frequent OTA updates in the field, but want to make sure, whenever we do OTA in field, it should work every single time.[/quote]

I understand your concern as you are potentially going to push an update to lots of devices out in the field. Fortunately, the WDT reset will help you recover from this scenario.

One more thing, would it require much effort to take a known failing device and sample the die temperature from the application with sd_temp_get() to see if it increases over time due to surrounding circuitry (esp32 and voltage regulator)? I'm just trying to exhaust all possibilities.

Best regards,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 25 Nov 2021 11:38:33 GMT

Hi Vidar,

Here are answers to your questions

[quote userid="4240" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340698#340698"]1. Are the devices under test battery powered, or do they have a stable supply?[/quote]

These devices are not battery powered and have the stable power supply coming from AC-DC module on the board.

[quote userid="4240" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340698#340698"]2. Are they attached to a debugger during the test, and in that case, is the chip in debug interface mode?[/quote]

No, they are not attached to a debugger, they are like a finished product in an enclosure.

[quote userid="4240" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340698#340698"]3. Are the devices tested over a wide temperature range, or is it just at room temperature?[/quote]

No, all current on-going tests are at room temperature

[quote userid="4240" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340698#340698"]4. Does your test setup allow you to get log messages from devices under test?[/quote]

On my local dev setup, I tried to setup RTT debug log, but it does not work properly every time. And it is challenging because out of multiple devices, we are never sure, on which device issue can show up! as we cannot have debugger connected on each device.

[quote userid="4240" url="~/f/nordic-q-a/81311/dfu-bootloader-get-stuck-in-softdevice-at-0x0000c100---0x0000c0fc-then-jumps-to-0x00016348---0x0001635e-and-loops-forever-in-these-code-segments/340698#340698"]5. Roughly, how many OTA updates do you perform during the course of this test?[/quote]

For this test, we do OTA after every half hour in daytime and then again after overnight ops. As said earlier, issue occurrence is more after overnight running, so testing after overnight is more meaningful for us... So maybe 20-25 OTA updates in a day.

Obviously, this is for testing only, and we do not expect frequent OTA updates in the field, but want to make sure, whenever we do OTA in field, it should work every single time. and this issue is more concerned because occurrence of issue is more obvious after keeping devices operational for long time, which is the exact field scenario...

Thanks,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Thu, 25 Nov 2021 11:21:15 GMT

Hi Alankar,

Sorry for the delayed response. We still haven't managed to replicate the issue on our end, which does make challenging to find a root cause. But It is helpful to know that the issue still occurs with the HFXO kept on, that's one more thing we can cross off the list.

A few more follow up questions:

1. Are the devices under test battery powered, or do they have a stable supply?

2. Are they attached to a debugger during the test, and in that case, is the chip in debug interface mode?

3. Are the devices tested over a wide temperature range, or is it just at room temperature?

4. Does your test setup allow you to get log messages from devices under test?

5. Roughly, how many OTA updates do you perform during the course of this test?

Thanks,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Wed, 24 Nov 2021 16:43:44 GMT

Hi Vidar, did you get a chance to look at this?

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Sun, 21 Nov 2021 08:43:55 GMT

Hi Vidar,

We have finished testing with WDT, and everything seems to be working okay with it. Not a single event of hang or any abnormal behaviour, which was expected with WDT.

Now, we have updated the code to keep HFXO on as you suggested, below is the screenshot of modifications

And updated all the devices (5 with my local dev setup) and started to test.

It worked well every time (more than 30 times) yesterday and then I kept the setup running for normal operation overnight.

Then when I started testing with OTA again in the morning today, one of the devices got hung during the first trial only. So above change seems not helping on this issue.

Here, I want to note one more thing, which I forgot to mention during my first post is, the occurrence of issue is seen more obviously when you do the OTA after keeping the devices running for more time. So, if you start doing OTA just after power on, it works well, even if you do it multiple times or with a gap of say 1-2 hours it works well. But if you keep devices running for longer time like overnight (10-12 hours) and do OTA after that, it shows the issue (at least few devices). As said on my dev setup with 5 devices, it showed up on one device. But same thing I observed in our test lab too, where after running overnight, 5 to 6 devices out of 23, showed the issue.

Please see if above observation is pointing towards any direction...

Regards,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Fri, 19 Nov 2021 08:11:32 GMT

Hi Alankar,

No problem. The reason I was asking is that we have still not been able to replicate the issue, which makes it difficult to troubleshoot further on our end. The outcome of this test will hopefully help us narrow down the problem more.

Thanks,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Thu, 18 Nov 2021 17:22:55 GMT

Hey Sorry Vidar, we first wanted to implement and test the WDT fix, which is under testing and till now it is working well...

We shall finish this testing by weekend and can do some testing with HFXO on in next week. I will keep you posted on HFXO results

Thanks,

Alankar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Vidar Berg — Thu, 18 Nov 2021 08:11:00 GMT

Did you get a chance to repeat the tests with HFXO on?

Thanks,

Vidar

RE: DFU bootloader get stuck in SoftDevice at 0x0000C100 - 0x0000C0FC then jumps to 0x00016348 - 0x0001635E and loops forever in these code segments

Alankar — Tue, 09 Nov 2021 07:27:26 GMT

Thanks Vidar, I will try this out and update you on results...