URGENT : NRF52840 hanging up after working for some time!!!

RE: URGENT : NRF52840 hanging up after working for some time!!!

ovrebekk — Tue, 03 Oct 2023 12:51:34 GMT

Hi Kaushalya

I responded to you in the private ticket, and will park this case for now.

Best regards
Torbjørn

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Fri, 29 Sep 2023 02:58:32 GMT

Hi Stian,

Did you happen to have a look at the code? Is there anything that could lead to hang-ups?

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Thu, 21 Sep 2023 02:45:27 GMT

Hi Stian,

Got you. Please let me know what you think about the code. This is based on CoAP client example code, so there are still certain sections of it remaining in the code, which we don't need. I don't know if that could cause things like this.

I have got another sensor in the lab, which has hung up. Hung up in the sense no response of the LED to button press and no radio comms. When you connected your power analyzer, how did you do it without disrupting the power to the module? I have tried parallel powering and then remove the battery, but it caused a POR every time.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

Stian Røed Hafskjold — Wed, 20 Sep 2023 08:12:14 GMT

Hi, I think I'm the only one who can make this ticket private. Anyways, now that you have shared the code in the other ticket, I can just get it from there, and we can keep this one public. I can access the ticket.

[quote user="kaushalyasat"]How could we know its doing the LFRC clock calibration? Just by looking at the 4 sec current draws?[/quote]

Yes, 4 second interval and the length + current during the calibration event. Debugging using the current consumption profile is very useful, as you can see exactly what is going in (as long as you know what to look for of course)

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Wed, 20 Sep 2023 03:07:19 GMT

Hi Stian,

I have created a private ticket under the same name to include the FW. I dont know how to share it with you. If you cant access it, please let me know how to add you to that ticket.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Tue, 19 Sep 2023 07:10:54 GMT

Thanks Wendell, I tried looking for ways to move this to private, but couldn't find. I can move my original post, but there is no 'private' section in the list. Could you please elaborate a bit?

Thanks,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

WENDELL V BOYD — Tue, 19 Sep 2023 04:54:19 GMT

Kaushalya,

You should be able to move the DevZone ticket to 'Private' this will allow you to post any more sensitive or confidential materials such as your code directly to the Nordic team. Regards.

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Tue, 19 Sep 2023 02:03:03 GMT

Hi Stian,

All your support much appreciated. That is a good news. How could we know its doing the LFRC clock calibration? Just by looking at the 4 sec current draws?

To send the project, by PM, you mean as a personal message ?

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

Stian Røed Hafskjold — Thu, 14 Sep 2023 16:23:57 GMT

Hello,

I was able to power the devices from a power analyzer without resetting them. And I can clearly see that they are in system ON idle, consuming around 2 uA and doing the LFRC clock calibration every 4 seconds. So everything looks normal. Except that it is not advertising.

I don't think there are any reasons to suspect any battery issues, or any hardware issues, as the CPU is waking up to calibrate the LFRC, which means that the LFRC, RTC, CPU interrupts, etc, are working as normal. So this looks to me like a software issue.

I will continue to investigate, but it would be nice if you could share your software project, so I can build the code here. Maybe zip it and send it in a PM or similar. Thanks.

Best regards,
Stian

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Wed, 06 Sep 2023 01:03:06 GMT

Hi Stian,

Ok, I see. Unfortunately the two hung-up sensors are packaged to be send to you, so I cant run this anymore. You can try when you get the devices, hopefully soon. I have the other sensors which showed this behavior with me, but with WDT fw. So I dont know if if the WDT will heal the issue.

I am will do an IR test for the batteries on the failed ones to see how bad the IR is.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

Stian Røed Hafskjold — Tue, 05 Sep 2023 10:11:31 GMT

[quote user="kaushalyasat"]So if this is what happened, is it possible to recover when power cycled? If the battery is discharged, the internal resistance should have increased to a level which cannot sustain the current draw from the SoC isn't it? So if not immediate, we should see another hang-up quite soon from the same device. (we didn't replace batteries in these devices)[/quote]

Yes, I agree. So not likely the cause. (But I don't think we should rule out anything at this point)

[quote user="kaushalyasat"]As you can see, after the command is executed, it does seem to affect the current draw and the device seems locked up. The reset I was seeing earlier was due to the WDT. So I have not done this on the hang-up sensor I have, which we may ship to you for further analysis, as you suggest. If you have any thoughts on this, please let me know.[/quote]

So after issuing the nrfjprog --readregs command, the chip will enter debug mode, and the CPU will halt. Hence the change in current consumption. But it should not do a reset. It will connect to the running target, halt the CPU, read out the registers, then print them to the screen. Not sure if it is possible to resume the CPU and exit debug mode again without resetting, but you should already have the relevant register information printed to the screen.

You can try to resume the CPU with nrfjprog --run.

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Tue, 05 Sep 2023 05:00:40 GMT

Thanks again hmolesworth, From where you get the above graph? I couldn't find that in any of the documents. From it, I can see worst case IR could go up to ~130R when the battery is near end of juice.

I am looking at ways to add bulk capacitance. The only way I can do is multiple ceramic caps in parallel due to space restrictions. This will also reduce the ESR of the bulk capacitance.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Mon, 04 Sep 2023 02:47:06 GMT

Hi Stian,

Ok, so the sequence of events are like this as you suggest,

1. SoC enters a BOR and doesn't recover or hang-up

2. Due to this hang-up, no current drawn from the coin cell.

3. Due to no current draw, the coin cell voltage jumps back to 3V.

4. Because the SoC is in a hang-up state, the coin cell remains at 3V, which we see now.

So if this is what happened, is it possible to recover when power cycled? If the battery is discharged, the internal resistance should have increased to a level which cannot sustain the current draw from the SoC isn't it? So if not immediate, we should see another hang-up quite soon from the same device. (we didn't replace batteries in these devices)

When I tried ''nrfjprog --readregs" on a working device, it seemed reset, even after I disconnect the nReset line from the nRF52840K. I dug bit deeper into this. I downloaded the original release fw version to a sensor powered via a profiler and I was monitoring the current draw.

As you can see, after the command is executed, it does seem to affect the current draw and the device seems locked up. The reset I was seeing earlier was due to the WDT. So I have not done this on the hang-up sensor I have, which we may ship to you for further analysis, as you suggest. If you have any thoughts on this, please let me know.

I am thinking of using the battery from another sensor which hang-up earlier, but is now working after power cycling to a test to compare the IR with a brand new coin cell. I will keep you posted.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

Stian Røed Hafskjold — Fri, 01 Sep 2023 08:24:15 GMT

Hi, thanks for the answers.

If you are measuring a constant 3V on the failing device, I doubt that we are looking at a brown out reset (BOR) loop or similar. At 3V the device will recover, and you would have seen VDD dips below BOR threshold if the battery was not able to supply enough current for the boot sequence (i.e. reset loop). The only thing I can think of is that the device ended up in a BOR loop and because of that enters an unresponsive state, where it does not consume much current, so that the battery recovers back to 3V, but the device is still unresponsive.

The comment regarding nRESET was to check if the internal pullup resistor had been enabled, which it is, according to your measurements. But I would still like you to disable pin reset at some point, to see if the changes anything. So please try this after the WDT test.

The nrfjprog --readregs should not reset the device. I think that should be the next debugging step.

You are also welcome to send a couple of devices to our lab. I understand that it takes a long time to reproduce, so I guess you want to keep the unresponsive devices. But if you want, you can send me one of these unresponsive devices, and I can take a look, or you can send one that has not yet failed, and I can leave it running and see if it fails. It's up to you. I will send you a PM with the address.

RE: URGENT : NRF52840 hanging up after working for some time!!!

hmolesworth — Thu, 31 Aug 2023 06:57:40 GMT

We measured even higher than 300R on some discharged cells, I'll see if I can find our old work; another source is shown below. Some App Notes on pulse discharge of coin cells might prove useful in case you haven't come across them:

TI App Note swra349

Freescale App Note AN4573

Edit: Came across the original Nordic paper:

High-pulse-drain-impact-on-CR2032-coin-cell-battery-capacity

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Thu, 31 Aug 2023 03:18:27 GMT

Hi Stian,

Many thanks for your reply, much appreciated.

To answer your questions.

1. No this is a super rare event. We have seen three sensors done this in lab. None of them showed the behavior again. Having said that, we have restarted two devices last week and one this week. So dont know if it may pop up again in the future. We are observing these three.

There was a one reported from field about two weeks back. That one also working so far.

2. No we haven't. Now we have implemented WDT using the task wdt API with HW foldback enabled.

3. Yes it recover from power cycle.

4. I have one sensor in this state I am keeping without power cycle yet. Following are the voltages.

VDD : 3.02V (No drops detected on Oscilloscope, dont know if any tx is happening)

nRESET : 3.02V (No drops detected)

DEC1 & DEC4 : I am using Raytac MDBT50Q module, so these signals are not exposed for measurements.

5. CONFIG_GPIO_AS_PINRESET is enabled. You reckon a false reset happening on the nRESET? If after enabling WDT

we get this issue, I will try this.

6. 'CONFIG_RESET_ON_FATAL_ERROR' in .config file is enabled. I guess this means our sensor would reset for FW asserts.

7. When I tried this on a different sensor, it seems like the sensor got reset. Is there a way to read the registers without resetting the SoC? I have only one sensor in this state and the issue is not recreatable as of yet.

8. Unfortunately, to connect the profiler, I need to disconnect the onboard battery, which will take the sensor out of this state. I tried powering another sensor from the profiler with the sensor battery on board and then removing the sensor battery very carefully, but it reseted the sensor, no matter how saftely I tried to do it.

If there are no other tests you want me to carry out on this locked-up sensor, I will try to read the registers as in Step 7.

Thanks,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Thu, 31 Aug 2023 02:32:44 GMT

Thanks hmolesworth. Good point you raised to increase the bulk cap. I have obtained pulse characteristic data from the battery manufacturer and that showed that we should be able to handle the kind of pulse current demand we have for more than 1 year. So I didnt think that I would need any larger bulk capacitance . Unfortunately I don't have internal resistance data for pulse discharge for long time like an year. I will try to get that from the manufacturer.

I guess size of capacitance depends on what is the worst battery internal resistance we want to tolerate. You recon the worst would be like 300R?

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

hmolesworth — Thu, 31 Aug 2023 00:30:56 GMT

Good idea to try series resistance, though the coin cell can have an internal resistance much higher than 86R. As an aside, better to use 6.3V rating ceramics on a 3V coin cell due to the ceramic capacitor derating effect which can be as much as 50% of the rated capacitance. 4u7 is way, way too small in any case :-) Try just loading good and used coin cells with (say) 300R to get about 10mA drain and observe the coin cell voltage (which gives an indication of internal resistance) and how quickly each decays.

faq: ceramic capacitor derating

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Wed, 30 Aug 2023 23:27:08 GMT

Hi hmolesworth, Thanks again. Yesterday I did test the sensor which failed and as I mentioned earlier, I only saw about 0.2V drop from the oscilloscope. I dont think that is high enough to disrupt the radio behavior. I didnt see any voltage drops higher or closer to 1.7V. This is with the same battery that showed the hangup behavior earlier.

I will hookup a low power voltage detector to see if any such voltage drops can be detected.

The total capacitance on battery side is 4u7 at the moment. I can see what is the largest I can get in the same footprint so that it's just a BoM change. I will add up more bulk capacitance in the next rev of the PCB.

We have an LED that gets activated on button presses. It is not activated on any Tx cycles. When these devices lockup, most probably none of the LED or the button would have been activated.

I am thinking of running a test with a series resistor of around 86R inserted to simulate a weak battery and see if I can recreate the issue.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

Stian Røed Hafskjold — Wed, 30 Aug 2023 10:16:29 GMT

Hi Kaushalya,

Just a FYI first:

HFINT is the internal high frequency clock, and has nothing to do with the 32k clock
LFRC is the internal 32k low frequency clock

Your questions:

[quote user=""]1. What possible causes could be there for the hang-up of nRF52840? As mentioned, it is powered by a 3V coin cell, which is not discharged and running off the internal HFINT oscillator. [/quote]

As hmolesworth mentions, voltage dips can lead to the device being stuck in a reset loop
Or it could be software related. How has the device been configured to handle software asserts? Is it resetting on assert? If not, it will be stuck and not recover when an assert occurs.

[quote user=""]

2. Can the 32k external oscillator being present in the sensor affect the stability of HFINT oscillator?

[/quote]

No, it will not affect the stability of the LFRC oscillator

[quote user=""]3. Is the way in which we have enabled the internal oscillator good enough? Do we need to do any calibrations for it?[/quote]

Calibration should be automatically set when you enable LFRC in the configs. If you look through the build/zephyr/.config file you can check if calibration has been enabled or not. This file shows the status for all configs after build.

[quote user=""]4. Can the internal WDT restart the HFINT oscillator, if it has stopped for whatever reason?[/quote]

The WDT must be enabled. It will reset the chip if a hardfault/CPU lockup has occurred.

[quote user=""]5. How to enable, load and hit the internal WDT from application FW?[/quote]

Here is a sample that shows how to use the WDT:
https://github.com/nrfconnect/sdk-zephyr/tree/main/samples/drivers/watchdog

API:
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/hardware/peripherals/watchdog.html

Questions for you:

Has the failure occurred multiple times on the same device?
Have you been able to reproduce this on the returned devices?
Does it recover after a power on reset?
When the device is in this lockup state, are you able to probe voltage levels on different pins and post the results here? Pins of interest are, DEC1, DEC4 and nRESET as well as VDD.
Is pin-reset enabled (Look for this in the build/zephyr/.config file: CONFIG_GPIO_AS_PINRESET)? If so, can you try to disable it and see if the lockup still occurs?
How is a software assert being handled? Is it resetting? More specifically is RESET_ON_FATAL_ERROR being set?
Are you able to connect a debugger and see where in the program is when the device stops working? You can use nrfjprog --readregs to read out the relevant registers. Note that starting a debug session may reset the device, unless you choose to connect to running target. So I will suggest using nrfjprog --readregs which will not reset the device.
Can you post the current profile (.ppk file) of the device when it is in this lockup state?

Best regards,
Stian

RE: URGENT : NRF52840 hanging up after working for some time!!!

hmolesworth — Tue, 29 Aug 2023 21:45:25 GMT

OpenThread or BLE, same issue. Power Profiler II unfortunately only measures current and not voltage, although I believe this option has been requested. That means really a 'scope is required to show both current and voltage decay; no obvious voltage dips is highly improbable, if you'll pardon me suggesting that. The difference between a depleted battery and new is a much higher internal impedance in the former, leading to more problematic voltage dips for a given current spike. Ironically network traffic of unrelated devices in the immediate physical vicinity in the 2.4GHz band - BLE, OT, Wi-Fi, ... will cause greater problems as a retransmission attempt due to a transmit interfered with can effectively double the voltage dip time which deepens the dip as the coin cell voltage falls at a pretty constant rate during transmission and recovers slowly, even more so on a depleted battery.

Power Profiler II has a slow sample rate, 100KHz if I remember correctly. That's too slow to see fast spikes which can trigger the reset. Using brown-out detection or a comparator at a higher voltage level can capture progressively lower voltage drops which may occur prior to a reset, worth considering, as these can indicate the chance of a reset in the near future.

The side effect of these voltage dips even if no reset issue if using any form of analogue sensor is easily seen by taking an FFT on the data stream, where the periodic transmit pulse will show as a frequency spike at the update rate of the transmission assuming regular transmission intervals. That's a bit of a nuisance when diagnosing medical conditions from biometric sensors, for example.

What is the total capacitance on the coin cell? Are there any other circuits (maybe LEDs or flash memory) which can have current spikes coincident with a transmission (typically asynchronous spikes to transmission but periodically coinciding)? It is possible to suppress all other current pulses during a transmission pulse by using radio events.

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Tue, 29 Aug 2023 03:13:07 GMT

Hi Kazi,

Thanks for your help. We dont use the external oscillator now, even though the actual parts are still present in the PCB. We use the internal oscillator by having the following in the prj.conf

CONFIG_CLOCK_CONTROL=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL=n

The 32k crystal we used is Abracon ABS05-32.768KHZ-9-T. Data sheet Abracon ABS05-32.768KHZ-9-T. The load caps are 8pF.

We may not have selected any accuracy settings. Please let us know what to set and how to set it.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

kaushalyasat — Tue, 29 Aug 2023 03:03:44 GMT

Hi hmolesworth,

Thanks for this. We use OpenThread, not BLE, if that matters. We have used the power profiler to check the peak current demand on transmit periods, and didnt see any voltage drops. Having said that, the coin cells probably are newish. But the ones we have seen hanging up, the batteries are pretty much brand new. I have attached a current profile I have taken from Power Profiler.

Also the two sensors which failed in the lab, after power cycling they seem to be going on. Do you reckon if a depleted battery can show this kind of behavior for like a week?

I have tested one of these sensors. The 3V drops to about 2.8V when the TX is active and the sensor is wakeup. But I haven't seen any drops like close to 1.7V.

I will see how to have a bulk capacitance in parallel to the VBAT, but due to space limitations, it would be tricky to get something like 150u. We use the nRF52 DC-DC regulator by

CONFIG_BOARD_ENABLE_DCDC_HV=n

CONFIG_BOARD_ENABLE_DCDC=y

in prj.conf.

Cheers,

Kaushalya

RE: URGENT : NRF52840 hanging up after working for some time!!!

hmolesworth — Mon, 28 Aug 2023 16:43:04 GMT

As an aside, a single CR2032 coin cell will not be able to support BLE transfers for very long without the voltage dip on BLE transmit (even if just advertising) leading to repeated resets and eventual hang-up of the nRF52 as the voltage recovery rise time of the coin call slows down and eventually violates the VCC rise time. When the CR2032 coin cell is new this may not be an issue, but with time the internal impedance of the coin call increases so much that the transmit power pulse causes a large voltage dip. This can be confusing to debug as a coin cell - even when nearly fully discharged - can recover to 2.9 volts while the nRF52 is in reset.

Avoid this by increasing the size of the bulk capacitance on the coin cell; typically 150uF is a minimum using ceramic capacitors rated at double the coin cell voltage, 10 volt rating preferred. More is better. Ensure the nRF52 DC-DC is enabled, which can halve the required pulse current drain from the coin cell which can in turn reduce the voltage dip on transmit.

Edit: More info in these 2 posts:

nrf52810-using-cr2032-in-freezing-temps

issues-with-power-on-reset

RE: URGENT : NRF52840 hanging up after working for some time!!!

Kazi Afroza Sultana — Mon, 28 Aug 2023 07:22:45 GMT

Hello Kaushlya,

I have started looking into the case. How have you made sure ''Now we have an external 32k oscillator in our design, which showed stopping after some time probably due to capacitor loading not being correct.''? The HFXO must be running to use the RADIO, or the calibration mechanism associated with the 32.768 kHz RC oscillator.

When you turned to internal oscillator did you select the correct accuracy of the crystal? For higher LFCLK accuracy (when better than +/- 500 ppm accuracy is required), the low frequency crystal oscillator (LFXO) must be used. I can not see these seatings in your prj.conf file.

Could you please share your crystal datasheet with us?

Thanks.

Kazi