URGENT : NRF52840 hanging up after working for some time!!!

[kaushalyasat]

Hi,

THIS IS AN URGENT REQUEST. PLEASE HELP ASAP.

After deploying our sensors based on nRF52840 running coap client on OpenThread and powered by 3V 2032 Lithium coin cell, we have noticed that two out of about 50 sensors just not sending data to the host. One of the sensors we know has hung up with no response to the button push and lo LED indication. The other have the characteristics of it, but not yet confirmed by the client. The issue is not immediate, some like couple weeks, some couple of days. We have seen two of our sensors, which are being tested in our lab, to hang up as well. We dont have an external watchdog. Not sure if the internal watchdog is enabled.

Once the sensor hangs up, you can make it come back alive by power cycling. 

Now we have an external 32k oscillator in our design, which showed stopping after some time probably due to capacitor loading not being correct. So we revert to internal oscillator, which seem to solve the issue. That is how these sensors been working so far and release to customers. But the external oscillator and two loading caps exist in the PCB. I dont know if this can be an issue if we use HFINT internal oscillator. 

We have enabled the internal oscillator by the following in the prj.conf.

CONFIG_CLOCK_CONTROL=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL=n

We have not done any calibrations for the HFINT. 

So these are my questions.

1. What possible causes could be there for the hang-up of nRF52840? As mentioned, it is powered by a 3V coin cell, which is not discharged and running off the internal HFINT oscillator. 

2. Can the 32k external oscillator being present in the sensor affect the stability of HFINT oscillator?

3. Is the way in which we have enabled the internal oscillator good enough? Do we need to do any calibrations for it?

4. Can the internal WDT restart the HFINT oscillator, if it has stopped for whatever reason?

5. How to enable, load and hit the internal WDT from application FW?

Cheers,

Kaushalya

[hmolesworth]

As an aside, a single CR2032 coin cell will not be able to support BLE transfers for very long without the voltage dip on BLE transmit (even if just advertising) leading to repeated resets and eventual hang-up of the nRF52 as the voltage recovery rise time of the coin call slows down and eventually violates the VCC rise time. When the CR2032 coin cell is new this may not be an issue, but with time the internal impedance of the coin call increases so much that the transmit power pulse causes a large voltage dip. This can be confusing to debug as a coin cell - even when nearly fully discharged - can recover to 2.9 volts while the nRF52 is in reset.

Avoid this by increasing the size of the bulk capacitance on the coin cell; typically 150uF is a minimum using ceramic capacitors rated at double the coin cell voltage, 10 volt rating preferred. More is better. Ensure the nRF52 DC-DC is enabled, which can halve the required pulse current drain from the coin cell which can in turn reduce the voltage dip on transmit.

Edit: More info in these 2 posts:

nrf52810-using-cr2032-in-freezing-temps

issues-with-power-on-reset

[kaushalyasat]

Hi hmolesworth,

Thanks for this. We use OpenThread, not BLE, if that matters. We have used the power profiler to check the peak current demand on transmit periods, and didnt see any voltage drops. Having said that, the coin cells probably are newish. But the ones we have seen hanging up, the batteries are pretty much brand new. I have attached a current profile I have taken from Power Profiler.

Also the two sensors which failed in the lab, after power cycling they seem to be going on. Do you reckon if a depleted battery can show this kind of behavior for like a week?

I have tested one of these sensors. The 3V drops to about 2.8V when the TX is active and the sensor is wakeup. But I haven't seen any drops like close to 1.7V. 

I will see how to have a bulk capacitance in parallel to the VBAT, but due to space limitations, it would be tricky to get something like 150u. We use the nRF52 DC-DC regulator by 

CONFIG_BOARD_ENABLE_DCDC_HV=n

CONFIG_BOARD_ENABLE_DCDC=y

in prj.conf. 

Cheers,

Kaushalya

[hmolesworth]

OpenThread or BLE, same issue. Power Profiler II unfortunately only measures current and not voltage, although I believe this option has been requested. That means really a 'scope is required to show both current and voltage decay; no obvious voltage dips is highly improbable, if you'll pardon me suggesting that. The difference between a depleted battery and new is a much higher internal impedance in the former, leading to more problematic voltage dips for a given current spike. Ironically network traffic of unrelated devices in the immediate physical vicinity in the 2.4GHz band - BLE, OT, Wi-Fi, ... will cause greater problems as a retransmission attempt due to a transmit interfered with can effectively double the voltage dip time which deepens the dip as the coin cell voltage falls at a pretty constant rate during transmission and recovers slowly, even more so on a depleted battery.

Power Profiler II has a slow sample rate, 100KHz if I remember correctly. That's too slow to see fast spikes which can trigger the reset. Using brown-out detection or a comparator at a higher voltage level can capture progressively lower voltage drops which may occur prior to a reset, worth considering, as these can indicate the chance of a reset in the near future.

The side effect of these voltage dips even if no reset issue if using any form of analogue sensor is easily seen by taking an FFT on the data stream, where the periodic transmit pulse will show as a frequency spike at the update rate of the transmission assuming regular transmission intervals. That's a bit of a nuisance when diagnosing medical conditions from biometric sensors, for example.

What is the total capacitance on the coin cell? Are there any other circuits (maybe LEDs or flash memory) which can have current spikes coincident with a transmission (typically asynchronous spikes to transmission but periodically coinciding)? It is possible to suppress all other current pulses during a transmission pulse by using radio events.

[kaushalyasat]

Hi hmolesworth, Thanks again. Yesterday I did test the sensor which failed and as I mentioned earlier, I only saw about 0.2V drop from the oscilloscope. I dont think that is high enough to disrupt the radio behavior. I didnt see any voltage drops higher or closer to 1.7V. This is with the same battery that showed the hangup behavior earlier.

I will hookup a low power voltage detector to see if any such voltage drops can be detected.

The total capacitance on battery side is 4u7 at the moment. I can see what is the largest I can get in the same footprint so that it's just a BoM change. I will add up more bulk capacitance in the next rev of the PCB.

We have an LED that gets activated on button presses. It is not activated on any Tx cycles. When these devices lockup, most probably none of the LED or the button would have been activated.

I am thinking of running a test with a series resistor of around 86R inserted to simulate a weak battery and see if I can recreate the issue.

Cheers,

Kaushalya

[hmolesworth]

Good idea to try series resistance, though the coin cell can have an internal resistance much higher than 86R. As an aside, better to use 6.3V rating ceramics on a 3V coin cell due to the ceramic capacitor derating effect which can be as much as 50% of the rated capacitance. 4u7 is way, way too small in any case :-) Try just loading good and used coin cells with (say) 300R to get about 10mA drain and observe the coin cell voltage (which gives an indication of internal resistance) and how quickly each decays.

faq: ceramic capacitor derating

[kaushalyasat]

Thanks hmolesworth. Good point you raised to increase the bulk cap. I have obtained pulse characteristic data from the battery manufacturer and that showed that we should be able to handle the kind of pulse current demand we have for more than 1 year. So I didnt think that I would need any larger bulk capacitance . Unfortunately I don't have internal resistance data for pulse discharge for long time like an year. I will try to get that from the manufacturer.

I guess size of capacitance depends on what is the worst battery internal resistance we want to tolerate. You recon the worst would be like 300R?

Cheers,

Kaushalya

[hmolesworth]

We measured even higher than 300R on some discharged cells, I'll see if I can find our old work; another source is shown below. Some App Notes on pulse discharge of coin cells might prove useful in case you haven't come across them:

TI App Note swra349

Freescale App Note AN4573

Edit: Came across the original Nordic paper:

High-pulse-drain-impact-on-CR2032-coin-cell-battery-capacity

[kaushalyasat]

Thanks again hmolesworth, From where you get the above graph? I couldn't find that in any of the documents. From it, I can see worst case IR could go up to ~130R when the battery is near end of juice. 

I am looking at ways to add bulk capacitance. The only way I can do is multiple ceramic caps in parallel due to space restrictions. This will also reduce the ESR of the bulk capacitance. 

Cheers,

Kaushalya