High current state that persists across SW reset?

RE: High current state that persists across SW reset?

Vidar Berg — Mon, 31 Jan 2022 09:57:37 GMT

Hi Mark,

I appreciate you taking the time to post this update! It's good hear you found the problem, and that it turned out to be something you could resolve through a FW update.

[quote user="bloecmo1"] I implemented the C_DEBUGEN check, and on the board I tried it on at least the bit did clear without a power-cycle after I disconnected the debugger.[/quote]

The ARM documentation stated that it is not reset on a system reset, but by a power-on-reset. However, when testing it here, I observe that pinreset had the same effect. Maybe you used pinreset when disconnecting the debugger, and not a soft reset (i.e. debug reset).

Best regards,

Vidar

RE: High current state that persists across SW reset?

Mark — Sat, 29 Jan 2022 02:23:33 GMT

Hi Vidar,

It looks like we've found the culprit: the malfunctioning tags came from an early prototype run, which had an external flash IC that we've since not been populating. The firmware wasn't doing anything with the pins--they were floating, in fact--and it appears that under certain circumstances the flash is entering into a high current state. I'm not 100% sure of the mechanism, since it primarily occurs in a couple of locations--maybe it is something with the magnets. In any case, we FOTA'd a version of code that sets the pins for the flash properly and the batteries on the affected tags are recovering.
Thanks so much for your assistance; I'm planning on keeping some of your suggestions in the code (e.g. having the tags report the time spent in idle).
One random thing to report: I implemented the C_DEBUGEN check, and on the board I tried it on at least the bit did clear without a power-cycle after I disconnected the debugger.

Thanks again!
Mark

RE: High current state that persists across SW reset?

Vidar Berg — Tue, 25 Jan 2022 20:28:05 GMT

Hi Mark,

It's good to have confirmed that it does indeed enter sleep. Then there shouldn't be any unhanded interrupts/events sources in your code as you said.

Regarding magnetic fields, I have not been able to find other reports where strong fields have been shown to have had adverse effects on 52 series devices. Martin answered a somewhat related question in this thread: https://devzone.nordicsemi.com/f/nordic-q-a/33295/does-the-nrf52832-have-ferromagnetic-components.

[quote user="bloecmo1"]if somehow the debug port was activated I think the tag could still sleep, so if there is a way for the application to tell that debug is active--or a way for the application to preemptively clear it even without checking--I'd still like to know[/quote]

Yes, the device will be able to enter sleep (System ON mode) in debug mode, but with an elevated sleep current. I just measured it to around 1.5 mA here on my desk with a nRF52833DK. So maybe a bit low to fully explain discharge curve?

It can be difficult to check have the FW determine whether the chip is in debug interface mode or not. You have the C_DEBUGEN bit in CoreDebug->DHCSR (link), but this register is only reset after a power-on reset (i.e. it will remain set after the debugger has exited debug mode), so it only makes sense to check this status bit if your devices have been power-cycled after they where programmed. And It's not possible to exit debug interface mode programmatically.

I have been discussing this case with some of my colleagues to try come up with ideas. Will let you know if we can think of anything else to try.

Best regards,

Vidar

RE: High current state that persists across SW reset?

Mark — Tue, 25 Jan 2022 16:52:50 GMT

Hi Vidar,

I've ID'd two tags that started exhibiting the issue last night. This morning we FOTA'd both tags to a version that tracks and uplinks the percentage of time they spend in the sleep state. Both tags are reporting that they're spending >99% of their time asleep (I realized too late that my code truncates the decimal instead of rounding, so I'm guessing it's closer to 100% than 99%). So I don't think there's a rogue or unhandled interrupt keeping the processor awake. Notably, the error state also persisted through the FOTA process, which entails several resets.
If you can think of any other register values that might help inform what's going on, please let me know ASAP and we can program the tag to uplink them before it dies--which will probably be in the next week or so.
I think this obviates my question about polling the state of the debug interface, but I am still curious about any possible effects from cutting through strong magnetic fields. I think both tags got into this state when they were placed in a rack with other tags (which are mounted to the flags that have strong magnetic bases).

EDIT: if somehow the debug port was activated I think the tag could still sleep, so if there is a way for the application to tell that debug is active--or a way for the application to preemptively clear it even without checking--I'd still like to know

Thanks!
Mark

RE: High current state that persists across SW reset?

Mark — Mon, 24 Jan 2022 21:01:35 GMT

Hi Vidar,

Two other questions I just thought of:

Is there a way for the application firmware to know if the debug interface is active? E.g. a status register?
We were brainstorming here and one question that came up: it's possible this issue manifests when the flags and tags are placed on a rack of other flags. When that occurs, the tag may cut through some relatively strong magnetic fields--do you know if it's possible for that to cause unexpected behavior on the processor? I've tried replicating it here and haven't seen anything.

Thanks!

Mark

RE: High current state that persists across SW reset?

Mark — Mon, 24 Jan 2022 19:30:15 GMT

Hi Vidar,

Yes, I think I can have them report idle time. As soon as I catch one malfunctioning I'll FOTA it with a version that uplinks that.

Thanks!
Mark

RE: High current state that persists across SW reset?

Vidar Berg — Mon, 24 Jan 2022 15:42:25 GMT

hmolesworth has a good point regarding FPU. But the problem is that should not have triggered again after reset, not by itself at least. The interrupt pending register in NVIC is (i.e. NVIC->ICPR[n]) cleared on reset: https://developer.arm.com/documentation/ddi0439/b/Nested-Vectored-Interrupt-Controller/NVIC-programmers-model so there would have to be an event source (e.g. FPU) that triggers the same interrupt again on every reboot until the next power on reset.

[quote userid="42643" url="~/f/nordic-q-a/83893/high-current-state-that-persists-across-sw-reset/349325#349325"]I don't have the __DSB() instructions though--are they required?[/quote]

I believe the dummy read ("(void) __get_FPSCR();) will have the same effect as the __DSB() instruction here. If not, it should at least become cleared on the next iteration.

Here is how we clear FPU events in in our Power Management library:

#if (NRF_PWR_MGMT_CONFIG_FPU_SUPPORT_ENABLED && __FPU_PRESENT)
    #define PWR_MGMT_FPU_SLEEP_PREPARE()     pwr_mgmt_fpu_sleep_prepare()

     __STATIC_INLINE void pwr_mgmt_fpu_sleep_prepare(void)
     {
        uint32_t original_fpscr;

        CRITICAL_REGION_ENTER();
        original_fpscr = __get_FPSCR();
        /*
         * Clear FPU exceptions.
         * Without this step, the FPU interrupt is marked as pending,
         * preventing system from sleeping. Exceptions cleared:
         * - IOC - Invalid Operation cumulative exception bit.
         * - DZC - Division by Zero cumulative exception bit.
         * - OFC - Overflow cumulative exception bit.
         * - UFC - Underflow cumulative exception bit.
         * - IXC - Inexact cumulative exception bit.
         * - IDC - Input Denormal cumulative exception bit.
         */
        __set_FPSCR(original_fpscr & ~0x9Fu);
        __DMB();
        NVIC_ClearPendingIRQ(FPU_IRQn);
        CRITICAL_REGION_EXIT();

        /*
         * The last chance to indicate an error in FPU to the user 
         * as the FPSCR is now cleared
         *
         * This assert is related to previous FPU operations 
         * and not power management.
         *
         * Critical FPU exceptions signaled:
         * - IOC - Invalid Operation cumulative exception bit.
         * - DZC - Division by Zero cumulative exception bit.
         * - OFC - Overflow cumulative exception bit.
         */

Would it be possible to get the devices to report the idle time similar to how you are getting the voltage and distance readings?

Edit: hmolesworth, I'm not sure what MPU is in this context. Did you mean NVIC? Thanks.

RE: High current state that persists across SW reset?

Mark — Mon, 24 Jan 2022 14:58:40 GMT

Thanks for the suggestions! I do have the fpu irq clear before __WFE() in there already:

void _fpu_irq_clr(void)
{
    uint32_t fpscr_reg = __get_FPSCR();

    __set_FPSCR(fpscr_reg & ~(0x0000009F));
    (void) __get_FPSCR();
    NVIC_ClearPendingIRQ(FPU_IRQn);
}

I don't have the __DSB() instructions though--are they required?
The MPU reset/event clear register you mentioned would be great--I can add it and FOTA a tag when it malfunctions, and if that recovers it that would be an indicator that an unhandled event is keeping the processor awake.

RE: High current state that persists across SW reset?

hmolesworth — Sat, 22 Jan 2022 19:02:40 GMT

Longshot: the symptoms are identical to those of an unhandled event, typically caused by the FPU which are not necessarily cleared by a software reset. The effect is that the call to sd_app_evt_wait() returns immediately so the CPU never sleeps. This can be confirmed by putting a port pin driven low on entry to sd_app_evt_wait() and high on return; if it is mostly high it shows a pending event is preventing sleep. 3mA is suspiciously close to the MPU 5mA or so assuming the DC-DC convertor is running so battery consumption is approx 5mA * (1.3/3). Trouble is that assumes you can recreate the problem in the lab.

There is a secret power on/off register for the MPU (I forget the address but could find it if needed), which clears the pending events, or the pending events can just be cleared on entry to sleep (can check if set or just don't care, clear anyway). Quite how the pending event got set is irrelevant, though if you are using the hardware floating point it may be some extreme calculation caused by physical shock on the accelerometer. If not using the h/w FPU then some other peripheral could do the same, though most peripherals are cleared by a soft reset (not FPU).

// Floating-point Status Control Register - Cortex-M4 FPU
// ======================================
// 7 - IDC Input Denormal cumulative exception bit, see bits [4:0].
// 6 - Reserved
// 5 - Reserved
//   IXC Cumulative exception bits for floating-point exceptions, see also bit [7]. Each of these bits is
//   set to 1 to indicate that the corresponding exception has occurred since 0 was last written to it
// 4 - IXC Inexact cumulative exception
// 3 - UFC Underflow cumulative exception
// 2 - OFC Overflow cumulative exception
// 1 - DZC Division by Zero cumulative exception
// 0 - IOC Invalid Operation cumulative exception
// Set bit 7 and bits 4..0 in the mask to one (0x ...00 1001 1111)
#define FPU_EXCEPTION_MASK 0x0000009F

    // Clear any exceptions and PendingIRQ from the FPU unit
    __set_FPSCR(__get_FPSCR()  & ~(FPU_EXCEPTION_MASK));
    __DSB();
    NVIC_ClearPendingIRQ(FPU_IRQn);
    __DSB();
    sd_app_evt_wait();

Edit: There is a sdk_config.h setting which might do this handling for you, if needed (I haven't tried it):

NRF_PWR_MGMT_CONFIG_FPU_SUPPORT_ENABLED - Enables FPU event cleaning

RE: High current state that persists across SW reset?

Mark — Fri, 21 Jan 2022 15:20:21 GMT

On the battery position: the bad luck I was referring to was that I received some malfunctioning tags, but they were dropped off late and sat on my porch overnight in very cold weather, which caused the batteries to droop to 1.5V. After I brought them in the voltage rose to 2.6V on some, but they didn't kick back on. I don't have access to the reset lines, so I very briefly shorted the battery--without shifting its position--and they came back and the current looked OK (evidenced by the battery voltage continuing to rise over time). So if it is something related to battery position, it seems to be a transient event that in turn causes a persistent state.
We're not using system OFF.
The persistence across reset is really stumping me too. That and the fact that the tag appears to be running perfectly normal otherwise--timing, location updates, etc.--makes me feel like it's not just a bug keeping the processor awake. And while I wouldn't say the issue is common, it's also not exceptionally rare--out of the 40 or so active tags there's anywhere from 1-5 that are exhibiting it at any given moment. Also interesting: I don't think we've seen any tags that have had the issue recur after replacing the batteries, which again would imply some transient triggering event.
We do have the ability to remotely FOTA these tags, so if there's anything we can try let me know.
One thought: is there a way to emulate a power down reset in terms of register clearing/device state? It might help narrow down the problem, and until we figure out the issue it would at least be nice to be able to recover tags without having to physically touch them.

RE: High current state that persists across SW reset?

Vidar Berg — Fri, 21 Jan 2022 14:20:05 GMT

Hello,

This is pretty strange - the device is operating normally, does not use the I2C, and also does not recover through a soft reset. Maybe it could be the position of the battery even though it seems unlikely.

As you indicated, peripherals and interrupt states will generally not persist across soft resets. One exception is the debug interface (see Debug Interface mode), but I think you have to be quite unlucky to get it enabled by some random clock pulses ( assuming the bottom side of the coincell can come into contact with the pads - the SWD lines have internal pull-ups).

Do you use System OFF mode in your application? I'm asking because it's not possible to enter this mode while the debug interface is enabled.

Best regards,

vidar

RE: High current state that persists across SW reset?

Mark — Thu, 20 Jan 2022 22:17:41 GMT

One other note: in the units I've looked at, the battery is shifted to some extent from the holder, but there's a foam block that prevents it from moving too far. It does imply that the tags are seeing some kind of shock--enough to shift the battery, at least.