GPIOTE interrupt stops firing after thousands of successful calls

RE: GPIOTE interrupt stops firing after thousands of successful calls

Kenneth — Sun, 13 May 2018 11:20:24 GMT

Hi, the behavior is documented in the datasheet:
http://infocenter.nordicsemi.com/topic/com.nordic.infocenter.nrf52832.ps.v1.1/gpiote.html?cp=2_1_0_20#concept_knz_kww_lr

The problematic behavior here is this: "Putting the system into System ON IDLE while DETECT is high will not cause DETECT to wake the system up again. Make sure to clear all DETECT sources before entering sleep."

So before entering sleep make sure that both signals are high.

RE: GPIOTE interrupt stops firing after thousands of successful calls

charles — Wed, 09 May 2018 15:56:39 GMT

Thanks for the response, Kenneth. Could you point us to anything in the nRF52832 datasheet or nRFSDK v14.0 docs that describe this behavior in more detail? Is there an errata we missed somewhere?

The gpiote API seems to encourage multiple input pin port events (it allows you to register up to 4 by default), but it sounds like using that in a straightforward manner is guaranteed to fail in this not-uncommon scenario where multiple events happen in a short timespan.

RE: GPIOTE interrupt stops firing after thousands of successful calls

Kenneth — Wed, 09 May 2018 11:13:54 GMT

Hi, I believe it is a race conditions by having two input pins both configured with wakeup on port event, and if both go low at the same time, you may end up in a deadlock until both are high again. Having a timer to handle this, or possible better, do a check before you return to sleep or exit your interrupt handler that both pins are high may help.

RE: GPIOTE interrupt stops firing after thousands of successful calls

rjshade — Tue, 08 May 2018 19:35:22 GMT

Pinging this thread as it hasn't seen any activity, but "a support engineer has been assigned to your ticket" 12 days ago.

Any thoughts from Nordic on what may be the problem here?

RE: GPIOTE interrupt stops firing after thousands of successful calls

rjshade — Mon, 30 Apr 2018 20:29:59 GMT

Update: this does seem to be a bad interaction between two peripheral interrupts. We have two sensor interrupt lines connected to separate GPIO pins. Both of these are configured as active-low, low-accuracy GPIOTE interrupts. Everything seems to work until *both of these pins go low at the same time* and then we appear to get into a state where we never again get interrupted.

Thanks to hmolesworth for the periodic timer idea: I added a 2 Hz timer in our main loop that just reads pin state and logs it. I also added logging inside the IRQ handlers for sensors A and B:

...
3.807 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.809 sensor_B.c:34 Sensor B IRQ: sensor_a: 1, sensor_b: 0
3.897 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.903 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.908 main.c:147 MAIN: sensor_a: 1, sensor_b: 1
3.910 sensor_1.c:175 sensor_a: sensor_a: 0, sensor_b: 1
3.916 sensor_1.c:175 sensor_a: sensor_a: 0, sensor_b: 1
4.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
4.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
5.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
5.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
6.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
6.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
7.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
7.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
8.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0

As expected, inside the interrupt handler for Sensor A we see that Sensor A's GPIO pin value is low, and B's is high. Vice versa in Sensor B's interrupt handler. You can then see that from timestamp 4.408 onwards both GPIO pins are low, but our interrupt handlers never get called.

RE: GPIOTE interrupt stops firing after thousands of successful calls

rjshade — Mon, 30 Apr 2018 18:37:01 GMT

This unanswered post from a year ago sounds like it is describing the same sort of problem:

devzone.nordicsemi.com/.../gpiote-simultaneous-interrupts

RE: GPIOTE interrupt stops firing after thousands of successful calls

charles — Mon, 30 Apr 2018 16:17:51 GMT

That's very interesting, and could possibly explain what we're seeing; thank you for responding! I'm curious to hear what Nordic support thinks...

RE: GPIOTE interrupt stops firing after thousands of successful calls

koniho — Fri, 27 Apr 2018 19:15:07 GMT

In low-accuracy mode - only a single interrupt is generated for all inputs. Typically, the event handler would have to determine which signal triggered the GPIOTE. What if another unrelated GPIOTE event was triggered right before the GPIOTE of interest was generated? Would that result in the the GPIOTE of interest being missed forever?

I think we might be running into a similar issue ...

RE: GPIOTE interrupt stops firing after thousands of successful calls

charles — Wed, 25 Apr 2018 21:49:03 GMT

A periodic app_timer is triggering the TWI write. The peripheral asserts a pin that connects to the nRF52832, which we see in "interrupt goes low". The pin is only de-asserted when we issue a subsequent TWI read to read the results, so it's effectively a latching mechanism.

It's very interesting to us that setting high accuracy mode on the GPIOTE configuration "fixes" this, and we never see any "missed" interrupts. This led to the question of the data sheet: we expect low-accuracy GPIOTE events to miss pulse functions, but will we ever miss simple step functions?

RE: GPIOTE interrupt stops firing after thousands of successful calls

hmolesworth — Wed, 25 Apr 2018 21:04:57 GMT

Perhaps a race hazard, even though the transactions are slow .. what is driving the 'scope trace? Is s_drv volatile? How is it related to drv? Often even if this behaviour is never observed, a periodic test doesn't hurt, typically driven by an independent timed event:

        // Test for interrupt signal from active-low /DRDY signal
        if (!mAfeSampleDataReady)
        {
            // Check for missed interrupt, perhaps due to hardware race hazard
            if (TimeSinceLastPacket > 10)
            {
                // Looks suspicious, check if the signal indicates data ready
                if (nrf_gpio_pin_read(ADS_DRDY_PIN) == 0)
                {
                    // Set sample pending signal, as it should have been set
                    mAfeSampleDataReady = true;
                    // Update error log
                    mAfeLateDrdyCount++;
                }
                else
                {
                    // Looks like AFE has ceased streaming, try to recover it
                    amInitialized = false;
                    // Update error log
                    mAfeRestartCount++;
                }
            }
        }