GPIOTE interrupt stops firing after thousands of successful calls

[rjshade]

Hi,

We're running into a frustrating and tricky to debug issue with handling interrupts on the nrf52832. We have an active-low interrupt to pin 28, that is activated periodically (usually at 1 Hz but pushed to 1 kHz to reproduce this bug more quickly). The interrupt is handled in firmware using the nrf_gpiote library:

const bool accuracy = false;  // true = "high accuracy"
drv->gpiote_cfg = (nrf_drv_gpiote_in_config_t)GPIOTE_CONFIG_IN_SENSE_HITOLO(accuracy);
drv->gpio_interrupt_pin = gpio_interrupt_pin;
drv->gpio_interrupt_fired = false;

// Initialize the GPIO interrupt and start handling interrupts.
const ret_code_t err_code =
    nrf_drv_gpiote_in_init(drv->gpio_interrupt_pin,
                           &drv->gpiote_cfg,
                           &interrupt_handler);
ASSERT(err_code == NRF_SUCCESS);

nrf_drv_gpiote_in_event_enable(drv->gpio_interrupt_pin, true);

and the interrupt handler looks like:

void interrupt_handler(nrf_drv_gpiote_pin_t pin,
                       nrf_gpiote_polarity_t action) {
  (void)pin;
  (void)action;
  s_drv->gpio_interrupt_fired = true;
}

In normal operation the interrupt_handler code is called on every interrupt, and we do some i2c operations in response. This works fine, for a while. After some time, ranging from a few seconds to a few minutes, the interrupt_handler just stops being called -- and we have verified on a scope that the interrupt line is going low as expected:

The only way we have been able to make this work consistently is to change the accuracy argument to GPIOTE_CONFIG_IN_SENSE_HITOLO(accuracy) from false to true. Putting it in high accuracy mode seems to make things work reliably (but perhaps just masks the failure/makes it much less likely).

We are using the nRF SDK 14.0.0.
Reading the documentation about low vs high accuracy (infocenter.nordicsemi.com/index.jsp it doesn't seem like low accuracy should ever result in us missing an interrupt entirely, rather there is just less accurate timing about when the interrupt fires.

Any help or debugging advice would be appreciated!

Thanks,

Robbie

[rjshade]

Update: this does seem to be a bad interaction between two peripheral interrupts. We have two sensor interrupt lines connected to separate GPIO pins. Both of these are configured as active-low, low-accuracy GPIOTE interrupts. Everything seems to work until *both of these pins go low at the same time* and then we appear to get into a state where we never again get interrupted.

Thanks to hmolesworth for the periodic timer idea: I added a 2 Hz timer in our main loop that just reads pin state and logs it. I also added logging inside the IRQ handlers for sensors A and B:

...
3.807 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.809 sensor_B.c:34 Sensor B IRQ: sensor_a: 1, sensor_b: 0
3.897 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.903 sensor_A.c:175 Sensor A IRQ: sensor_a: 0, sensor_b: 1
3.908 main.c:147 MAIN: sensor_a: 1, sensor_b: 1
3.910 sensor_1.c:175 sensor_a: sensor_a: 0, sensor_b: 1
3.916 sensor_1.c:175 sensor_a: sensor_a: 0, sensor_b: 1
4.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
4.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
5.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
5.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
6.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
6.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
7.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
7.908 main.c:147 MAIN: sensor_a: 0, sensor_b: 0
8.408 main.c:147 MAIN: sensor_a: 0, sensor_b: 0

As expected, inside the interrupt handler for Sensor A we see that Sensor A's GPIO pin value is low, and B's is high. Vice versa in Sensor B's interrupt handler. You can then see that from timestamp 4.408 onwards both GPIO pins are low, but our interrupt handlers never get called.

[rjshade]

Pinging this thread as it hasn't seen any activity, but "a support engineer has been assigned to your ticket" 12 days ago.

Any thoughts from Nordic on what may be the problem here?

[Kenneth]

Hi, I believe it is a race conditions by having two input pins both configured with wakeup on port event, and if both go low at the same time, you may end up in a deadlock until both are high again. Having a timer to handle this, or possible better, do a check before you return to sleep or exit your interrupt handler that both pins are high may help.

[charles]

Thanks for the response, Kenneth. Could you point us to anything in the nRF52832 datasheet or nRFSDK v14.0 docs that describe this behavior in more detail? Is there an errata we missed somewhere?

The gpiote API seems to encourage multiple input pin port events (it allows you to register up to 4 by default), but it sounds like using that in a straightforward manner is guaranteed to fail in this not-uncommon scenario where multiple events happen in a short timespan.

[Kenneth]

Hi, the behavior is documented in the datasheet:
http://infocenter.nordicsemi.com/topic/com.nordic.infocenter.nrf52832.ps.v1.1/gpiote.html?cp=2_1_0_20#concept_knz_kww_lr 

The problematic behavior here is this: "Putting the system into System ON IDLE while DETECT is high will not cause DETECT to wake the system up again. Make sure to clear all DETECT sources before entering sleep."

So before entering sleep make sure that both signals are high.