Large delay when using app scheduler in GPIOTE interrupt

RE: Large delay when using app scheduler in GPIOTE interrupt

Einar Thorsrud — Fri, 21 Feb 2020 12:56:57 GMT

Hi,

[quote user="rolandash"]So the question is scheduler event not able to wake up CPU? or it is because the GPIOTE interrupt handler somehow put CPU back to sleep again after completion?[/quote]

The schedule is just a queue that you pop in the main loop, so it does not wake the CPU from sleep. However, that is not necessary either. In this case, you call it from an interrupt/event handler, so the CPU is awake. Therefore, it will run until it executes sd_app_evt_wait(). And similarly, after any events/interrupts are processed, the main/thread will run from the last call to sd_app_evt_wait() (or WFE/WFI if called directly).

Do you call sd_app_evt_wait(), WFE or WFI from anywhere other than a single place in your main loop (via idle_state_handle and possible nrf_pwr_mgmt_run())? If so, that could explain this behaviour.

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Fri, 21 Feb 2020 07:04:17 GMT

Another related problem about GPIOTE confused me during the test is, can scheduler event wake CPU up or not?

To isolate the delay issue, I built a new app from scratch, with the same GPIOTE handling model as original app, but without irrelevant timers and other components.

as shown above in my test application, the main loop is like:

    // Enter main loop.
    for (;;)
    {
        //handle usbd event sperately;
        //app_usbd_event_queue_process();
        
        app_sched_execute();
        if (NRF_LOG_PROCESS() == false)
        {
			idle_state_handle();
        }
    }

this is the regular recommended routine. It is expected when GPIOTE interrupt happening and the event is issued, the CPU wakeup from the sleep mode and process the event.

However, I found that after my GPIOTE interrupt handler put the event into queue, the CPU seems go back again into sleep, instead of processing the event. So the program just stuck there after GPIOTE interrupt happened for one time.

The event will only get processed if I commented out the

//idle_state_handle();

line, but now this means CPU never go to sleep.

So the question is scheduler event not able to wake up CPU? or it is because the GPIOTE interrupt handler somehow put CPU back to sleep again after completion?

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Fri, 21 Feb 2020 06:49:40 GMT

thanks, I will look into it.

RE: Large delay when using app scheduler in GPIOTE interrupt

hmolesworth — Thu, 20 Feb 2020 16:33:43 GMT

Maybe not directly relevant, but have a look at this nrf_log_frontend_dequeue-must-be-atomically-protected-against-re-entry-from-interrupt-context issue, where I just posted some code following the suggestions by the reporter

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Thu, 20 Feb 2020 13:40:51 GMT

Hi,

As you suggested, I took some screenshots from a oscilloscope, toggling a GPIO pin.

First one is taken when nrf_log is on (I use deferred log)

Second one is taken when nrf_log is totally disabled.

As you can see when log is on, the delay is between 500us ~ 2.5ms,

and when log is off, the delay is ~ 4us.

I have to say that this result surprised me, as I did not expect logging can make such big difference, even in deferred mode. Is it normal? or did I miss some configuration to make it correct?

RE: Large delay when using app scheduler in GPIOTE interrupt

Einar Thorsrud — Wed, 19 Feb 2020 13:43:37 GMT

[quote user="rolandash"]Any other thing I can try?[/quote]

I don't see why you get such a large number this way, but anyway, the LFCLK is very slow compared to the CPU clock, so you will not get an accurate measurement this way. Can you try simply toggling a GPIO in gpiote_irq_handler() and data_event_dispatch() and measure the time using a logic analyzer or an oscilloscope? That would give you a definitive answer about the time, removing any doubt about how it is measured.

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Wed, 19 Feb 2020 13:38:49 GMT

Hi,

I use following method, in GPIOTE IRQ handler, record the current ticks into data event,

void gpiote_irq_handler(nrf_drv_gpiote_pin_t pin, nrf_gpiote_polarity_t action)
{
    
    if (pin == EXT_DATA_IRQ_PIN) {
     

        ex_data_event_t evt;
        //fill in data evt with current ticks;
        evt.cur_ticks = app_timer_cnt_get();
        ....
        app_sched_event_put(&evt, sizeof(evt), data_event_dispatch);

    }
    
}

and then in scheduler event handler, calculate the delay,

void data_event_dispatch(void *p_event_data, uint16_t event_size) {
    ex_data_event_t* p_evt = (ex_data_event_t*)p_event_data;
    ...
    case EXT_DATA_OPS_TYPE_IRQ:
        {
            //calc delay;
            uint32_t cur_ticks = app_timer_cnt_get();
            uint32_t elapsed = cur_ticks - p_evt->cur_ticks;
            float ms = elapsed / 32.768;
            NRFX_LOG_DEBUG("\t call IRQ handler....after "NRF_LOG_FLOAT_MARKER, NRF_LOG_FLOAT(ms));    
        }
        break;
    ...

}

The printed result varies between 1~2ms.

Today I did more tests, basically disabled any irrelevant things, like timers, ble activities, other GPIOTE channels, but the result just remains the same :-(

Any other thing I can try?

RE: Large delay when using app scheduler in GPIOTE interrupt

Einar Thorsrud — Wed, 19 Feb 2020 13:02:38 GMT

Hi,

I do not see how it should be related. How do you measure the time from you call app_sched_event_put() and app_sched_event_put() runs? Could it be a problem with your time measurement, rather than that it actually takes that long time?

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Mon, 17 Feb 2020 17:15:58 GMT

One thing I forgot to mention is, I have an external PA(RF2411) connected to 52840, so it is also using GPIOTE channel to switch between PA/LNA.

52840 is also advertising while the code is running. Do you think it is possible that the PA/LNA GPIOTE task could be the reason of this delay?

RE: Large delay when using app scheduler in GPIOTE interrupt

rolandash — Mon, 17 Feb 2020 14:52:23 GMT

thx for the swift reply @Einar Thorsrud. this also seems the same strange to me.

I could not think of what the processor is really doing during this 2ms. I added log at the end of gpiote_irq_handler and the first beginning of app_sched_execute, this delay appears exactly between every GPIOTE interrupt ends and the beginning of app_sched_execute. And this delay does not happen whenever app_shed_event_put is not called from GPIOTE interrupt, for example, a timer interrupt context.

I did have few other timers defined, but the intervals are all above 30ms, so seems not possible this inteference can happen in this accuracy.

I tried to tap into the scheduler queue to see if any other events blocked there. From the output I can see when this delay happens there is only 1 event in queue, i.e., the GPIOTE interrupt generated event itself. So the queue is really quite empty.

I also tried to config different GPIOTE/SPIM priority, but the result is just the same.

I tend to guess this delay is somewhat related with GPIOTE module specificly. Do you have any suggestions on how to debug further that I can find out what CPU is really doing in this 2ms delay?

My setup:

SDK v15.2 with softdevice v610 presented.

RE: Large delay when using app scheduler in GPIOTE interrupt

Einar Thorsrud — Mon, 17 Feb 2020 13:54:27 GMT

Hi,

The app scheduler is just a queue that you pop ad process in the main loop. Your main loop does not do much else, and the queue is processed before logs (assuming you use deferred logging), so I do not see an explanation there. Could it be caused by other interrupts that are processed at the same time, and that takes some time? (they happen at the same time for some reason)?