nRF52840 app not sleeping, SoftDevice seems involved

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Thu, 24 Nov 2022 12:30:30 GMT

Hi Charles,

Thanks for the info. It's a good practice to avoid complex activities or blocking functions inside a interrupt handler. But I assume this doesn't explain the interrupt histogram you got.

Let us know if you still see the issue occurs.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Wed, 23 Nov 2022 22:25:34 GMT

Update-

We were doing a lot of work (blocking TWI / SPI transactions etc) inside of our SoftDevice event handler, which happens from inside the SWI2/EGU2 interrupt handler.

We changed our code to simply enqueue inbound writes to a queue in our handler, and our main loop drains the queue and processes from the thread context, with the rest of our logic.

I'm not sure exactly what caused the system from sleeping, but it might have been us doing something "bad" from inside the SWI2 interrupt observer callback.

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Tue, 15 Nov 2022 15:52:00 GMT

Hi Charles,
I don't see a problem calling sd_ble_gap_disconnect () from a BLE event handler. Unless you changed the default priority.

For example in most of our examples you can find that we do this ( ble_app_hrs example):

You can have a look at this statement here:

So as long as you don't call the softdevice API from an interrupt priority at higher level than 4 (at level 2 or 3) you should be fine.
By default SWI2 interrupt has interrupt priority level 6.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Mon, 14 Nov 2022 19:06:42 GMT

Hello! Maybe a clue-

It seems to start happening right around when I disconnect from BLE. I notice that I'm calling sd_ble_gap_disconnect from my event handler, which is called by the nRFSDK observer, which I believe happens from a SWI2 interrupt. I haven't changed any of the interrupt priorities; is it safe to call arbitrary SoftDevice functions from inside an observer?

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Thu, 10 Nov 2022 12:30:04 GMT

I see, thanks for the response. I'll keep collecting data and searching; I'll update this thread with any evidence or questions as they arise.

Thanks also for continuing to engage with me on this, I appreciate it!

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Thu, 10 Nov 2022 12:28:35 GMT

Hi Charles,

The problem of not having a spinlock loop is that there is a chance that in your functions that is executed before the sd_app_evt_wait() or __WFE, you may trigger an interrupt.
This interrupt may wake the chip up immediately after sd_app_evt_wait/WFE is called. If there is a spinlock, the loop continue and the next WFE at the beginning of the next loop will put the device to sleep as the event register is cleared by the WFE in the last loop.

If there is no spinlock loop (a loop checking for a app_wakeup flag) the CPU will just jump to your poll functions which may trigger another interrupt.

You may run to this issue if you process a UART logging right before the call. There can be an interrupt after you finish transmitting UART and that can wake the chip up. But if you already monitor all interrupts you should be able to spot that.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Wed, 09 Nov 2022 16:40:47 GMT

We have 2x UARTs in our system.

1 is for logging data from our external WiFi chip, but we disable the UART when WiFi is inactive (and verify this by dumping the GPIO pin states + directions to the same server that creates the IRQ histograms). I'd love to hear your thoughts, though- are you thinking that the UART interrupts might be keeping the system awake or something?

Another UART is for factory-mode tests between nRF52840 and nRF9160 (LTE). In the field we use the TX/RX for SPI instead, and never initialize this UART peripheral.

It's absolutely possible that something in my poll functions triggers an interrupt, especially since starting and stopping timers triggers an interrupt in nRFSDK. But- If anything in my main loop triggered an interrupt , I would see it on my IRQn histogram, right? Is there a scenario where I wouldn't?

I clear SEVONPEND once, in thread level, from my main function, early after boot.

What happens if I don't have WFE/SEV/WFE in a loop? Worst case, I run my main loop again, right?

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Wed, 09 Nov 2022 14:43:34 GMT

Hi Charles,

There is a mentioning about SEVONPEND in the softdevice specification at section 7.6:

I am not aware that softdevice explicitly set SEVONPEND => After check the source I saw that the softdevice may set/clear it , but it will return to the value has been set by the application after it's done.

I assume you cleared the bit on low interrupt level or in thread level ?

If you think sd_app_evt_wait() could be the culprit you can try using:

__WFE();

__SEV();

__WFE();

ADDED: Please make sure you use this in a spinlock loop as Vidar has suggested.

It would have the same effect (except that the function returna when there is any softdevice activity)

Unlike what you think of sd_app_evt_wait(), the spin for sleeping inside this function is not executed in the SVC context but in main context.

Have you made sure the poll_a();poll_b(); doesn't trigger any interrupt in a corner case or something ?

Just to be extra sure, you don't use any UART logging ?

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Tue, 08 Nov 2022 15:06:44 GMT

The difference between "mainLoopTurnCount" and "mainLoopSleepEnterCalls" is due to us realizing that because of the order of operations in our main loop, we can't sleep yet, roughly like this:

void main_loop(void) {
  increment_counter(mainLoopTurns);
  
  poll_a();
  poll_b(); // can provide immediate work for poll_a() to do!
  
  if (idle) { // no immediate work to do
    increment_counter(mainLoopSleepEnterCalls);
    sd_app_evt_wait();
  }
}

We have RTT logging going, but of course there's no debugger attached in the field 🙂

I collect this histogram from all of our employee modules, and I always see this pattern when the battery collapse happens. Sometimes it heals itself and stops, other times the battery is drained to exhaustion.

I agree that waking up at 2khz is really wild! I'm also extremely confused that I'm not seeing any of the histogram counts growing 1:1 with the main loop / sleep counts. I also agree that the SVCALL IRQn histogram growing almost 1:1 makes perfect sense- every call to sd_app_evt_wait() does an SVC, and then we have a few more for normal SD API calls that do the same.

I have never been able to reproduce this at my bench; it only happens in the field, and it always starts when the device receives a connection from a BLE central (phone or dedicated base).

I'm running out of data to gather, and hypotheses to test, which is starting to make me nervous. If it's not an IRQ, what else could be waking the system?

Perhaps another thread to explore: I'm explicitly disabling SEVONPEND. Does anything in SoftDevice turn it back on? Could it be related to that? If the ISR associated with an IRQ never actually runs, my counting shim won't ever see it.

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Tue, 08 Nov 2022 14:01:21 GMT

Hi Charles,

Thanks for the explanation about the IRQ histogram I wasn't sure what it was earlier.

So you have a counter on each of the IRQ handler and in the main loop.

Could you explain why there is a different of the counter between MainloopTurns and mainloopSleepEntercalls ?

[quote user="charles_fi"]Note that the number of SVCALL interrupts is almost identical to the number of calls to sd_app_evt_wait(), which is also almost identical to the number of main loop turns.[/quote]

I think the above behavior is true for any application, that the number of SVCall is almost identical to sd_app_evt_wait() and to the number of the main loop turns. It's how it should be, isn't it ?

The question here is only on why it waking up so often. As I can see from the table it's about 2230 times per second.

It doesn't ring any bell for us. It's too fast to be any BLE activity. And without more detail or profiling it's not possible to tell what could be the culprit here.

I assume you only managed to record the histogram once ? If you can reproduce the issue again you either using the power profiler or can think of toggling a GPIO in the main loop so that it's easier to visualize the wake up behavior.

Do you have any other activity for example UART logging , RTT, ADC ? anything else that can involve ? Can you share the source code ? You can convert the case to a private case to share the code if needed.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Mon, 07 Nov 2022 17:02:06 GMT

(btw the IRQn shim thing isn't causing the wakeups; I wrote the IRQn shim thing in an attempt to gather data to diagnose and fix the issue :) )

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Mon, 07 Nov 2022 17:00:08 GMT

I'm incrementing a counter every time I turn through my main loop, and I'm incrementing a separate counter every time I call sd_app_evt_wait(), and I'm incrementing a counter per-IRQn (all of them, via an assembly VTOR shim that intercepts all interrupts), so I can see all of the softdevice interrupts as well.

Note that the number of SVCALL interrupts is almost identical to the number of calls to sd_app_evt_wait(), which is also almost identical to the number of main loop turns.

This data tells me that I am regularly calling my main loop, and that I am also regularly calling sd_app_evt_wait(), and I'm not blasting SoftDevice with other calls.

Please look to my first and second posts in this thread to observe the data that I've captured; perhaps something will jump out at you?

If I had some other high-frequency interrupt occurring, I'd expect to see it in my table above.

I'm using nRFSDK 17.0.2, SD140 7.0.1, nRF52840.

Also; I'm explicitly _clearing_ SEVONPEND, is it possible that disabled-but-pending interrupts are somehow contributing to wake?

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Mon, 07 Nov 2022 16:06:13 GMT

Hi Charles,
Is there any chance that it's locked up somewhere and couldn't enter sd_app_evt_wait() ?

Short interval of 7.5ms shouldn't be a problem. The BLE activity only takes from 2ms for an connection event. And if there is no BLE activity that requires the app involvement sd_app_evt_wait() will not return because of the BLE activity.

Could you give me some more info about the softdevice ? Which version are you using ?

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Mon, 07 Nov 2022 14:03:46 GMT

Thanks for the response. Because this is is very hard to reproduce, I'm looking for ways to gather more clues from units in the field.

Another question- if I have a BLE central connect and hold a very fast connection interval, 7.5ms, could that cause behavior like this? sd_app_evt_wait() seems like it doesn't return to my code from radio/timer0/rtc0 interrupts, but I'm curious if it's possible.

I'm seeing ~100k+ main loop runs per minute, is the only option left that I actually have an interrupt firing that many times (kilohertz range)? I'd expect to see it on my IRQ histogram (see initial post) but I don't.

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Mon, 07 Nov 2022 12:09:41 GMT

Hi Charles,

What I meant by "clear the event" is to set an event register to 0 when you handle the event. For example like this:

But I was wrong when thinking that not clearing it may keep the CPU to not entering sleep mode. The interrupt handler should be able to clear the interrupt source.
I don't see any problem in the code you provided. I couldn't think of anything else but an application interrupt that keep waking the CPU up.

If you can reproduce the issue and can capture a power profile it should be possible to see if the CPU could enter sleep mode at all.
And if possible to reproduce, I would suggest to test on one of our example (with additional code to control PA LNA and coex) to see if you can see the same problem.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Fri, 04 Nov 2022 15:09:47 GMT

Sorry- to be more complete here, I'm doing this:

void ble_pa_lna_enable(void) {
  if (s_util.pa_lna_enabled) {
    return;
  }

  APP_ERROR_CHECK_BOOL(s_util.radio_gpios.notification_gpio != NULL);
  APP_ERROR_CHECK_BOOL(s_util.radio_gpios.lna_gpio != NULL);
  APP_ERROR_CHECK_BOOL(s_util.radio_gpios.pa_gpio != NULL);

  s_util.pa_lna_enabled = true;
  nrf_ppi_channel_t ch_set, ch_clr;
  APP_ERROR_CHECK(nrfx_ppi_channel_alloc(&ch_set));
  APP_ERROR_CHECK(nrfx_ppi_channel_alloc(&ch_clr));
  APP_ERROR_CHECK(sd_ble_opt_set(BLE_COMMON_OPT_PA_LNA,
                                 &(ble_opt_t) {
                                   .common_opt = {
                                     .pa_lna = {
                                       .pa_cfg = {
                                         .enable = 1,
                                         .active_high = 1,
                                         .gpio_pin = s_util.radio_gpios.pa_gpio->pin,
                                       },
                                       .lna_cfg = {
                                         .enable = 1,
                                         .active_high = 1,
                                         .gpio_pin = s_util.radio_gpios.lna_gpio->pin,
                                       },
                                       .ppi_ch_id_set = ch_set,
                                       .ppi_ch_id_clr = ch_clr,
                                       .gpiote_ch_id = PA_LNA_GPIOTE_CHANNEL_ID,
                                     },
                                   },
                                 }));

  nrf_gpiote_task_configure(RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID,
                            s_util.radio_gpios.notification_gpio->pin,
                            (nrf_gpiote_polarity_t)GPIOTE_CONFIG_POLARITY_None,
                            NRF_GPIOTE_INITIAL_VALUE_LOW);

  APP_ERROR_CHECK(nrfx_ppi_channel_fork_assign(
    ch_set,
    nrf_gpiote_task_addr_get(
      nrf_gpiote_set_task_get(RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID))));

  APP_ERROR_CHECK(nrfx_ppi_channel_fork_assign(
    ch_clr,
    nrf_gpiote_task_addr_get(
      nrf_gpiote_clr_task_get(RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID))));
}

void ble_radio_notification_enable(void) {
  if (s_util.enabled) {
    return;
  }
  s_util.enabled = true;
  APP_ERROR_CHECK_BOOL(s_util.pa_lna_enabled);
  nrf_gpiote_task_enable(RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID);
}

void ble_radio_notification_disable(void) {
  if (!s_util.enabled) {
    return;
  }
  s_util.enabled = false;
  APP_ERROR_CHECK_BOOL(s_util.pa_lna_enabled);
  nrf_gpiote_task_disable(RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID);
}

This code is doing two things-

1. It's loading the desired PA/LNA pin behavior into SoftDevice using channel 0 (PA_LNA_GPIOTE_CHANNEL_ID)

2. It's forking the set and clear events into a new task using channel 1 (RADIO_NOTIFICATION_GPIOTE_CHANNEL_ID)

This way the PA set event sets my WiFi coex pin, and the LNA clear event clears my WiFi coex pin.

I don't have the WiFi coex pin set up to trigger an interrupt, because my nRF52840 firmware doesn't need to react to it; I just need the PA + LNA tasks to also set and clear this WiFi coex pin.

Note that my "disable" call just calls nrf_gpiote_task_disable here- I'm not ever explicitly clearing the PA/LNA events. Is it possible that a poorly-timed call to my ble_radio_notification_disable function is leaving the event in a state where it's preventing the nRF52840 from sleeping in sd_app_evt_wait()?

In general, is this approach reasonable and sound?

Thanks again for reading!

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Fri, 04 Nov 2022 14:21:20 GMT

Thanks for the response, I appreciate it. Unfortunately this only happens sometimes, and I can't get it to happen at my bench, so I can't use local setups like the Power Profiler Kit.

I do have real-time battery statistics I can watch that we collect for this purpose, though, and this issue is happening very rarely.

Since writing this post, I reverse-engineered the SoftDevice binary, and disassembled the functions on the path from the SVCall ISR into the implementation of sd_app_evt_wait(), I see that it loops back into the wfe call after every softdevice interrupt and only returns when a non-softdevice interrupt happens. So, I don't think it's SoftDevice's fault here!

I am curious what you mean by

"I think what important here is that if you have an interrupt that trigger an event you clear the event after you handle the interrupt. If not the CPU won't be able to enter sleep mode if the event has not been cleared."

Do you meant the PPI event here? I am using the PPI task/event system to tie the PA + LNA pins to a WiFi coex pin on a separate chip on our board. Could I ask you for more information about "clear the event" please?

Thank you again,

Charles

RE: nRF52840 app not sleeping, SoftDevice seems involved

Hung Bui — Fri, 04 Nov 2022 13:57:12 GMT

Hi Charles,

The main purpose of PPI is to not get the CPU involved in the automation of event and task. So PPI wouldn't wake the CPU up. But the event/interrupt can.
I think what important here is that if you have an interrupt that trigger and even tyou clear the event after you handle the interrupt. If not the CPU won't be able to enter sleep mode if the event has not been cleared.

Have you tried to disable the GPIO interrupt to check if it has something to do with waking up the CPU ?

The softdevice only trigger an event when there is BLE activity that require the application to handle. So It's not normal that the application get waking up on every sd_app_evt_wait(). Note that sd_app_evt_wait() will not exist when the softdevice is active to handle BLE event. It only exist when there is application interrupt or when there is BLE event.

I would also suggest to use the Power Profiler Kit to monitor the activity of the CPU so that we have better view of when the CPU waking up or if the CPU enters sleep mode at all.

RE: nRF52840 app not sleeping, SoftDevice seems involved

charles_fi — Thu, 03 Nov 2022 13:33:19 GMT

Ah, sorry, I'm going to give readers some whiplash here-

I think my hypothesis was incorrect! If I'm seeing an almost exactly 1:1 relationship between SVCALL_IRQn calls and main loop turns and sleep calls, that means that things are working ok because sd_app_evt_wait() is run via a SVCALL.

So I want to explore a different track, perhaps it's a GPIO issue. I see in section 6.9 of the nRF52840 Product Specification:

I see that "Wake-up from high or low level triggers on all pins" is a separate line item from "Trigger interrupt on state changes on any pin".

Is it possible to configure the GPIO and/or GPIOTE subsystems so that the level of a pin will cause a wake-up, and/or keep the system from sleeping? We are using PPI to drive a wifi/ble coexistence pin and it's tied to PA/LNA, does the PPI system have level-based wakes anywhere?

Note that my IRQ histogram shows that I am not getting GPIOTE_IRQn 100,000+ times per minute, so it's not a strobing pin:

I'll keep studying the data sheet, but if anyone has any information or quick answers about pin levels being able to keep the nRF52840 from entering WFE sleep, that would help me greatly 🙂

Thanks,

Charles