nRF52840 sometimes won't wake up.

RE: nRF52840 sometimes won't wake up.

david4255334 — Wed, 27 Sep 2023 05:33:52 GMT

Seems like this concludes the discussion.

Thank you guys for your help.

RE: nRF52840 sometimes won't wake up.

Kenneth — Thu, 21 Sep 2023 12:48:46 GMT

Hello,

Thanks for trying, then I guess it likely is related to requirement that Runar suggested earlier: "Before entering System OFF mode, all on-going EasyDMA transactions need to have completed. See peripheral specific chapters for more information about how to acquire the status of EasyDMA transactions." This requirement can for instance be ensured by disabling bt stack before going to system OFF, alternatively just stop advertisment and disconnect any potential link before entering system OFF.

Kenneth

RE: nRF52840 sometimes won't wake up.

david4255334 — Thu, 21 Sep 2023 09:01:33 GMT

Hi Kenneth,

After programming the nRF52840DK, and before running the test, I always switch the DK off, unplug and plug the USB cable back in.

Regarding proposed modifications, here's the results:

Mod 1:

    for (int i = 5; i >= 0; --i) {
        k_sleep(K_MSEC(1000));
        printk("Shutting down the system in %d seconds (w/ __disable_irq();)...\n", i);
    }

    // When bt_disable() is called, there is no wakeup issue.
    // bt_disable();
    
    gpio_pin_set(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, 1);
    __disable_irq();
    pm_state_force(0u, &(struct pm_state_info){PM_STATE_SOFT_OFF, 0, 0, 0});

err:
    while (1) {
        k_sleep(K_MSEC(2000));
        printk("Error. Infinite loop reached\n");
    }
}

This resulted in failure to enter OFF state. I observed "Error. Infinite loop reached" logs running very fast, without 2000 ms delay, as most likely disabled interrupts make the scheduler nonfunctional.

Mod 2:

    for (int i = 5; i >= 0; --i) {
        k_sleep(K_MSEC(1000));
        printk("Shutting down the system in %d seconds (w/NRF_POWER->SYSTEMOFF = 1;)...\n", i);
    }

    // When bt_disable() is called, there is no wakeup issue.
    // bt_disable();
    
    gpio_pin_set(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, 1);
    NRF_POWER->SYSTEMOFF = 1;

err:
    while (1) {
        k_sleep(K_MSEC(2000));
        printk("Error. Infinite loop reached\n");
    }
}

The board got stuck after several minutes from starting the test. It won't wake up.

Mod 3:

    for (int i = 5; i >= 0; --i) {
        k_sleep(K_MSEC(1000));
        printk("Shutting down the system in %d seconds (w/ __disable_irq(); and NRF_POWER->SYSTEMOFF = 1;)...\n", i);
    }

    // When bt_disable() is called, there is no wakeup issue.
    // bt_disable();
    
    gpio_pin_set(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, 1);
    __disable_irq();
    NRF_POWER->SYSTEMOFF = 1;

err:
    while (1) {
        k_sleep(K_MSEC(2000));
        printk("Error. Infinite loop reached\n");
    }
}

Same here, stuck after several minutes from starting the test.

RE: nRF52840 sometimes won't wake up.

Kenneth — Wed, 20 Sep 2023 11:24:57 GMT

Hello,

Two questions from me:

- Do you power cycle the nRF52840DK after programming the board and before executing testing? This to ensure the chip is not in debug interface mode (which is not proper system OFF, but emulated system OFF). That is kind of the only thing I can think of that may cause this to fail.

- Is there any difference if you add a __disable_irq(); before pm_state_force()? What if you use NRF_POWER->SYSTEMOFF=1; directly instead of pm_state_force().

Kenneth

RE: nRF52840 sometimes won't wake up.

david4255334 — Tue, 12 Sep 2023 08:13:25 GMT

Hi Runar,

I finally found some time to prepare a simple project running on nRF52840dk.

I was able to reproduce the wake-up issue in the setup running on the DK. Same situation here - disabling BLE before going to sleep fixes the problem.

Please find below the source code. There are only 3 files, but if you need a zip file, let me know then I can attach it as well.

If there are any new findings or better (proper?) ways to overcome this wake-up issue, then please let me know.

nrf52_wakeup_issue/src/main.c:

// nrf52_wakeup_issue/src/main.c
/* How to reproduce:
Power the DK via J3 micro USB.
After DK goes to sleep for the first time, start Arduino code and immediatelly connect THE_PIN to the Button 4 of DK.
Keep running for a while until DK won't wake up.

I did this test 4 times:
1. won't wake up after 12 min
2. won't wake up after 16 min
3. won't wake up after 3 min
4. won't wake up after 35 min

Arudino Mega board
Code simulating pressing the button after DK goes to sleep

#define THE_PIN 53

void setup() {
    pinMode(THE_PIN, INPUT);
    Serial.begin(9600);
    Serial.println("Start");
    delay(10000);
}
int counter = 0;
void loop() {
    digitalWrite(THE_PIN, LOW);
    pinMode(THE_PIN, OUTPUT);
    Serial.println("Low for 100 ms");
    delay(100);

    // Wait until the board wakes up and goes to sleep again
    pinMode(THE_PIN, INPUT);
    Serial.println("Float for 10 seconds");
    delay(10000);
}

 */
#include <stdint.h>
#include <zephyr/kernel.h>
#include <zephyr/drivers/gpio.h>
#include <zephyr/devicetree.h>
#include <hal/nrf_gpio.h>
#include <zephyr/pm/pm.h>
#include <zephyr/bluetooth/bluetooth.h>
#include <zephyr/settings/settings.h>

static const struct bt_data ad_[] = {
    BT_DATA_BYTES(BT_DATA_FLAGS, (BT_LE_AD_GENERAL | BT_LE_AD_NO_BREDR)),
    BT_DATA(BT_DATA_NAME_COMPLETE, CONFIG_BT_DEVICE_NAME, sizeof(CONFIG_BT_DEVICE_NAME) - 1),
};

static const struct bt_data sd_[] = {
    BT_DATA(BT_DATA_NAME_COMPLETE, CONFIG_BT_DEVICE_NAME, sizeof(CONFIG_BT_DEVICE_NAME) - 1),
};

void main(void) {
    gpio_pin_configure(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, GPIO_OUTPUT_HIGH); // LED3 on DK
    gpio_pin_configure(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 25, GPIO_INPUT | GPIO_PULL_UP); // BUTTON4 on DK
    nrf_gpio_cfg_sense_set(25, NRF_GPIO_PIN_SENSE_LOW);
    gpio_pin_set(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, 0);

    if (bt_enable(NULL)) {
        printk("bt_enable() failed\n");
        goto err;
    }

    if (bt_le_adv_start(BT_LE_ADV_CONN, ad_, ARRAY_SIZE(ad_), sd_, ARRAY_SIZE(sd_))) {
        printk("bt_le_adv_start() failed\n");
        goto err;
    }

    for (int i = 5; i >= 0; --i) {
        k_sleep(K_MSEC(1000));
        printk("Shutting down the system in %d seconds...\n", i);
    }

    // When bt_disable() is called, there is no wakeup issue.
    // bt_disable();
    
    gpio_pin_set(DEVICE_DT_GET(DT_NODELABEL(gpio0)), 15, 1);
    pm_state_force(0u, &(struct pm_state_info){PM_STATE_SOFT_OFF, 0, 0, 0});

err:
    while (1) {
        k_sleep(K_MSEC(2000));
        printk("Error\n");
    }
}

nrf52_wakeup_issue/CMakeLists.txt:

# nrf52_wakeup_issue/CMakeLists.txt
cmake_minimum_required(VERSION 3.20.0)
# list(APPEND BOARD_ROOT ${CMAKE_CURRENT_SOURCE_DIR})
set_property(GLOBAL PROPERTY CSTD c11)
find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
project(wakeup_test)
file(GLOB APPLICATION_SOURCES src/main.c)
target_sources(app PRIVATE ${APPLICATION_SOURCES})
target_include_directories(app PRIVATE src)

nrf52_wakeup_issue/prj.conf:

# nrf52_wakeup_issue/prj.conf
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_BT=y
CONFIG_NEWLIB_LIBC=y
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_SMP=y
CONFIG_PM_DEVICE=y
CONFIG_NEWLIB_LIBC_MIN_REQUIRED_HEAP_SIZE=2048

RE: nRF52840 sometimes won't wake up.

runsiv — Mon, 21 Aug 2023 08:00:16 GMT

Thanks for the good explantion, I have forwarded the information to the dev teams for further input.

There is one other thing we can check out. From the PS of the 52840 it states the following:

"Before entering System OFF mode, all on-going EasyDMA transactions need to have completed. See peripheral specific chapters for more information about how to acquire the status of EasyDMA transactions."

The BLE stack uses DMA, so it might be that the stars align and it's powered down during advertisment. I have talked with a few collegues regarding what will happen if this occure, and there was no clear answare.

infocenter.nordicsemi.com/index.jsp

Regards

Runar

RE: nRF52840 sometimes won't wake up.

david4255334 — Fri, 18 Aug 2023 08:19:02 GMT

Hi Runar,

I'm sorry, I did not make it clear - the code is running on our custom PCB that has nRF52840 on it. We power it from USB, and then the USB power goes through the power supply management chip to the nRF.

I'll find out if I can prepare some small code that can reproduce it. But I would need to see if this can be reproduced on the DK.

I added code that immediately reads RESETREAS register after startup and saves it in a variable for printing it later. I do not clear RESETREAS register.

#include <helpers/nrfx_reset_reason.h>
static uint32_t reason___ = 0;
reason___ = nrfx_reset_reason_get();

// Shutdown code
const int ble_disable_err = bt_disable();
if (ble_disable_err) {
    NVIC_SystemReset();
}
k_sleep(K_MSEC(1000));
PowerSupply_vSys5vDisable();
pm_state_force(0u, &(struct pm_state_info){PM_STATE_SOFT_OFF, 0, 0, 0});
k_sleep(K_SECONDS(2U));
NVIC_SystemReset();

With the code below, after wakeup, RESETREAS is equal to 0x00010000, indicating "Reset due to wake up from System OFF mode when wakeup is triggered from DETECT signal from GPIO". As expected.

Then, I run exactly the same code with only one change - remove first 4 lines that disable BLE.

// Shutdown code
//const int ble_disable_err = bt_disable();
//if (ble_disable_err) {
//    NVIC_SystemReset();
//}
k_sleep(K_MSEC(1000));
PowerSupply_vSys5vDisable();
pm_state_force(0u, &(struct pm_state_info){PM_STATE_SOFT_OFF, 0, 0, 0});
k_sleep(K_SECONDS(2U));
NVIC_SystemReset();

After many sleep-wakeup cycles, I expect the issue to occur again. I checked the logs with RESETREAS and there was one entry that instead 0x00010000 is showing 0x00010002, that indicates the reset from watchdog, which confirms what I wrote in the previous post.

This is why in the old version of the code, without watchdog configured, the board was bricked. It somehow got stuck and without watchdog nothing could be done.

RE: nRF52840 sometimes won't wake up.

runsiv — Fri, 18 Aug 2023 07:34:50 GMT

Hi David

its unfortunate that you can't upload the project, it would make it eaiser for me to help you find the root cause. If I make this case private it would be confidential between you, Nordic Semiconductor and your chosen distributor(non in this case). Do you think you are able to create a small dummy version of the code?

When you say powered over USB, which usb port are you using? If you are using J2 then the debugger is powered on

Regarding power measurement and how to prepear the DK we have a section on it on our infocenter

The sleep current sounds very high for a sleep in my opinon, but its hard to say for sure when I'm nore sure about what's beeing measured.

What log options have you enabled?

As for the rest I need to investigate a bit and get back to you.

Thanks for the update

RE: nRF52840 sometimes won't wake up.

david4255334 — Fri, 18 Aug 2023 05:22:17 GMT

Hi Runar,

thank you for you quick reply.

Unfortunately, I cannot upload the project.

When it comes to logs, when it's stuck, I see nothing int the console. No system logs, and no logs from my application. I've never tweaked the OS logs to a custom setting, but I guess that system error logs would be printed by default.

When performing the tests, my debugger is always disconnected. The board is only powered from USB.

When it comes to current, there is no way to measure the current consumed by nRF only. When we put the board to sleep, we power off most of the power rails, byt there are still some components that are powered on. And the consumption is still a couple of mA even in the sleep mode.

However, I connected the USB power meter (Fnirsi FNB58) between the power source and USB connector of the board to compare the consumption when it goes to sleep correctly and when it hangs.

When it goes to sleep correctly, then I can see consumption around ~7.7 - 7.8 mA.

When it hangs, the consumption is slightly higher: ~8.3 - 9.5 mA.

Another thing is that recently I added the watchdog functionality.

I noticed, that even when the BLE is not disabled before going to sleep, but watchdog is enabled, the overnight test passes and the board does not hang. This looks to me like calling pm_state_force() causes the watchdog reset.

Every time it hangs, I'm sure it entered the function where I call pm_state_force(), because just before putting the board to sleep, I disable other power supplied and this turns off the LED indicating PSU state. This disableOtherPsu() call is literally above pm_state_force(). I did not include this detail in the first post.

disableOtherPsu(); // This calls gpio_pin_set() Zephyr API, and  results in turnign off PSUs and LED
pm_state_force(0u, &(struct pm_state_info){PM_STATE_SOFT_OFF, 0, 0, 0});
k_sleep(K_SECONDS(2U));

What pm_state_force() does, is making the Power Management thread setting the bit in SYSTEMOFF (0x500) register? Do you think that setting that register manually instead of callibg pm_state_force() would help to troubleshoot it? Or is it a bad idea and we should control the power state by accessing the registers directly?

RE: nRF52840 sometimes won't wake up.

runsiv — Wed, 16 Aug 2023 12:55:43 GMT

I can't say that understand at the moment why disabling BT would solve the problems.

Would it be possible for you to upload your project? I can set the case to private if you like.

Do you have any logs from when it hangs? It would be helpful while trying to figure out the root cause

Is your debugger attached when you tried to set it into the power state? The debugger would prevent it from going into power down mode.

Do you have any current measurements from the tests?

Regards

Runar