How to add a fallback Hardware watchdog to a device tree/overlay file

[Learner]

Hello,

I am using the task watchdog sample to test the task watchdog subsystem. It is working fine.

I would like to use a hardware watchdog as a fallback; for this I need to define it in the devicetree or an overlay file.

How can I do either of the above?

I am using the following hardware/build environment,

- A proprietary board built around the nRF52833 SoC

- SEGGER Embedded Studio for ARM
  Release 5.60  Build 2021081102.47262
  Nordic Edition
  Windows x64

- Zephyr OS build v2.6.99-ncs1

- SDK 1.7.0

Thank you.

Kind regards

Mohamed

[Susheel Nuguru]

Hi Mohamed,

I would like to use a hardware watchdog as a fallback; for this I need to define it in the devicetree or an overlay file.

How can I do either of the above?

It seems like you said that the task watchdog sample that you tested is working fine. What do you mean by "either of the above"?

I would like to use a hardware watchdog as a fallback; for this I need to define it in the devicetree or an overlay file.

In the nRF Connect SDK and in the below file v2.5.0 zephyr\tests\drivers\watchdog\wdt_basic_api\boards\nrf52840dk_nrf52840_counter.overlay

you can see how watchdog is defined in the overlay file. You can enable the watchdog by changing the 

status = "okay"; inside the &wdt0 node

[Learner]

#include <drivers/watchdog.h>
#include <sys/reboot.h>
#include <task_wdt/task_wdt.h>
#include <logging/log.h>
#include <stdio.h>

#include "../../include/compiler.h"
#include "../../include/system.h"
#include "../../include/wdgt.h"
#include "../../include/event_handler.h"

LOG_MODULE_REGISTER(wdgt, LOG_LEVEL_INF);

/*
 * To use this sample, either the devicetree's /aliases must have a
 * 'watchdog0' property, or one of the following watchdog compatibles
 * must have an enabled node.
 *
 * If the devicetree has a watchdog node, we get the watchdog device
 * from there. Otherwise, the task watchdog will be used without a
 * hardware watchdog fallback.
 */
#if DT_NODE_HAS_STATUS(DT_ALIAS(watchdog0), okay)
#define WDT_NODE DT_ALIAS(watchdog0)
#elif DT_HAS_COMPAT_STATUS_OKAY(st_stm32_window_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(st_stm32_window_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(st_stm32_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(st_stm32_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(nordic_nrf_watchdog)
//#elif DT_HAS_COMPAT_STATUS_OKAY(nordic_nrf_wdt)

/* According to the comment in the sample example at
 * https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/watchdog/src/main.c
 * Nordic supports a callback, but it has 61.2 us to complete before
 * the reset occurs, which is too short for this sample to do anything
 * useful. Explicitly disallow use of the callback.
 */
#define WDT_ALLOW_CALLBACK 1
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(nordic_nrf_watchdog)

#elif DT_HAS_COMPAT_STATUS_OKAY(espressif_esp32_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(espressif_esp32_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(silabs_gecko_wdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(silabs_gecko_wdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(nxp_kinetis_wdog32)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(nxp_kinetis_wdog32)
#elif DT_HAS_COMPAT_STATUS_OKAY(microchip_xec_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(microchip_xec_watchdog)
#endif


/*
 *****************************************************************************************************************************************************
 *  !GLOBAL VARIABLES
 *****************************************************************************************************************************************************
 */
int             task_wdt_id;


#if WDT_ALLOW_CALLBACK
/**************************************************************************
 * Watchdog timeout callback function.
 *
 **************************************************************************
 */
static task_wdt_callback_t task_wdt_callback(int channel_id, void *user_data)
{
    LOG_DBG( "Task watchdog channel %d callback, thread: %s\n",
             channel_id, k_thread_name_get( (k_tid_t)user_data ) );

    /*
     * If the issue could be resolved, call task_wdt_feed(channel_id) here
     * to continue operation.
     *
     * Otherwise we can perform some cleanup and reset the device.
     */

//  printf( "Resetting device...\n" );
    LOG_WRN("Resetting device...");

    sys_reboot(SYS_REBOOT_COLD);

    return 0;
}
#endif /* WDT_ALLOW_CALLBACK */


/**************************************************************************
 * Sets up necessary kernel timers and the hardware watchdog, if desired
 * as fallback).
 *
 * Note, It has to be called before wdgt_refresh().
 *
 **************************************************************************
 */
void wdgt_init( void )
{
#ifdef WDT_NODE
    const struct device *hw_wdt_dev = DEVICE_DT_GET(WDT_NODE);
#else
    const struct device *hw_wdt_dev = NULL;
#endif
    int result;

    LOG_DBG( "Watchdog Init" );

    if ( !device_is_ready( hw_wdt_dev ) )
    {
        LOG_WRN( "Hardware watchdog %s is NOT ready - ignored.", hw_wdt_dev->name );

        hw_wdt_dev = NULL;
    }
    else
    {
        LOG_INF( "Hardware watchdog %s is ready.", hw_wdt_dev->name );
    }


    result = task_wdt_init( hw_wdt_dev );
    if ( result )
    {
        LOG_ERR( "%i - Hardware watchdog not supported. ", result );
    }

    /* passing NULL instead of callback to trigger system reset */
#if WDT_ALLOW_CALLBACK
    task_wdt_id = task_wdt_add(15000U, (task_wdt_callback_t)task_wdt_callback, NULL);
#else
    task_wdt_id = task_wdt_add( 30000U, NULL, NULL );
#endif

    if ( task_wdt_id >= 0 )
    {
        LOG_INF( "Watchdog Timeout OK - %i.", task_wdt_id );
    }
    else
    {
        LOG_ERR( "Watchdog Timeout Error - %i.", task_wdt_id );
    }
}


/**************************************************************************
 * Refresh the watchdog peripheral to prevent timeout.
 *
 **************************************************************************
 */
void wdgt_refresh( void )
{
        LOG_DBG( "Application still alive..." );

        task_wdt_feed( task_wdt_id );
}

Thank you Susheel.

Susheel Nuguru said:

You can enable the watchdog by changing the 

status = "okay"; inside the &wdt0 node

I can already see in my build folder a .dts file under zephyr\zephyr.dts, where the watchdog status is already set to "okay". See below.

...[snip]

wdt: wdt0: watchdog@40010000 {
compatible = "nordic,nrf-watchdog";
reg = < 0x40010000 0x1000 >;
interrupts = < 0x10 0x1 >;
status = "okay";
label = "WDT";
};

[snip]...

So, I added these macros (taken from one of Nordic's examples) to my code,

/*
* To use this sample, either the devicetree's /aliases must have a
* 'watchdog0' property, or one of the following watchdog compatibles
* must have an enabled node.
*
* If the devicetree has a watchdog node, we get the watchdog device
* from there. Otherwise, the task watchdog will be used without a
* hardware watchdog fallback.
*/
#if DT_NODE_HAS_STATUS(DT_ALIAS(watchdog0), okay)
#define WDT_NODE DT_ALIAS(watchdog0)
#elif DT_HAS_COMPAT_STATUS_OKAY(st_stm32_window_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(st_stm32_window_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(st_stm32_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(st_stm32_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(nordic_nrf_watchdog)

#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(nordic_nrf_watchdog)

#elif DT_HAS_COMPAT_STATUS_OKAY(espressif_esp32_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(espressif_esp32_watchdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(silabs_gecko_wdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(silabs_gecko_wdog)
#elif DT_HAS_COMPAT_STATUS_OKAY(nxp_kinetis_wdog32)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(nxp_kinetis_wdog32)
#elif DT_HAS_COMPAT_STATUS_OKAY(microchip_xec_watchdog)
#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(microchip_xec_watchdog)
#endif

Unfortunately, when I ran my code, the watchdog timer keeps timing out causing the application to keep rebooting.

However, if I don't enable the hardware watchdog as fallback but instead use only the task watchdog then the app runs fine and reboots only when expected to reboot i.e. when I deliberately stop refreshing the watchdog timer.

*** Booting Zephyr OS build v2.6.99-ncs1 ***
[00:00:00.000,213] <inf> main: Booting-up time [0 s]
....

*** Booting Zephyr OS build v2.6.99-ncs1 ***
[00:00:00.000,213] <inf> main: Booting-up time [0 s]
...

*** Booting Zephyr OS build v2.6.99-ncs1 ***
[00:00:00.000,213] <inf> main: Booting-up time [0 s]
...

Please tell what I am doing wrong.

Kind regards

Mohamed

[Susheel Nuguru]

Mohammed, 

In the code you attached, I do not see main function. I also do not see how wdgt_refresh is being called. 

In the jist of it, the wdt sample shows you that the wdt initialization works and if you are feeding the wdt correctly, then wdt does not do anything and if the feeding stops, then wdt resets the chip as intended. I do not see any other issues. here

[Learner]

Hi Susheel,

I appreciate your prompt response. Thank you.

Susheel Nuguru said:

I do not see main function.

My main function looks something like this,

void main( void )
{

   ...

    while ( 1 )
   {

      ...

       /* Refresh the watchdog channel timer every second */
       wdgt_refresh();

      ...

    }

}

So, despite the fact I am feeding wdt every second, wdt keeps timing out but this happens only when I enable the hardware watchdog as a fallback.

If I disable the hardware watchdog then the problem disappears and the application runs without the wdt timing out.

Kind regards

Mohamed

[Susheel Nuguru]

I do not understand what fallback means here? 

Can you please give me the whole snippet of your main function so that I can test why wdt timesout even though you expect it not to?

[Learner]

Hi Susheel,

Susheel Nuguru said:

I do not understand what fallback means here? 

According to the comment in the Nordic's example, it means if the watchdog timer task (based on kernel's timers) does not timeout and does not perform as expected then the hardware watchdog timer will come into action. You should be able to find the sample code at,

...\v1.7.0\zephyr\samples\subsys\task_wdt

I am sorry Susheel but I am not allowed to share the whole main() with you. However, here is a skeleton version of it. Note, the functions dealing with wdt have the prefix, 'wdgt_'.

void main( void )
{
    ...[snip]
    LOG_INF( "Booting-up time [%u s]\n", time_s );

    /* Set up debug port first */
    system_SetDebugPortAccess();

    if ( initialise( device_mode ) == RESULT_OK )   
    {
        ...[snip]
        
        if ( device_mode )
        {
            /* Initialise only the i2c and mp2695 charger to be able to detect 'mains' supply. */
            io_initialise_batt_charger();

            /* Initialise the rest of the peripheral devices after the we've detected 'mains' supply. */
            io_initialise();

            LOG_INF("Waiting for DC supply - Plug In");

            /* Initialise the Watch Dog Timer Task */
            wdgt_init();

            /* wait for the charger to be plugged in to signal installation */
            for(;;)
            {
                batt_charger_get_info();

                if ( ( io.charger->plugged_in == true ) && ( ( io.charger->charging == true ) || ( io.charger->ChargeDone == true ) ) )
                {
                    /* Update the battery voltage value read from the ADC */
                    io_update_batteryvoltage();

                    if ( io.battery.mv_raw >= 2700 ) /*!!!TBD replace magic number 2700 with BATT_DISCHARGE_PROTECT */
                    {
                        LOG_INF( "Batt_raw = %d mV", io.battery.mv_raw );

                        break;
                    }
                }

                LOG_INF( "Plug In DC - Charger Status: %d\n", io.charger->charging | io.charger->ChargeDone );

                /* Feed the watchdog channel timer. */
                wdgt_refresh();

                k_sleep(K_MSEC(10)); 
            }

            LOG_INF("Plugged");
            k_sleep(K_MSEC(10));

            /* Initialise the radio state machine. */
            hb_radio_initialise();

            k_work_init( &homebeacon_radio_status_check, homebeacon_radio_check_status_tasks );

            /* Initialise the tamper button */
            io_tamper_button_initialise();

            /* Also, we can seed random number effectively */
            srand( time( NULL ) );

            io_led_off( RED_LED );

            while ( 1 )
            {
                ...[snip]
                
                if ( flash_static_data_copy_hb.shutdown_flag == POWER_SHUTDOWN_SOURCE_NONE )
                {
                    /* This is the path for normal operation */
                    if ( b_one_sec_tasks )
                    {
                        b_one_sec_tasks = FALSE;

                        /* Refresh the watchdog channel timer. */
                        wdgt_refresh();

                    ...[snip]
                    
                    }
                }/* POWER_SHUTDOWN_SOURCE_NONE */
                else
                {
                    /* This is the path when the device is in SHUTDOWN mode */
                    ...[snip]
                }
            }
        }
        else    /* device mode NOT known */
        {
            ...[snip]

            /* What else can we do? */
            LOG_INF( "Unsupported device mode\n" );
        }
    }
    else
    {
        /* Initialisation error condition */
        ...[snip]

        /* What else can we do? */
        LOG_INF( "Initialisation Failed\n" );
    }

}

Note the same code works fine if the hardware watchdog is not defined i.e. this line is commented out.

#define WDT_NODE DT_COMPAT_GET_ANY_STATUS_OKAY(nordic_nrf_watchdog)

Kind regards

Mohamed

[Susheel Nuguru]

Learner said:

According to the comment in the Nordic's example, it means if the watchdog timer task (based on kernel's timers) does not timeout and does not perform as expected then the hardware watchdog timer will come into action. You should be able to find the sample code at,

I see, this is called the callback feature not the fallback feature.

Learner said:

According to the comment in the Nordic's example, it means if the watchdog timer task (based on kernel's timers) does not timeout and does not perform as expected then the hardware watchdog timer will come into action. You should be able to find the sample code at,

I think you misunderstood this feature. the callback in the task_wdt_add will be triggered when the watchdog has already expired. That means that if you call wdt_feed function in this callback it will not reset the wdt timer. This callback function needs to be used in places to run put your device in safe mode where it should expect that the device reset will happen soon.

If you see in the file zephyr\subsys\task_wdt\task_wdt.c, you will see that the callback is called with task_wdt_trigger->channels[channel_id].callback and the description of this function is here.

/**
 * @brief Task watchdog timer callback.
 *
 * If the device operates as intended, this function will never be called,
 * as the timer is continuously restarted with the next due timeout in the
 * task_wdt_feed() function.
 *
 * If all task watchdogs have longer timeouts than the hardware watchdog,
 * this function is called regularly (via the background channel). This
 * should be avoided by setting CONFIG_TASK_WDT_MIN_TIMEOUT to the minimum
 * task watchdog timeout used in the application.
 *
 * @param timer_id Pointer to the timer which called the function
 */
static void task_wdt_trigger(struct k_timer *timer_id)

[Learner]

Hi Susheel,

Thank you.

Susheel Nuguru said:

I think you misunderstood this feature. the callback in the task_wdt_add will be triggered when the watchdog has already expired

Yes, I agree.

Susheel Nuguru said:

I see, this is called the callback feature not the fallback feature.

Not according to the comment in Nordic's example 

...\v1.7.0\zephyr\samples\subsys\task_wdt

/*
* To use this sample, either the devicetree's /aliases must have a
* 'watchdog0' property, or one of the following watchdog compatibles
* must have an enabled node.
*
* If the devicetree has a watchdog node, we get the watchdog device
* from there. Otherwise, the task watchdog will be used without a
* hardware watchdog fallback.
*/

Susheel Nuguru said:

If you see in the file zephyr\subsys\task_wdt\task_wdt.c, you will see that the callback is called with task_wdt_trigger->channels[channel_id].callback

Does the function static void task_wdt_trigger(struct k_timer *timer_id) get called when the hardware watchdog timer timesout?

In my prj.conf file I have,

CONFIG_TASK_WDT_MIN_TIMEOUT=100

whereas my task watchdog timer is set with 30000 ms 

task_wdt_id = task_wdt_add(30000U, (task_wdt_callback_t)task_wdt_callback, NULL);

Must I set CONFIG_TASK_WDT_MIN_TIMEOUT=30000 

to avoid the problem I ma seeing?

Kind regards

Mohamed

[Learner]

Hi Susheel,

For some obscure reason I am not seeing the watchdog timer timing out when I enable the hardware timer. It is rather worrying because I think it will come back at some point. However, let's forget about it for now.

My question now is this,

How do I check that the hardware timer will timeout if the the task watchdog does not work as expected?          In other words I would like to simulate the scenario where the hardware watchdog fallback is used instead of the task watchdog as suggested in this Nordic example, see its description below.

...\v1.7.0\zephyr\samples\subsys\task_wdt

/*
* To use this sample, either the devicetree's /aliases must have a
* 'watchdog0' property, or one of the following watchdog compatibles
* must have an enabled node.
*
* If the devicetree has a watchdog node, we get the watchdog device
* from there. Otherwise, the task watchdog will be used without a
* hardware watchdog fallback.
*/

Also, I have set my task watchdog timer to 600000 ((one minute) but the maximum value CONFIG_TASK_WDT_MIN_TIMEOUT can take is 10000, so I am setting it to 10000 in prj.conf.

According to the description of static void task_wdt_trigger() you sent me above I was expecting this function to be called regularly but it is not. 

Is it because I am refreshing the task watchdog timer every second?

I am printing the reset reason in my callback function task_wdt_callback_t task_wdt_callback() expecting it to be 2 since it was caused by a task watchdog timeout, instead I am getting reason = 0.

printf( "Reas = %u\n", NRF_POWER->RESETREAS );

Can you please explain why this is the case?

Finally, when I try to write to flash in the callback function task_wdt_callback(), nvs_write() is returning the error ECANCELED (140) meaning the write operation has been cancelled. Is it because, as stated in the task_wdt example comments, the callback has only 61.2 us to complete before the reset occurs and writing to flash is taking longer than this?

Kind regards

Mohamed

[Maria Gilje]

Hi Mohamed,

We are severely understaffed this week because of the Christmas holidays, and Susheel will have to get back to you on this next week.

Sorry for the inconvenience.

[Learner]

Hi Maria,

Thank you for letting me know.

Kind regards

Mohamed

[Susheel Nuguru]

Thanks for waiting Mohamed, Happy New Year. I am just back from the holidays and wondering if you still have this issue or have you found the way to fix it?

Learner said:

Is it because I am refreshing the task watchdog timer every second?

That could be.

Learner said:

I am printing the reset reason in my callback function task_wdt_callback_t task_wdt_callback() expecting it to be 2 since it was caused by a task watchdog timeout, instead I am getting reason = 0.

printf( "Reas = %u\n", NRF_POWER->RESETREAS );

That is strange. Can you please give me your project for me to reproduce and debug this? Maybe the reset reason is cleared somewhere else in the code?