NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Wed, 19 Mar 2025 11:03:20 GMT

I agree that it's likely an app issue, from what is described here. But I'm not new to watchdog, not new to zephyr, not new to threads, ... and I can't figure out why the watchdog fires in this case. That's why I raised the ticket in the first place. I've provided a stripped down reproduction sample. If anyone is able to point out where the mistake is (in the app), I'm happy to learn from that. But so far, my feeling is that there's a deeper issue.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Achim Kraus — Mon, 24 Feb 2025 17:39:05 GMT

I'm just an other user.

AFAIK the watchdog is located on the app CPU not on the modem. If changing the modem firmware fires that watchdog of the app CPU, it is in my experience still an issue in the app.

What changing the mfw may cause is a slightly different timing, which then may unexpected block. Your countermeasures are pointing to something like that.

Anyway, it's your time and so your decision.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Mon, 24 Feb 2025 10:50:15 GMT

I don't agree with your conclusions. The GPS thread does not fully take over CPU control since in the internal loop of "app_gps_internal_get_fix", it periodically blocks on a semaphore. Also, again, this whole program runs fine with modem fw 2.0.1 and watchdog enabled.

I have even changed the watchdog channel period to 20 seconds (20000 msec) and in combination with modem fw 2.0.1, the watchdog kicks in at the expected time if it is not fed. Also, the callback is called (task_wdt_callback) as expected.

With modem FW 2.0.2 this is not the case, the watchdog resets way sooner, and a callback is not called. The solutions or fixes that can make the application work with FW 2.0.2 are one of those:

Delay of gps thread startup with e.g. 1 second
Or changing the sleep of the main loop to something like 50 msec instead of 1 second what it was in the past
- 90 msec also works fine, 100msec does not. By chance (or not?), this is equal to CONFIG_TASK_WDT_MIN_TIMEOUT.

So please stop asking me to toggle GPIOs. The main loop is running and kicking, but the problem is that the watchdog decides to reset nevertheless, and it resets way before the configured task period of e.g. 5 seconds or 20 seconds. This behavior only occurs with modem fw 2.0.2, not with fw 2.0.1.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Simonr — Mon, 24 Feb 2025 09:57:06 GMT

After a brief look at your project, it seems like your GPS thread completely takes over the main thread, so that the watchdog never gets fed in the first place. How exactly have you timed these system resets on your end?

To confirm whether the feeding in the main loop ever is reached, you can try switching it out with a GPIO toggling instead to see if that ever happens. We assume this will be blocked when you run the GPS, and that it just never is fed. Making some changes to the priorities in your application should fix that.

Best regards,

Simon

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Fri, 21 Feb 2025 08:08:12 GMT

Hi Simon,

Thanks for jumping in.

The SDK version and toolchain is 2.9.0.

Actually, the watchdog reset occurs several seconds before a first task timeout can occur. The task timeout is set to 5 seconds, but the system resets after about 2 seconds (without calling the watchdog callback either). So there's no relationship with the duration of the tasks.

Changing the task timeout to 10 seconds or more does not make any difference: the system still resets after about 2 seconds.

The application can work completely fine if the watchdog is disabled, or if I put a sleep of 1 second (overkill) before the initialization of the GPS stack, which is the workaround that I'm using now.

In my opinion, I think it's best to try to reproduce based on the code that I provided. Let me know if I should re-share the code sample.

The issue is not urgent for me, but I have the feeling that there's something fundamentally wrong that should be understood better or it can cause more issues in the future.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Simonr — Thu, 20 Feb 2025 14:50:11 GMT

Sorry, but Susheel is not available, and I've been tasked with looking into this ticket while he's away. So sorry that we have to take a couple of steps back as I need some time to get into this case. One thing I'm not able to find info on in this case is which SDK version you're working on here. Can you share some more information on that. I'm currently waiting for some replies internally on ideas as to why the watchdog would misbehave like this. I have not looked at the project you uploaded, but I'll get to that. Just wanted to give you a status update.

My initial thought, and this is just me taking a stab in the dark, is that the GPS location takes longer time than what the watchdog timeout is set as, and thus the watchdog triggers a reset while you're trying to get a GPS location. Is there a specific reason the WDT is set to 5 seconds, and could you try to increase it substantially to see if that makes a difference? You say that your application works as expected if you disable the WDT. Can you confirm you're able to initialize GPS and get a GPS ping as expected without the WDT?

Best regards,

Simon

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Susheel Nuguru — Mon, 17 Feb 2025 09:07:58 GMT

I downloaded it now, will try to reproduce it

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Sat, 15 Feb 2025 13:34:57 GMT

The GPIO toggling is not very relevant, the reset occurs even faster than that (rather 1.5 to 2 seconds).

A more or less minimal reproduction codebase can be found at https://drive.google.com/file/d/1ivy9eGtTYCgckw6WrWdGo_HJNOSL1UGk/view?usp=sharing. Let me know once you downloaded it so I can remove it again.

This is for nrf9151, it's reproducible without sim card, you also don't need nrf7002.

It doesn't matter if you set GPIO's to IO-expander (via board configuration tool) or keep them on GPIO.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Susheel Nuguru — Thu, 13 Feb 2025 12:39:35 GMT

That seems strange that you are feeding the WDT in time and still it resets before the 5 seconds timeout expires. Can you toggle a gpio everytime you feed the WDT to see if the context you are feeding it is getting CPU in time or if that context is masked by some other high priority contexts?

After you GPIO toggle test, if you think the feeding is still in time, then yes, please give me a minimalistic project that I will look into early next week.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Wed, 12 Feb 2025 13:14:35 GMT

The reset reason seems to be watchdog. Verified by nrfx_reset_reason_get and hwinfo_get_reset_cause function.

Indeed, watchdog is enabled. But it's initialized with a task configured with a timeout of 5 seconds, whereas the device resets way faster than 5 seconds. I've made sure that watchdog init and feeding is started before anything else, but that doesn't help anything.

It seems to me that GPS initialization tampers with the watchdog and that results in a reset, that's the best conclusion I can draw so far. Removing the main task from watchdog control (so that there's no task whatsoever under WD control), or not initializing the watchdog at all seems to prevent the issue, but that's not sufficient.

BTW I also have a callback registered to the watchdog, which should print if it expires (in the normal way, this has been verified) and that print is also not taking place.

If you want, I could try to strip down my code to a bare minimum and share it? I'm not new to using NRF91, but this seems like a strange problem and I don't think it's due to my code.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Susheel Nuguru — Wed, 12 Feb 2025 11:54:22 GMT

Sebastian, I am sorry, I got this from a forum thread which seemed logical but it does not look like this AT command will work on our product.

You can manually print the reset reason as done in many other samples like below.

static void print_reset_reason(void)
{
	uint32_t reset_reason;

	reset_reason = nrfx_reset_reason_get();
	LOG_INF("Reset reason: 0x%x", reset_reason);
}

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Tue, 11 Feb 2025 15:32:07 GMT

Are you sure this command exists? I can't find it in the documentation https://docs.nordicsemi.com/bundle/ref_at_commands_nrf91x1/page/REF/at_commands/intro_nrf91x1.html and I also couldn't get it to work, it returns ERROR while e.g. "AT%DEVICEUUID" does provide a response.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Susheel Nuguru — Tue, 11 Feb 2025 12:51:23 GMT

Sorry for late response. Then you should be able to read the reset reason

If you see 0x00000004 then it is a brown out reset and then we need to see if your power source is stable.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Fri, 07 Feb 2025 13:22:01 GMT

Hi Susheel,

I rather think this is a brownout reset. The function sys_reboot() is never hit.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Susheel Nuguru — Fri, 07 Feb 2025 11:13:06 GMT

The first thing that comes to my mind when you say autonomous resets is that this should be trackable. Can you put a breakpoint in the sys_reboot() function, run the application in debugger and let the breakpoint hit. When you look at the function call stack at this point it will give us more context on why this is happening and we can take it from there on how to avoid or workaround.

RE: NRF9151DK autonomous resets when simultaneously starting LTE connection and GPS on modem FW 2.0.2

Sebastiaan Merckx — Thu, 06 Feb 2025 14:43:33 GMT

I just observed that a 100msec sleep is not always sufficient, I have now just increased it to 1 second to be safe.