nRF9160 - sent UDP message do not longer reach destination after 4 weeks

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Sat, 26 Nov 2022 10:18:47 GMT

As I started, I assume, that Nordic is able to make traces for long-term scenarios, while I'm not. Using a PC obviously doesn't work.

The shortest way to go is Trace on Raspi .

Anyway, it's much easier for me, to reboot such devices than to dig more into it.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Fri, 25 Nov 2022 14:31:44 GMT

Thank you for the update. Our modem experts were not able to recognize this as any known issue, based on the data provided. Please let us know if you are able to record anything of interest, or if you would like to investigate further.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Tue, 22 Nov 2022 13:48:21 GMT

One device was stopping to send data int the "wild", but in order to overcome that, I had to power off/on and with that the cause was also gone.

It's not possible with the current tool set to make such long-term captures in the wild. So for now, I stop to spend time in this issue.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 22 Nov 2022 13:41:28 GMT

Hi Achim,

Were you able to set up the device to record the events? Has the issue occurred again?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Wed, 12 Oct 2022 07:54:54 GMT

In the meantime I'm migrate to version 2.1.0. it's now:

-----------------------------------

pdn_default_ctx_cb_reg(pdn_handler);

nrf_modem_at_printf("AT+CGEREP=1");

nrf_modem_at_printf("AT+CNEC=24");

-----------------------------------

I get the events and we will see, if an error occurs (I need to store that then somehow in the eeprom in order to read it then on the next restart.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Fri, 30 Sep 2022 14:08:29 GMT

Hi Achim, sorry for the delay.

These should come as notifications from your handler, for example the PDN handler:

https://github.com/nrfconnect/sdk-nrf/blob/v2.1.99-dev1/doc/nrf/libraries/modem/pdn.rst#L20

I will get back to you with some more details over the weekend.

Best regards,

Håkon

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Mon, 26 Sep 2022 17:29:34 GMT

I guess:

 pdn_default_callback_set(pdn_handler);
 pdn_init();
 err = nrf_modem_at_printf("AT+CGEREP=1");
 if (err) {
 LOG_WRN("Failed to enable CGEREP, err %d", err);
 }
 err = nrf_modem_at_printf("AT+CNEC=24");
 if (err) {
 LOG_WRN("Failed to enable CNEC, err %d", err);
 }

is the idea to get these information?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Mon, 26 Sep 2022 16:03:52 GMT

If "AT+CGEREP=1" is used, how are the "AT+CGEV" then reported?

Usually, my app doesn't listen to the AT output channel (I even don't know how).

For AT+CEREG=5 that's done by the lte_lc_evt_handler_t.

Same question AT+CNEC=24 (I guess AT+CENEC is a typo?)

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Mon, 26 Sep 2022 13:41:10 GMT

"AT+CEREG=5" is already in use, and I will add the others.

If it reoccurs, I will report the outcome, otherwise it's just not that relevant, if it occurs that rare.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Mon, 26 Sep 2022 13:13:44 GMT

Hi Achim,

The modem should send an error response to the 'sendto' call if there is not connection to network.

Without any more logs or modem trace we cannot tell for sure what happened. The best hint we have is the "PDP contest deleted" indication at the SIM card provider's service. This is a good hint because it is clear that no data can be transferred if there is no PDN connection. The question is how it was deleted. Based on specifications the most likely scenario is that UE sent ESM Data Traponsport PDU to network and network responded with ESM Status PDU with cause #43 "Invalid EPS Bearer identity". That will cause UE to locally deactivate related EPS bearer context meaning that the PDN connection will be deactivated without signalling to network.

When a modem trace is not possible to obtain, logging AT-commands would help obtain a better understanding in case the event occurs again. Subscribing to CEREG, CGEREP and perhaps CNET AT notifications would be useful to reveal what is happening in the modem.

If the previous scenario is what actually has happened, client should notice this by subscribing certain AT notifications. Deactivation of PDN connection is notified by modem with AT+CGEV which can be subscribed with AT+CGEREP=1. Just in case the root cause is something else, then we need to know possible reject causes sent by network. Reject cause related notifications can be subscribed with AT+CENEC=24. In addition, I would suggest subscribing also CEREG notifications with AT+CEREG=5 which will give us information about generic modem status related to registration, searches and currently camped on cell.

As a heads up, please be aware that mfw 1.3.2 is targeted for nRF9160 SiP Revision 2(B1) and that Thingy:91 hardware v1.0.0 and v1.4.0 are populated with nRF9160 SiP Revision 1(B0). Revision 1 can be used for testing and development, however, new firmware versions have not gone through full release tests with the old revision. That said, we are not aware of any limitations that could cause the issue we are discussing.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Thu, 15 Sep 2022 14:19:22 GMT

Hi Helsing,

did you forward the information:

- no error code returned from sendto

- no message received at the server with 2 retransmissions

- "PDP context delete" on the status page of the SIM card provider

---------------------

Thingy HW version: 1.0.0, 1.4.0, and 1.5.0 (it occurred once with the 1.0.0 device)

mfv: all 1.3.2

nrf Connect 2.0.1

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Thu, 15 Sep 2022 14:09:28 GMT

Hi Achim,

[quote user="hahe"]

In order to investigate furter,

which hardware version of the Thingy:91 are you using?
which nRF Connect SDK version?
which mfw?

[/quote]

Would you would be able to disclose this information?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Wed, 14 Sep 2022 20:28:52 GMT

Collection of modem trace via UARTE1 or SEGGER RTT might not be possible in all cases. Nordic is working on new functionality for capturing modem trace through custom trace backends. Fore example, storing modem trace to flash could help us learn more in case messages are lost in the future.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Tue, 13 Sep 2022 14:15:52 GMT

I have no concers, I just don't think that it's the right thing for this case.

So, your modem experts are not sure, what "PDP Context deleted" means?

Was my issue never forwarded to them? After that weeks?

I see two ways to go:

- consider, it's a bug in my application, then we can close it. For prod-mode my app will just reboot.

- consider, the modem should have detected the error and report that as error-code, then it may be of interest for your modem developer. And for me, if the outcome is, that the next firmware version will report that.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Tue, 13 Sep 2022 14:03:41 GMT

As I wrote in my first comment:

> (And no, I'm not able to provide a capture of a 4 weeks run ;-)! But I hope, that the Nordic development team does also long term tests and so have already the experience with that.)

So, no I don't plan to apply memfault for that.

It may be a bug on my side, but I see the probability more on Noridc's side.

Isn't the simple call to the function "sendto" a robust thing? And this call doesn't report an error, but I think it should have in the described scenario.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 13 Sep 2022 13:37:57 GMT

Please let us know if you have any concerns regarding how to set up Memfault. A stack trace and core dump would be among the most interesting to see.

It would in any case also be great if you could try provide a modem trace from when the modem crashes.

[quote user=""]The status page of the SIM-card provider shows as last event "PDP Context deleted".[/quote]

Have you asked the provider for any more information regarding "PDP Context deleted". What does this indicate?

Another thing to try(which is less optimal than Memfault and modem trace) is to subscribe to any kind of AT commands(however, this also requires you to be able to receives logs from the device).

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 13 Sep 2022 12:13:14 GMT

Hi Achim,

I suggest you check out the Memfault module as a tool for investigating further. The module provides "functionality to collect debug information from devices in the form of coredumps, error traces, metrics, logs and more".

In order to investigate furter,

which hardware version of the Thingy:91 are you using?
which nRF Connect SDK version?
which mfw?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Tue, 06 Sep 2022 14:44:50 GMT

> Does this only happen after 4 weeks, or could it also happen after a shorter time interval?

I detected this only once after 4 weeks. Other devices run also for longer. And other fails with other errors earlier or have been updated before that 4 weeks.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Tue, 06 Sep 2022 14:37:31 GMT

The kind of support happens again and again.

I provide information, precise as possible, but I get answers like "UDP is unreliable".

Take this other issue:

https://devzone.nordicsemi.com/f/nordic-q-a/90290/nrf9160-dk---sometimes-busy-for-1-minute-before-entering-psm

I provide a capture. instead of having this capture analyzed, I get a lot of proposals, what I should do or test. Even proposals, which are clearly fails as the capture contains already the information on that proposal.

What I would really appreciate is to have just either a "not of general interest", that's OK for me, or really an analysis of your modem experts. if you search this forum, you will find the one or other issue with network loss. And maybe the idea to more consequently report them as error is a not too bad one.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 06 Sep 2022 14:35:44 GMT

[quote user="AchimKraus"]I try to provide precise information and I would welcome, if an modem expert of Nordic spends some time in. [/quote]

Sorry for the delay here, I was out of office for a few days. We will try our best to support you and figure this out.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 06 Sep 2022 14:34:09 GMT

[quote user="AchimKraus"]Not sure, but it seems to happens again and again[/quote]

Does this only happen after 4 weeks, or could it also happen after a shorter time interval?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Wed, 31 Aug 2022 11:30:46 GMT

Not sure, but it seems to happens again and again:

I try to provide precise information and I would welcome, if an modem expert of Nordic spends some time in. But it's no problem at all, if this is not possible. I'm only evaluating.

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

Achim Kraus — Wed, 31 Aug 2022 11:24:27 GMT

> As UDP does not have guaranteed delivery, a successful sendto will not guarantee that the data will reach or have reached the server

You see the retransmission in the log, or? And you read, that the server received the messages from other devices? And that the SIM-card provider shows as last event "PDP Context deleted"?

> Could this be a data quota issue?

No.

> What happens if you reset the device?

It works again.

> As UDP ... not guarantee

In my experience (more than 5 years with UDP, CoAP/DTLS !) UDP with retransmission as CoAP CON, is much more reliable than TCP!

The only culprit in all this years have been bugs in software developed by peoples, who then telling, that it may be caused by "unreliable UDP". OK, sometimes it's a firewall, which blocks all UDP traffic, but that has changed a lot in the last years.

Beside of that, UDP works brilliant!

Let me ask:

Does Nordic have any example, which could be tested to run for 4 weeks?

From battery? Without watchdog reset?

Did you do that test on your own? Successfully?

I already propose, that Nordic demonstrates the "best practice", but beside of videos, paper, and a lot of words, I miss a real word demonstration of your best practice.

Best practices for cellular iot

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Wed, 31 Aug 2022 10:56:49 GMT

Hi Achim, sorry for the delay.

Something to point out is that both sendto (and send) returns success if the data is transferred to the modem. As UDP does not have guaranteed delivery, a successful sendto will not guarantee that the data will reach or have reached the server.

Could this be a data quota issue?

What happens if you reset the device?

RE: nRF9160 - sent UDP message do not longer reach destination after 4 weeks

helsing — Tue, 23 Aug 2022 10:58:31 GMT

Hi Achim,

We are looking into this and will get back to you.