nRF9160 - sent UDP message do not longer reach destination after 4 weeks

[Achim Kraus]

Running a Thingy:91 now for 4 weeks from batter with one message exchange per hour, it stopped working (NB-IoT mode).

The client (Thingy:91) uses "sendto" and that function returns success.

The client's log:

[18:01:38.586,822] <inf> COAP_CLIENT: CoAP request prepared, token 0xfbe72850, 139 bytes
[18:01:38.594,146] <inf> COAP_CLIENT: send_to_peer 175
[18:01:38.618,927] <inf> COAP_CLIENT: LTE modem wakes up
[18:01:40.208,496] <inf> COAP_CLIENT: RRC mode: Connected
[18:01:40.594,757] <inf> COAP_CLIENT: 1/1/729-1621 ms: connected => sent
[18:01:44.595,489] <inf> COAP_CLIENT: CoAP request resend, timeout 6
[18:01:44.596,832] <inf> COAP_CLIENT: resent_to_peer 175
[18:01:51.597,991] <inf> COAP_CLIENT: CoAP request resend, timeout 12
[18:01:51.607,635] <inf> COAP_CLIENT: resent_to_peer 175
[18:02:04.609,924] <inf> COAP_CLIENT: CoAP request resend, timeout 24
[18:02:04.619,537] <inf> COAP_CLIENT: resent_to_peer 175
[18:02:17.364,807] <inf> COAP_CLIENT: RRC mode: Idle after 37156 ms (12745 ms inactivity)
[18:02:25.377,655] <inf> COAP_CLIENT: LTE modem sleeps
[18:02:32.625,030] <inf> COAP_CLIENT: 729/-1ms/-1ms: failure

It starts with passing the message to "sento", which returns success with 175.

Then the modem wakes up and reports RRC connected. Though no response is received, the modem resends the message also with "sendto", which also returns success with 175.

On the server side the ip-capture shows, that no UDP traffic form that modem comes in, but from other clients it is still working.

The status page of the SIM-card provider shows as last event "PDP Context deleted".

For me this look like the modem seems to have trouble with the "PDP Context deleted" but didn't report that in the return code of "sendto".

Any proposal how to overcome that?

(And no, I'm not able to provide a capture of a 4 weeks run ;-)! But I hope, that the Nordic development team does also long term tests and so have already the experience with that.)

[helsing]

Hi Achim,

The modem should send an error response to the 'sendto' call if there is not connection to network.

Without any more logs or modem trace we cannot tell for sure what happened. The best hint we have is the "PDP contest deleted" indication at the SIM card provider's service. This is a good hint because it is clear that no data can be transferred if there is no PDN connection. The question is how it was deleted. Based on specifications the most likely scenario is that UE sent ESM Data Traponsport PDU to network and network responded with ESM Status PDU with cause #43 "Invalid EPS Bearer identity". That will cause UE to locally deactivate related EPS bearer context meaning that the PDN connection will be deactivated without signalling to network.

When a modem trace is not possible to obtain, logging AT-commands would help obtain a better understanding in case the event occurs again. Subscribing to CEREG, CGEREP and perhaps CNET AT notifications would be useful to reveal what is happening in the modem.

If the previous scenario is what actually has happened, client should notice this by subscribing certain AT notifications. Deactivation of PDN connection is notified by modem with AT+CGEV which can be subscribed with AT+CGEREP=1. Just in case the root cause is something else, then we need to know possible reject causes sent by network. Reject cause related notifications can be subscribed with AT+CENEC=24. In addition, I would suggest subscribing also CEREG notifications with AT+CEREG=5 which will give us information about generic modem status related to registration, searches and currently camped on cell.

As a heads up, please be aware that mfw 1.3.2 is targeted for nRF9160 SiP Revision 2(B1) and that Thingy:91 hardware v1.0.0 and v1.4.0 are populated with nRF9160 SiP Revision 1(B0). Revision 1 can be used for testing and development, however, new firmware versions have not gone through full release tests with the old revision. That said, we are not aware of any limitations that could cause the issue we are discussing.

[Achim Kraus]

If "AT+CGEREP=1" is used, how are the "AT+CGEV" then reported?

Usually, my app doesn't listen to the AT output channel (I even don't know how).

For AT+CEREG=5 that's done by the lte_lc_evt_handler_t.

Same question AT+CNEC=24 (I guess AT+CENEC is a typo?)

[Achim Kraus]

I guess:

pdn_default_callback_set(pdn_handler);

pdn_init();

err = nrf_modem_at_printf("AT+CGEREP=1");

if (err) {

LOG_WRN("Failed to enable CGEREP, err %d", err);

}

err = nrf_modem_at_printf("AT+CNEC=24");

if (err) {

LOG_WRN("Failed to enable CNEC, err %d", err);

}

is the idea to get these information?

[helsing]

Hi Achim, sorry for the delay.

These should come as notifications from your handler, for example the PDN handler:

https://github.com/nrfconnect/sdk-nrf/blob/v2.1.99-dev1/doc/nrf/libraries/modem/pdn.rst#L20

I will get back to you with some more details over the weekend.

Best regards,

Håkon

[Achim Kraus]

In the meantime I'm migrate to version 2.1.0. it's now:

-----------------------------------

pdn_default_ctx_cb_reg(pdn_handler);

nrf_modem_at_printf("AT+CGEREP=1");

nrf_modem_at_printf("AT+CNEC=24");

-----------------------------------

I get the events and we will see, if an error occurs (I need to store that then somehow in the eeprom in order to read it then on the next restart.

[helsing]

Hi Achim,

Were you able to set up the device to record the events? Has the issue occurred again?

[Achim Kraus]

One device was stopping to send data int the "wild", but in order to overcome that, I had to power off/on and with that the cause was also gone.

It's not possible with the current tool set to make such long-term captures in the wild. So for now, I stop to spend time in this issue.

[helsing]

Thank you for the update. Our modem experts were not able to recognize this as any known issue, based on the data provided. Please let us know if you are able to record anything of interest, or if you would like to investigate further.