nRF9160 Modem MQTT Download Data Timeout

[Eizens]

Hello,

I have been chasing this problem for some time, but it has been difficult to capture.

The issue occurs when the device is sitting idle (no eDRX or other power-saving modes are enabled). The MQTT ping timeout is set to 60 seconds, so the device always has an open connection to the server, and the server can send data to the device at any time. The network is LTE-M.

This setup is used for a smart lock application. Most of the time, it works well: in about 90% of cases, data is delivered to the modem in ~2.5 seconds, which is roughly the same as the mobile network paging cycle. Sometimes the delay is 5 seconds or more if some paging cycles are missed, which is also acceptable.

However, in some cases, the device is not reachable for 30 seconds or more. This happens in approximately 1% or less of cases, so it has been very difficult to capture.

In this trace, there are mainly two types of MQTT commands: commands/... and ping/..., as well as the actual MQTT PING packet. The ping/... MQTT command is used only for testing from the server side. The device to the commands/... responses with state/...

The problem occurred at the end of the trace, when two commands/... messages were received almost at the same time (approximately 1 second apart).

1. 19:50:23 Last ping/pong exchange completes (device RRC Connected)
2. RRC release → idle. Device camped on cell 002CDA12.
3. 19:50:38  Broker sends /command, the command needed paging from this moment
3. 19:50:38–19:50:57 ~19.4s: device hears nothing. No Service request, no paging response.
4. 19:50:56 second command is sent from Broker
5. 19:50:57 Service request, a page finally gets through
6. 19:50:58 Both Command's are delivered; device publishes two state/ replies.

Could you please take a look at this part of the trace and see if you can identify anything that could explain the delay?

Thank you!

timeout.pcapngmodemtraces_21_01.bin

[Simon D-M]

Hi,

What SDK version and Modem firmware version are you using ?

Best regards,

Simon

[Eizens]

Hello,
SDK version is v3.1.1 and modem firmware is mfw_nrf9160_1.3.7.

[Simon D-M]

Hi,

I'm not sure that I see the same thing as you do... The only place that somewhat matches your description is this:

And in the trace we can see that the time from the "commands/" to the acknowledgment of the "state/" is less than a second.

Am I in the wrong place? It seems my timestamps don't go past 19:20.

Best regards,

Simon

[Eizens]

Yes, the place you are looking at is correct.

Regarding the time: I don't know why, but the time in my PC trace is off by ~30 minutes compared to the server/MQTT broker time. So yes, you are looking at the correct place.

Yes, when I receive the /command, I immediately send the /state response. But the problem happens before that.

The sequence is:

1. The server sends a /command.
2. The device does not receive this message for 19.4 seconds.
3. The server sends a second /command.
4. The device receives both /command messages at the same time.

So my question first question is: is there any clue about what caused this problem, such as missed pages, or why the device did not receive the command for 19.4 seconds?

[Eizens]

I will try to explain the problem a little further, but the text may be somewhat all over the place, as it is hard to precisely define the problem. However, here are some of my observations, and I will also add some additional questions.

1) Our devices are stationary, but one thing I have observed is that this problem happens more often when a device is continually switching between two cells.

My question is: is there any period when the modem is not reachable due to some internal process, such as periodic cell searching or something similar, where the modem does not report that there is no connection?

As I understand it, the GPS part of the modem could potentially cause something like this, but I don't use GPS.

2) Here is a histogram of the ping times for two devices. This is just an interesting observation.

The two devices are located at the same spot and have similar signal strength. Both are switching between two cells. The histogram shows the results of around 350 random pings. Time is in milliseconds.

For Device No. 1, the maximum ping time was 7 seconds, while for Device No. 2 it was 25 seconds + it is all over the place with timeout of 23s, 22s, 21s, 19s etc.

The only difference between these two devices is the MQTT keepalive time: for Device No. 1, it is 120 seconds, while for Device No. 2, it is 30 seconds.

Therefore, Device No. 2 has ping times of up to around 1.3 seconds more frequently, as the device is in the RRC connected state more often.

To rule out the possibility that this is related to the MQTT keepalive time, here is also a histogram for Device No. 3, which is in a different location, camped on a single tower, and also has a 30-second MQTT keepalive.

3) I also reported this problem to the operator with another device (we can call this device No4), and today I received their response.

I initially contacted them because the device had a period of around 170 seconds during which it was not reachable. This was Device No. 4, which is in my office for testing. The device is also bouncing between two cells.

I did not have the modem trace enabled at the time, but I noticed that the device switched cells, and the downlink data was not received for around 170 seconds.

Here is response from operator:

From the network side, it looks as follows (the time is given in UTC in the client traces; LV time was 12:42:31 when the message delivery time was 170 seconds):

At 12:40:03 — paging → OK
From 12:42:03 to 12:43:55, all paging messages went unanswered.

Paging messages are being sent to the BS and TAC. However, the device does not respond until it establishes a new active RRC connection after switching to idle mode on another cell.

While the device is in idle mode, it must listen to the paging channels and know when to listen. In our case, the paging occasion parameters are identical in all cells — Cell A does not differ from Cell B — so the device should be listening for paging in the same way.

From the core network side, it is visible that paging messages are being sent and retransmitted, but there are no responses from the device.

To see the full picture in more detail, we need to agree on recording new log files on both the BS and UE sides so that the complete picture is visible.

Please send us the number, location, and times when the test could be performed.

 

So I will try to create a new test where I have the modem trace enabled, along with logging enabled on the network side by the operator.

However, this happens so rarely that it is difficult to capture. Sometimes I get lucky and see an extensive timeout within a 1-hour period, while other times it takes more than 24 hours which makes the trace capture more difficult.

So yes, I have seen this problem with multiple devices in different locations, but they are all connected to the same operator in a single country. I don't know if the problem I am describing in this ticket is the same as what happened with Devices No. 1–4.