Diagnosing AWS IoT Communication failure

[Ivan Popov]

Hello,

I'm tinkering (motion detection mostly) with a Thingy:91 unit (I don't have a DK board), and in general it works fine reporting sensory data to AWS IoT Core.

Last night it some point the communication suddenly stopped. Today I opened the nRF Connect's Serial Terminal and I saw this:

A bit more context:

1. Faulty behaviour was noticed last Thursday (a week ago) and while searching for hints in this forum I found that some of the error messages (mostly -119 during that previous occasion) from this ticket match my case.
2. I was just about to start gathering the traces mentioned and the communication suddenly got restored. The peculiar part was that in AWS MQTT Test Client I could only see the Shadow Updates (none of the other service topics).
3. The situation remained like this up until last night when the next outage appeared, hence the output in the screenshot.

I spoke to my colleague who monitors closely the AWS activity and he said that everything in that side looks fine and unchanged (the lambdas ingesting the MQTT messages were acting optimally). And it can't be Amazon because we have some other types of devices that report MQTT messages through Wi-Fi successfully.

This leaves me to wonder if the mobile operator can be blamed for this outage. In my country currently there is only one operator supporting IoT data transfers and they support only NB-IoT (LTE-M is not an option).

The Question:

Should I try to get some modem traces and if the problem is with the mobile operator can these traces narrow down the exact issue?

Thanks and regards!

[AlekFlatley]

You can take a modem trace and analyze them using the LTE Link Monitor tool available in nRF Connect for Desktop. From there you can narrow down the exact problem with network connectivity and NB-IoT mode. getting over it

[Ivan Popov]

Yes, thanks!
I needed to catch up a bit on the subject so I followed this documentation section and this DevAcademy lesson. I have some finding to share and I'll do it in an update post later.

Regards

[Ivan Popov]

UPDATE

I followed the Cellular Monitor section in the documentation. As an added value I updated both the SIP and SoC firmwares to the latest versions (with no change in the behaviour).

I could test with a set of three applications:

1. The main application (App A) I'm using - it is the aws_iot sample configured the proper way and with additions only appending sensor readings (ext_sensor.h & ext_sensor.c) to the shadow update - the MQTT (or overall connectivity) functionality was not touched.
2. Basic MQTT client (App B) - only the aws_iot sample as it is configured with my credentials (tested was previously working - not touched since then).
3. The full Nordic Asset Tracker v2 (App C) - the sample from the repository only configured with my credentials - the MQTT (or overall connectivity) functionality was not touched.

The Wireshark for App A and App B showed sets of these:

App A

App B

App C

I can see that there is some resetting of the connection on the TCP protocol level but I'm not familiar with this type of logs and traces and I'm not sure how to extract the relations between the messages and the functionality in the code.

Here I provide the relevant trace in two file formats in full

aws-iot-mqtt-connection-problem.zip

I hope there are enough hints in there so I can get a proper advise and help to figure out the problem. I'll provide any additional information if required.

Almost forgot to mention that on the serial terminal the behviour looks like this:

Regards!

[Hakon]

Hello,

have you checked the signal quality? You can run the command AT+CESQ? to read various parameters including RSRP and RSRQ.

[Ivan Popov]

Hi,

Thanks for the input. Is this output relevant in some way? 

Also in the Cellular Monitor I get this while executing the modem trace:

I can add another trace file but from that I'm seeing in WireShark, the information would be the same as in the previously attached trace.