NB-IoT TCP Packet loss and retransmittions

[Vipul Panchal]

Hardware: nrf9160DK
Modem FW: v1.3.1

We have been evaluating and configuring our proprietary application for NB-IoT, which is initially designed for LTE-M (CAT-M1). The Application is designed to exchange data with the cloud using MQTT over TLS and works fine with LTE-M.
However, when explicitly configured to connect to NB-IoT only network, we see a lot of TLS handshakes getting timed out (even after increasing the TLS timeout from 20 seconds for LTE-M to 90 seconds for NB-IoT).
Evaluating the Network capture files we observed many instances of TCP packet retries especially when a large data is exchanged over several TCP segments (the case where the server and clients exchange the certificates and keys during a TLS handshake).

To remove the dependency of our application and to just test the MQTT over TLS on the NB-IoT network, we switched to using the nrfConnect sample application "asset_tracker_v2".
The "asset_tracker_v2" application was configured to connect to NB-IoT only network and modem trace enabled.
After letting the application execute for a few hours, analysis was done using the application logs, modem trace and network capture.
We still see the same behaviour (TCP packet retransmissions) with the large data exchanges over several TCP segments.

Typical behaviour is observed, after receiving the first TCP segment of a large data packet, the modem seems to be not accepting any further segments for a period of approximately 20 seconds. (see the attached screenshot of the comparison between modem trace(left) and network capture (right))
Are we missing any configuration or can you give an explanation of the behaviour?

Also, attached is the Application Log, Modem Trace and Network capture for your further investigation.attachments.zip

[Vipul Panchal]

Hi, 

Thanks for your response.

 
The device being tested is in UK and we are using a test NB-IoT network (using Nutaq).
The problem is not with registering to a network. The problem is with modem not accepting the TCP segments (which are transmitted by the server, seen in the network  capture. however not getting received on the client-side, seen in the modem trace)
Have a look at the image in the attachments.zip, where it shows a duration of approximately 20 seconds when the modem doesn't log any TCP packets whereas the server is sending them at the same time.

[Øyvind]

Can you provide more information on what network you are trying to connect to? What version of nRF Connect SDK are you working on? 

From your PCAP I see that you are receiving the following: +CGDCONT: 0,"IP","test123","192.168.3.2",0,0  OK  

The provided log output from LTE Link Monitor has the following output several times:

2022-04-13T05:18:31.917Z DEBUG modem << [25:52:00.274,871] [0m<inf> event_manager: APP_EVT_DATA_GET_ALL[0m
2022-04-13T05:18:31.970Z DEBUG modem << [25:52:00.281,524] [0m<inf> event_manager: APP_EVT_DATA_GET - Requested data types (MOD_DYN, BAT, ENV, NEIGHBOR_CELLS, GNSS)[0m
2022-04-13T05:18:31.971Z DEBUG modem << [25:52:00.299,285] [1;33m<wrn> gnss_module: Failed to set GNSS fix interval, error: -22[0m
2022-04-13T05:18:31.973Z DEBUG modem << [25:52:00.310,974] [0m<inf> event_manager: SENSOR_EVT_ENVIRONMENTAL_NOT_SUPPORTED[0m
2022-04-13T05:18:32.002Z DEBUG modem << [25:52:00.357,696] [1;31m<err> modem_info_params: Link data not obtained: 5 -22[0m
2022-04-13T05:18:32.016Z DEBUG modem << [25:52:00.370,574] [1;31m<err> modem_info_params: Link data not obtained: 8 -22[0m
2022-04-13T05:18:32.038Z DEBUG modem << [25:52:00.388,671] [1;31m<err> modem_info_params: Link data not obtained: 3 -22[0m
2022-04-13T05:18:32.074Z DEBUG modem << [25:52:00.429,748] [1;31m<err> modem_info_params: Network data not obtained: -66[0m
2022-04-13T05:18:32.087Z DEBUG modem << [25:52:00.437,774] [1;31m<err> modem_module: modem_info_params_get, error: -11[0m
2022-04-13T05:18:32.098Z DEBUG modem << [25:52:00.445,678] [0m<inf> event_manager: MODEM_EVT_MODEM_DYNAMIC_DATA_NOT_READY[0m
2022-04-13T05:18:32.145Z DEBUG modem << [25:52:00.501,251] [1;31m<err> modem_info_params: Link data not obtained: 5 -22[0m
2022-04-13T05:18:32.161Z DEBUG modem << [25:52:00.513,610] [1;31m<err> modem_info_params: Link data not obtained: 8 -22[0m
2022-04-13T05:18:32.177Z DEBUG modem << [25:52:00.531,677] [1;31m<err> modem_info_params: Link data not obtained: 3 -22[0m
2022-04-13T05:18:32.217Z DEBUG modem << [25:52:00.572,326] [1;31m<err> modem_info_params: Network data not obtained: -66[0m
2022-04-13T05:18:32.237Z DEBUG modem << [25:52:00.580,352] [1;31m<err> modem_module: modem_info_params_get, error: -11[0m
2022-04-13T05:18:32.242Z DEBUG modem << [25:52:00.588,226] [0m<inf> event_manager: MODEM_EVT_BATTERY_DATA_NOT_READY[0m

Vipul Panchal said:

The problem is not with registering to a network.

Yes, this is the problem from what I can see. 

[Vipul Panchal]

We have our test network which is configured for NB-IoT (NB1)
I agree, you may be seeing the logs showing problems registering to the network as these are complete logs and we may be in the middle of changing some configuration or setting up the test network.
That's why I am not concerned about the device not registering. What I am more concerned about is when it successfully registers to the network, it faces issues transferring data over TCP packets (The modem seems to not receive any packets for around 20 seconds, highlighted in the screenshot).

[Øyvind]

Please see answer from our nRF91 team:

There are two things that are causing delays and re-transmissions

1. The network has not enabled enableStatusReportSN-Gap configuration in eNB. This means that UE can't report about missing RLC PDU once it has been noticed. Instead, the report of missing RLC PDU is sent after there is nothing to transmit and network polls the device. This brings the delay in attached picture the customer is asking for. RLC PDU from the first TCP segment is lost and UE must wait for re-transmission from network.
2. TCP re-transmissions. Due to missing TCP ACKs the server starts to re-transmit TCP packets. This makes things even worse in lower layers as re-transmission of RLC PDUs happen when there is nothing to be send. TCP re-transmission makes the send queue even longer and missing RLC PDUs are sent with increasing delay.

When it might be difficult to add support for enableStatusReportSN-Gap in network the only way would be trying to increase TCP re-transmission timeout in server side. This may  shorten the RLC PDU re-transmission delay and make the overall transmission faster.

Kind regards,
Øyvind