RE: nrf9160 modem stops reconnecting

ubieda — Mon, 17 Aug 2020 13:41:31 GMT

Hakon, thanks for the reply.

We are very likely to have units deployed by the time this modem update comes, so it would be great if that update fix is upgreadable via FOTA (we are currently at MFW V1.2.0, as mentioned above).

Luis.

RE: nrf9160 modem stops reconnecting

Hakon — Mon, 17 Aug 2020 13:22:38 GMT

There will be a fix to this in the next modem firmware. I'm not sure when it will be released.

RE: nrf9160 modem stops reconnecting

ubieda — Fri, 07 Aug 2020 18:48:23 GMT

Guys, I would like to report an exact condition on my end, which is extremely consistent: I'll explain below:

Software used:
- NCS V1.3.0
- Modem Firmware V1.2.0

Issue Description:
- Our application communicates to AWS IoT (MQTT over TLS) periodically to exchange information. A key requirement is that we must support system operation with and without PSM/eDRX.
- Every 15 minutes, a socket is created, connection established, information exchanged and socket & connection closed until next time. After 22 times, the system gets stuck attempting to establish the MQTT connection (see console output below).
- We have more than 5 units running this code and the behavior is consistent across all of them.
- The behavior persists and does not change regardless of setting on each interaction AT+CFUN=1 (at the beginning) & AT+CFUN=0 (at the end), or not enabling/disabling LTE Link at all.
- Our code/variant using PSM does not have this behavior and has performed well so far on all the units in which it is running.

Here is some console output we got when enabling CONFIG_NET_LOG=y. We reduced the reporting rate from 15-min to 30-secs to replicate the issue faster:

/* Beginning of Log: Here's the work item that interact periodically with AWS IoT. After 22 times fails */

[00:11:53.324,768] <dbg> app.cloud_interact_work_fn: cloud_interact_work_fn(): 22
[00:11:53.332,794] <dbg> app.cloud_interact_work_fn: lte_link_manager_enable()
[00:11:53.350,189] <dbg> lte_link_manager.lte_link_manager_enable: Re-enabling lte-link
[00:11:55.258,880] <inf> lte_link_manager: LTE_LC_EVT_NW_REG_STATUS: 2
[00:11:55.972,137] <inf> lte_link_manager: LTE_LC_EVT_NW_REG_STATUS: 1
[00:11:55.979,339] <dbg> app.cloud_interact_work_fn: cloud_manager_open_connection()
[00:11:55.988,037] <dbg> aws_iot.broker_init: IPv4 Address found 13.236.246.246
[00:11:55.996,398] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20021490): Created socket 1
[00:12:00.547,760] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20021490): Connect completed
[00:12:00.557,373] <dbg> net_mqtt_enc.connect_request_encode: (0x20021490): Encoding Protocol Description. Str:MQTT Size:00000004.
[00:12:00.569,671] <dbg> net_mqtt_enc.pack_utf8_str: (0x20021490): >> str_size:00000006 cur:0x20025369, end:0x20025464
[00:12:00.580,963] <dbg> net_mqtt_enc.pack_uint16: (0x20021490): >> val:0004 cur:0x20025369, end:0x20025464
[00:12:00.591,247] <dbg> net_mqtt_enc.connect_request_encode: (0x20021490): Encoding Protocol Version 04.
[00:12:00.601,409] <dbg> net_mqtt_enc.pack_uint8: (0x20021490): >> val:04 cur:0x2002536f, end:0x20025464
[00:12:00.611,480] <dbg> net_mqtt_enc.pack_uint8: (0x20021490): >> val:00 cur:0x20025370, end:0x20025464
[00:12:00.621,520] <dbg> net_mqtt_enc.connect_request_encode: (0x20021490): Encoding Keep Alive Time 04b0.
[00:12:00.631,713] <dbg> net_mqtt_enc.pack_uint16: (0x20021490): >> val:04b0 cur:0x20025371, end:0x20025464
[00:12:00.642,028] <dbg> net_mqtt_enc.connect_request_encode: (0x20021490): Encoding Client Id. Str:CLIENT_DEV1 Size:0000000b.
[00:12:00.654,113] <dbg> net_mqtt_enc.pack_utf8_str: (0x20021490): >> str_size:0000000d cur:0x20025373, end:0x20025464
[00:12:00.665,374] <dbg> net_mqtt_enc.pack_uint16: (0x20021490): >> val:000b cur:0x20025373, end:0x20025464
[00:12:00.675,689] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20021490): << msg type:0x10 length:0x00000017
[00:12:00.686,431] <dbg> net_mqtt_enc.packet_length_encode: (0x20021490): >> length:0x00000017 cur:0x00000000, end:0x00000000
[00:12:00.698,394] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20021490): Fixed header length = 02
[00:12:00.708,251] <dbg> net_mqtt_enc.pack_uint8: (0x20021490): >> val:10 cur:0x20025367, end:0x20025464
[00:12:00.718,292] <dbg> net_mqtt_enc.packet_length_encode: (0x20021490): >> length:0x00000017 cur:0x20025368, end:0x20025464

/* Here is where it gets stuck. The reboot is issued by WDT after 1-min blocked here */

H*** Booting Zephyr OS build v2.3.0-rc1-ncs1  ***
Flash regions           Domain          Permissions
00 00 0x00000 0x08000   Secure          rwxl
01 31 0x08000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 0 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU1             Non-Secure      OK
22 NRF_EGU2             Non-Secure      OK
23 NRF_DPPIC            Non-Secure      OK
24 NRF_GPIOTE1          Non-Secure      OK
25 NRF_REGULATORS       Non-Secure      OK

SPM: NS image at 0xc000
SPM: NS MSP at 0x2002b028
SPM: NS reset vector at 0x17b0d
SPM: prepare to jump to Non-Secure image.
/* End of log */

We are really looking forward to the outcome of this issue, since it is an important requirement in our application.

Thanks in advance,
Luis.

RE: nrf9160 modem stops reconnecting

Hakon — Wed, 15 Jul 2020 12:13:31 GMT

Can you try using the latest version of bsdlib? Since you are using modem firmware 1.2.0 you should also use the latest bsdlib.

RE: nrf9160 modem stops reconnecting

Vlad T — Thu, 02 Jul 2020 09:45:47 GMT

Hello,

Thank you for your reply.

I think that the problem is bsdlib/modem related. The bug reproduces with the https_client example, there is no MQTT related stuff in the project. Please see my reply above.

Best regards,

Vlad

RE: nrf9160 modem stops reconnecting

Hakon — Thu, 02 Jul 2020 09:37:29 GMT

Hello,

have you checked that it isn't the broker that's dropping the connection after reconnecting a number of times? Do you get any error codes? Can you provide a more detailed log output with these settings enabled?

CONFIG_NET_LOG=y
CONFIG_MQTT_LOG_LEVEL_DBG=y

RE: nrf9160 modem stops reconnecting

Vlad T — Thu, 02 Jul 2020 08:51:35 GMT

Hello,

It seems that the bug is triggered by setsockopt(fd, SOL_TLS, TLS_HOSTNAME, ...., ...) line.

I have tested the "https_client" sample also. It can connect and disconnect forever until I set the TLS_HOSTNAME socket option. Then it starts to hang after 22 cycles.

Please see the attached project.

Best regards,

Vlad

devzone.nordicsemi.com/.../5287.https_5F00_client.zip