NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

Didrik Rokhaug — Tue, 10 Mar 2020 16:11:25 GMT

Hi.

We have identified a problem in the modem as the source of this bug.

As the problem is caused by enabling TLS session caching, the workaround until the modem problem is fixed is to not use TLS session caching.

The reason for this problem to surface in NCS v1.2.0 is that the newest version of bsdlib enables TLS session caching by default.

You can therefore either use the previous version of bsdlib (0.6.0), or you can disable TLS session caching using socket options:

#include <nrf_socket.h>

nrf_sec_session_cache_t session_cache = 0;

nrf_setsockopt(socket, NRF_SOL_SECURE,

        NRF_SO_SEC_SESSION_CACHE, &session_cache,

        sizeof(session_cache));

We are working on solving the problem in the modem.

Best regards,

Didrik

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

JVantol — Tue, 10 Mar 2020 15:39:41 GMT

This works fine, but I'm wondering when a proper fix will be forthcoming?

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

dvm_orbey — Fri, 06 Mar 2020 10:08:11 GMT

Also worked for me, thanks.

Best regards

Dvm

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

jreinhed — Thu, 05 Mar 2020 07:38:44 GMT

This worked for me. Thank you!

Re: my modem trace, it's possible that I aborted it too soon.

Regards,

Johan

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

Didrik Rokhaug — Wed, 04 Mar 2020 16:22:38 GMT

Thanks for the traces.

JVantol, the modem trace is indeed valid this time, and it does look like we are seeing the same problem.

jreinhed, I could not see any signs of the error in your trace. Are you certain the trace captured the error?

We are still working on identifying the source of the error, but it seems like it was introduced in bsdlib v6.1.

If I use v6.0 instead, the error does not occur for me.

You can revert to v6.0 by replacing the 1.2.0 revision of nrfxlib on line 72 in <ncs>/nrf/west.yml with 3e381d09f1d305e230435f5b6e4c9ef928b6a697, and running "west update".

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

jreinhed — Wed, 04 Mar 2020 07:44:42 GMT

Hi Didrik,

I am using NCS v1.2.0 on a Thingy:91 for my application, but am having the same problem on the nRF9160 DK. I'm connecting to an Azure Device Provisioning Service using secure MQTT, and then connecting to the IoT Hub that I get assigned. The application seems to use socket no. 2 for both connections. It worked using NCS v1.1.0.

I am attaching a modem trace from my attempt this morning.

Regards,

Johan

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_03_2D00_04T07_2D00_36_2D00_09.418Z.bin

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

JVantol — Tue, 03 Mar 2020 17:40:27 GMT

Here's a new trace. I think it's valid this time.

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_03_2D00_03T17_2D00_31_2D00_00.607Z.bin

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

JVantol — Tue, 03 Mar 2020 16:00:19 GMT

Didrik and Johan,

I think both of you were wondering how I worked around this problem.

Here is the procedure.

Open MQTT Connection 1 (with TLS)

Open MQTT Connection 2 (with TLS)

Service connection 1 (in this case AWS FOTA)

Close Connection 1.

Service connection 2.

The time require to service connection 1 is short enough that connection 2 does not time out, it's well within the mqtt poll interval.

Didrik,

I'll try the modem log again.

Josh

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

Didrik Rokhaug — Tue, 03 Mar 2020 15:38:57 GMT

Hi jreinhed and JVantol.

I have been able to re-create your issue on my end, using the cloud_client sample. I was not able to re-create it with the mqtt_simple sample, as the problem seems to be related to the TLS negotiation. I have also not been able to re-create JVantol's workaround yet. Simply opening another socket does not seem to be enough (I'll keep working on it tomorrow). While I have captured a modem trace and forwarded it to our R&D team for further analysis, I would still like a trace from both of you. That way, I can verify that we see the same problem.

jreinhed, could you give more details about your application? What other sockets do you have open (or are you using any libraries that could be using sockets)? Which socket number is the problem? What protocol are you using? What server are you connecting to?

I'll keep you updated when I know more.

Best regards,

Didrik

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

jreinhed — Tue, 03 Mar 2020 15:03:00 GMT

Hi,

I'm having the exact same problem. I'm also curious about how you force it to use another socket.

Regards,

Johan

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

JVantol — Fri, 28 Feb 2020 16:22:22 GMT

Didrik,

I followed the guide carefully, including all the steps you just mentioned. I'll re-check and see if I can figure out what might have gone wrong.

If you try to reproduce, simply make an mqtt connection, close it, and then make it again, that should demonstrate the issue.

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

Didrik Rokhaug — Fri, 28 Feb 2020 16:19:06 GMT

Your modem trace was empty (or rather, full of zeroes).

Is your application using SPI1, TWI1 or UART1 to something? If so, that could interfere with the modem trace.

Did you add CONFIG_BSD_LIBRARY_TRACE_ENABLED=y to your prj.conf (and re-open the project if you are using SES)?

Did you send AT%XSYSTEMMODE=1,2, then saved it to flash by sending AT+CFUN=0?

Could you try to capture a new trace?

I'll also try to re-create the issue on my end.

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

JVantol — Thu, 27 Feb 2020 15:20:37 GMT

Didrik,

I've attached a trace file. I had some trouble getting a trace in our target hardware and wound up using the DK to do it.

What I mean by "re-using" a socket is that the same socket number would get used for both connections. You can note this by watching the net_mqtt log output.

I've adapted the aws_fota stuff from the aws_fota sample. To demo this problem, I simply ran aws_fota_check twice. The first one works, the second hangs early on, when it trys to call nrf_connect. Our normal application has us connecting to AWS for a fota check at startup, then disconnecting and connecting to Azure via mqtt.

To demo this I'm simply doing what the aws_fota example does, except that after a timeout, I'm closing the mqtt connection.

You can see a trimmed log below, showing the issue.



AWS fota check starting

[00:00:04.098,114]

[0m<dbg> aws_fota_wrapper.aws_fota_check: MQTT AWS FOTA check starting, version: v1.0.14

[0m

[00:00:04.208,312]

[0m<dbg> aws_fota_wrapper.broker_init: IPv4 Address 35.153.241.198

[0m

[00:00:04.216,278]

[0m<dbg> aws_fota_wrapper.client_init: client_id: krmp352656100158107

[0m

[00:00:04.235,260]

[0m<dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20025940): Created socket 2

[0m

%CESQ: 47,2,12,1

rsrp value = 0

%CESQ: 52,2,12,1

rsrp value = 47

[00:00:08.480,560]

[0m<dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20025940): Connect completed

... lines elided - everything in the elided lines works as expected....

[0m<dbg> aws_fota.on_publish_evt: Received topic: $aws/things/krmp352656100158107/jobs/$next/get/accepted

[0m

[00:00:10.070,922]

[0m<inf> aws_fota: Checking for an available job

[0m

[00:00:10.077,667]

[0m<dbg> aws_fota.get_job_execution: Got only one field: {"clientToken":"","timestamp":1582816345}

[0m

[00:00:10.088,470]

[0m<inf> aws_fota: No queued jobs for this device

[0m

[00:00:38.681,243]

[0m<dbg> aws_fota_wrapper.aws_fota_check: Disconnecting MQTT client...

[0m

[00:00:38.689,727]

[0m<dbg> net_mqtt.client_write: (0x20025940): [0x200240a0]: Transport writing 2 bytes.

[0m

[00:00:38.708,587]

[0m<dbg> net_mqtt.client_write: (0x20025940): [0x200240a0]: Transport write complete.

[0m

[00:00:38.718,383]

[0m<dbg> net_mqtt_sock_tls.mqtt_client_tls_disconnect: (0x20025940): Closing socket 2

[0m

[00:00:38.729,370]

[1;31m<err> aws_fota_wrapper: [mqtt_evt_handler:183] MQTT client disconnected 0



... fota check run for a second time. Same code path, not that it hangs after creating the socket.....

[0m

AWS fota check finished

AWS fota check 2 starting

[00:00:38.743,011]

[0m<dbg> aws_fota_wrapper.aws_fota_check: MQTT AWS FOTA check starting, version: v1.0.14

[0m

[00:00:38.753,326]

[0m<dbg> aws_fota_wrapper.broker_init: IPv4 Address 35.153.241.198

[0m

[00:00:38.761,474]

[0m<dbg> aws_fota_wrapper.client_init: client_id: krmp352656100158107

[0m

[00:00:38.778,503]

[0m<dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20025940): Created socket 2

[0m

I've attached a modem trace of this scenario as well.

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_02_2D00_27T15_2D00_06_2D00_54.982Z.bin

RE: NRF modem 1.1.1 firmware, 1.2 SDK. Re-using sockets causes hang in nrf_connect.

Didrik Rokhaug — Wed, 26 Feb 2020 12:55:43 GMT

Hi.

One of the changes between NCS v1.1.0 and NCS v1.2.0 is that the MQTT library now closes the socket in mqtt_disconnect, instead of after another call to mqtt_live.

How do you "re-use" the socket, and how do you "force" it to use another new socket?

Also, if you could capture a modem trace, that would help me track what happens to the sockets on the modem side.

You can learn how to capture a modem trace in this guide: https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/how-to-get-modem-trace-using-trace-collector-in-nrf-connect

Best regards,

Didrik