cloud_connect() stuck in 22nd connection attempts (using AWS_IOT backend)

[Tjaz]

Hello,

I am on nrf-sdk v1.3.0 and mwf v1.2.0.

We are updating our product to the above mentioned nrf-sfk version and have noticed that MQTT connection gets stuck after 21 successful connections. I have produced a minimal example of the issue that I am sharing with you here.

In our actual firmware, we connect only every once in a while (from once per day to once per week), so this has only become apparent recently.

The test loop is quite simple:

while (1)
{
    lte_lc_connect();
    LOG_INF("connected to lte");

    cloud_connect(cloud_backend);
    k_sem_give(&cloud_conn_sem); // this activates the polling loop
    LOG_INF("connected to cloud");
    k_sleep(K_SECONDS(5));
    cloud_disconnect(cloud_backend);
    close(cloud_backend->config->socket);
    LOG_INF("disconnected from cloud");

    // go offline
    lte_lc_offline();
    LOG_INF("went offline");

    LOG_INF("test loop %d", loop_c++);
}

The code gets stuck in cloud_connect(cloud_backend);
The last thing printed is "IPv4 Address found 18.185.186.31", which appears here in nrf-sdk v1.3.0 in aws_iot.c, line 557.
Looking through the code, the next function called is then at line 711 here.

We had already encountered something similar in march, when we were testing nrf v.1.2.0 - See this bifravst issue:
https://github.com/bifravst/firmware/issues/30
We were stuck in the same place then, but it happened every time.

We need this to work, or, if not possible, a way to determine this has happened and recover from it.
I have attempted to call cloud_connect in a separate thread and then, if connect is stuck for more then 30 seconds, try to abort the thread and re-init the cloud backend. But this then seems to break the AT client, so I can't recover except via reboot.

I am attaching a modem trace, an otti file with power consumption and uart logs, as well as the example source code (it is reduced out of our whole project code).
Code execution gets stuck at around 9:33 in the otti file, and probably a second or two later in the trace.

 cloud_connect_bug_trace_code.zip

Any help would be apreciated.
Regards,
- Tjaž

[Martin Lesund]

Hi Tjaz,

The next modem firmware should have this fixed (releases in the near future), but you can try to use the workaround mentioned here.

For more information about release schedule please contact your RSM. If you do not know who that is please send me a direct message on Devzone and I will provide the information.

BR,
Martin L.

[Tjaz]

Will this be a minor (v1.2.1) release or a major release?
This is important since it is only possible to FOTA upgrade minor releases.

The workaround is thus to disable TLS hostname verification. How might we go about doing that?

Regards,
Tjaž

[Tjaz]

Simply setting CONFIG_MQTT_LIB_TLS=n in prj.conf does not work, since it is selected by AWS_IOT.

warning: MQTT_LIB_TLS (defined at subsys/net/lib/mqtt/Kconfig:28) was assigned the value 'n' but got
the value 'y'. See http://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_MQTT_LIB_TLS.html
and/or look up MQTT_LIB_TLS in the menuconfig/guiconfig interface. The Application Development
Primer, Setting Configuration Values, and Kconfig - Tips and Best Practices sections of the manual
might be helpful too.

as seen in the docs here: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/1.3.0/kconfig/CONFIG_MQTT_LIB_TLS.html

How can I then disable this?

BR,
Tjaž

[Martin Lesund]

Hi Tjaž,

Correct, It will be a minor release so there is possible to do a modem FOTA.

Easiest way to disable TLS host verification is to set 
CONFIG_TLS_HOSTNAME=0 in prj.conf.

Here is the function https://github.com/nrfconnect/sdk-zephyr/blob/master/subsys/net/lib/mqtt/mqtt_transport_socket_tls.c#L73

 

also, I recommend that you disable the AT host library if you do not need it when testing:

CONFIG_AT_HOST_LIBRARY=n 

(should be used just when testing, not recommended for the final application)

 

Best regards,
Martin L.

[Tjaz]

Thank you for the quick reply, but CONFIG_TLS_HOSTNAME does not seem to exist.
I get this when building:

/workdir/ncs/firmware/prj.conf:83: warning: attempt to assign the value '0' to the undefined symbol TLS_HOSTNAME

error: Aborting due to Kconfig warnings

I also can not find it in the Kconfig reference/documentation:

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/1.3.0/kconfig/search.html?q=CONFIG_TLS_HOSTNAME&check_keywords=yes&area=default#

[Martin Lesund]

You are correct, my mistake sorry.
You can instead change it directly in the function instead. (set it directly to avoid the TLS Hostname Verification)

[Tjaz]

Thank you for your help.

I have found it easiest to just comment out lines 72-79 in the file you have linked: https://github.com/nrfconnect/sdk-zephyr/blob/master/subsys/net/lib/mqtt/mqtt_transport_socket_tls.c#L73

I have then verified this fix to be correct over the weekend.

For anyone who might need it, I am attaching a git patch file that does this for you. It can then easily be reverted after the fix is introduced in mfw v1.2.1.
mqtt_tsl_hostname_verification_disable.patch
To apply the patch, cd to your zephyr folder and use:

git apply mqtt_tsl_hostname_verification_disable.patch

# to revert
git apply -R mqtt_tsl_hostname_verification_disable.patch

Best regards,
Tjaž Vračko