AZURE IoT Hub Demo on nRF9160: Error -111

[ExAg]

Hi
we have a similar problem as in this forum thread sample-nrf9160-azure-iot-hub.

Environment:
PCA10090 0.8.3 eval boared with modem FW 1.1.0 (latest version does not work)
nRF Connect SDK v2.1.1

What we did so far:
We created an IoT Hub with a device (X.509 CA Signed) on our Azure portal as described here and here.

So now we have a device name and the IoT Hub host name that we need for the demo.

Than we have downloaded the Baltimore CyberTrust Root Certificate

and generated all the necessary Certificates and keys as described here.
So now we have the following files:

We loded the RootCA.pem into our AZURE IoT Hub. It now has the status "verified".

After that we flashed the board with the at_client demo and loaded the certificates and key to the modem as described here, using the LTE Link Monitor.

Set modem to "offline" (AT+CFUN=4)

copy BaltimoreCyberTrustRoot.crt.pem into "CA certificate";
copy "devicename"-public.pem into "Client certificate";
copy "devicename"-private.pem into "Private key";
As "Security tag, we used 10, like the defaul in the Kconfig

With these settings, we  update the certificates.



The problem:
We load the azure_iot_hub project in VS code as active application and we set the Kconfig to
 - use AZURE_IOT_HUB
 - port is 8883
 - TLS sec Tag is 10
 - Azure IoT Hub hostname to our Azur IoT Hub hostname
 - Device ID to our device ID
 - no DPS


We added
CONFIG_LOG=y
CONFIG_AZURE_IOT_HUB_LOG_LEVEL_DBG=y
to our prjconf.

Building the project (for nrf9160dk_nrf9160_ns)

and debugging works fine.

The log output in TeraTerm shows that the modem successfully establishes a connection to our Cell provider.
But then the AZURE MQTT Broker connection fails with error code -111

Here is a part of the log:

connected to LTE network
Azure IoT Hub library initialized
AZURE_IOT_HUB_EVT_CONNECTING
azure_iot_hub_connect failed: -111
[00:01:00.750,610] <dbg> azure_iot_hub: iot_hub_state_set: State transition: STATE_UNINIT --> STATE_DISCONNECTED
[00:01:00.753,692] <dbg> azure_iot_hub: iot_hub_state_set: State transition: STATE_DISCONNECTED --> STATE_CONNECTING
[00:01:00.756,317] <dbg> azure_iot_hub_mqtt: mqtt_state_set: State transition: MQTT_STATE_UNINIT --> MQTT_STATE_DISCONNECTED
[00:01:00.756,408] <dbg> azure_iot_hub: azure_iot_hub_connect: User name: XXXXXXXXXXX.azure-devices.net/DeviceXXXXXXXXX/?api-version=2020-09-30&DeviceClientType=azsdk-c%2F1.4.0-beta.2
[00:01:00.756,408] <dbg> azure_iot_hub: azure_iot_hub_connect: User name buffer size is 128, actual user name size is: 104
[00:01:00.756,469] <dbg> azure_iot_hub_mqtt: broker_init: Resolving IP address for XXXXXXXXXX.azure-devices.net
[00:01:00.835,113] <dbg> azure_iot_hub_mqtt: broker_init: IPv4 Address found XXX.XXX.XXX.XXX
[00:01:00.835,144] <dbg> azure_iot_hub_mqtt: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_CONNECTING
[00:01:00.935,913] <err> azure_iot_hub_mqtt: mqtt_connect, error: -111
[00:01:00.935,943] <dbg> azure_iot_hub_mqtt: mqtt_state_set: State transition: MQTT_STATE_CONNECTING --> MQTT_STATE_DISCONNECTED
[00:01:00.935,943] <err> azure_iot_hub: mqtt_helper_connect failed, error: -111
[00:01:00.936,004] <dbg> azure_iot_hub: iot_hub_state_set: State transition: STATE_CONNECTING --> STATE_DISCONNECTED
+CSCON: 0
RRC mode: Idle
+CEREG: 1,"0978","01007102",7,,,"11100000","11100000"
LTE cell changed: Cell ID: 16806146, Tracking area: 2424

Any suggestions what we are doing wrong?

[Hakon]

We added
CONFIG_LOG=y
CONFIG_AZURE_IOT_HUB_LOG_LEVEL_DBG=y
to our prjconf.

Maybe also adding CONFIG_TLS_CREDENTIALS_LOG_LEVEL_DBG=y will give some valuable information.

[ExAg]

because this issue is very imporant for us, we updated our initial post with some screen shots and more precise descriptions to help understanding what we did and what is the problem with this.

[Hakon]

There could also be a stack overflow causing the sec_tag to be overwritten. Try to enable CONFIG_HW_STACK_PROTECTION or increase the stack and see if it helps.

[ExAg]

1.) Modem FW
We choosed modem FW 1.2.7 because all newer versions (1.3.0 and newer) need nRF9160 SiP Revision 2 according the Nordic release notes here.

We even tried V1.3.3 to be sure, but this version is not responding to any AT-commands anymore. (Tested with the LTE Link Monitor).

2.) sec_tag 212856
As you can see on our earlier screen shots, we have never configured sec_tag to 212856 but only to10. This was exactly to avoid such problems as seen in the modem trace, because 10 is the default value in the initial project configuration.

3.) Stack overflow/size
What should be the expected outcome of the config CONFIG_HW_STACK_PROTECTION you mentioned in your post? We do not see any difference to earlier logs.

By the way: this configuration is displayed in light blue in the prj.conf and not in dark blue, as all the other configurations are. If we hover over, the hint says: Already set to "y" here: nrf9160dk_nrf9160_ns_defconfig(11, 1).

Based on your hint about stack size, we have doubled the 3 heap and stack sizes (CONFIG_HEAP_MEM_POOL_SIZE, CONFIG_MAIN_STACK_SIZE. and CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE) from 4096 to 8000. This has no effect; the release build still runs fine while the debug build still has error -111.

[ExAg]

In addition to our last post, we have some more information according the DK version:
We just received our new 9160-DK that we have ordered to be sure that it is not an DK issue. The new board is a 1.1.0 version.
We have updated the modem firmware to version 1.3.3, loaded the certificates and builded the Azure_iot_hub sample project.
And still the same: Release runs fine; debug generates error -111. And still not depending on any stack sizes...

So we think that this issue definitly not depends on the DK revision.

[ExAg]

Hi Hakon
Any news according our issue? Without solving this, we are more or less blocked.
Without the possibility to debug, it makes no sense for us to go further with this project.

[Hakon]

Apparently there is a bug in the Azure library. You can try this github.com/.../9479

[ExAg]

Debugging seems to work fine now with the github bugfix version of azure_iot_hub_mqtt.c. It even runs on our old 0.8.3 DK.

Thank you for fixing this issue!