Azure IoT Hub DPS library seems to always reprovision our device

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Tue, 09 Apr 2024 11:33:30 GMT

Okay, so I got a reply pretty soon after my last update. It seems the answer is in the following piece of log:

[00:00:07.963,378] <inf> azure_iot_hub_dps: Assigned device ID found:
[00:00:07.963,531] <inf> azure_iot_hub_dps: Assigned device ID length found: 9
[00:00:07.963,714] <inf> azure_iot_hub_dps: Azure IoT Hub hostname found:
[00:00:07.963,928] <inf> azure_iot_hub_dps: Azure IoT Hub hostname length found: 29

This implementation depends on the order of how settings are restored. It expects hostname_len to be restored before the host name, but the nvs backend restores settings in reverse order, so you will have to swap these two settings or make the restore independent of the order.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Tue, 09 Apr 2024 06:23:45 GMT

Hi Vincent

Sorry for the late reply. This was put on hold over Easter, and I'm still looking for the right people to comment on this I'm afraid.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Fri, 15 Mar 2024 12:52:26 GMT

No updates in the internal ticket, so I have pinged the developer working on it for an update now. I will get back to you as soon as I hear something. I'm sorry it's taking so long, but it took a while before finding the correct engineers to assign to this as well.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Thu, 14 Mar 2024 18:01:41 GMT

Hi Simonr,

Any update?

Regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Tue, 20 Feb 2024 11:57:35 GMT

Hi Vincent

Will do. I can confirm that the internal ticket has been assigned to an engineer working on the Azure IoT hub sample.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Mon, 19 Feb 2024 14:29:06 GMT

Thank you for clarifying.

We have documented and included the lingering NVS backend problem in our bug analysis process.
While this isn't a time-critical issue for our immediate project, we are keenly interested in documenting the cause.

Could you please keep us updated on any progress made by your team in investigating the cause of the NVS behaviour on the DK? Even incremental insights would be appreciated.

Thank you for your continued assistance.

Regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Mon, 19 Feb 2024 14:21:58 GMT

Thank you for specifying. I pinged the devs, and there has not been any progress on this unfortunately. I have made an internal ticket so that we can track this issue/investigation properly now, and will get back to you with any updates on it. I don't have an estimate on when exactly that would be unfortunately. Is there any rush on getting the NVS backend up and running on your end so that this should be prioritized?

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Mon, 19 Feb 2024 13:13:05 GMT

Hi Simonr,

Thank you for the previous clarification regarding the fix that addressed the backend override issue. My inquiry today focuses on the ongoing investigation into the NVS backend problem that remains unresolved.

As noted in the commit documentation (e46e68c1cf4fdcf5e7a8d24df7e0b70561248c0b) by Simen S. Røstad: "Why it doesn't work with the NVS backend needs to be investigated."

Could you please provide an update on the status of this NVS backend investigation? Any insights into potential causes would be welcome.

Thank you for your assistance.

Sincerely,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Mon, 19 Feb 2024 07:04:34 GMT

Hi Vincent

Do you refer to the stability fixes or the issue where it doesn't work with the NVS backend?

The backend override has been removed and the mentioned stability fixes + clean-up has been done. The reason was that the nRF7002 board config overrode the hostname and device ID received during DPS. I'm not sure there is more to add than what's described in the sign-off by my colleague Simen.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Fri, 16 Feb 2024 13:26:47 GMT

Hi Simonr,

I am speaking of https://github.com/nrfconnect/sdk-nrf/pull/13180/commits/e46e68c1cf4fdcf5e7a8d24df7e0b70561248c0b

You had already clarify your previous point, in our previous discussion.

Regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Fri, 16 Feb 2024 10:16:02 GMT

Hi again

We concluded this is not a bug, as azure_iot_dps_init has to be run to initialize the library, and azure_iot_dps_start must be run to start the DPS process. Since there is no connection to the DPS server, the DPS library locates the host name and device ID in the flash memory as part of the DPS process and is in the DPS API, not the connection itself.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Tue, 13 Feb 2024 02:04:37 GMT

Hi Simonr,

Did your team finally understand the reason for this bug?

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Fri, 24 Nov 2023 12:22:33 GMT

Hi again Vincent

This isn't a bug, and is intended. The DPS isn't executed twice, the connection to the server that is. But azure_iot_dps_init and _start will still be run.

_init has to be run to initialize the library and _start must be run to start the DPS process. The difference here is that there isn't a connection to the DPS-server, but the DPS library locating the host name and device ID in the flash memory. This is also part of the DPS process, and is happening in the DPS API.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Thu, 23 Nov 2023 15:04:45 GMT

The azure_iot_hub_dps_init gets called precisely twice. When I looked at the code, the azure_iot_hub_connect function in azure_iot_hub.c seems to unnecessarily redo the DPS provisioning process, which should be redundant in this context.

I'm trying to understand the rationale behind its implementation. Could you clarify if the coding approach was intentional due to its minimal impact or if it might be an oversight? I want to ensure I’m not missing any critical aspects of our development strategy.

Additionally, I find identifying the appropriate buffer or configuration parameters challenging. Any guidance or resources you could provide to help me navigate these settings would be precious.

In the meantime, I will continue to explore the Zephyr networking layer in hopes of finding a solution. However, your expert support would expedite this process and enhance my understanding and learning.

Thank you for your continuous support and exceptional work.

Looking forward to your insights.

Best regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Thu, 23 Nov 2023 12:47:24 GMT

Hi Vincent

Does the azure_iot_hub_dps_init get called exactly twice before successfully connecting, or will it just keeps looping like this for X amount of tries before succeeding? If it's always two inits I would guess that the initialization itself is actually called twice, but it could also be that the application waits for X time before retrying to init and connect, and that it just needs more time to connect successfully.

For the errors, I think you can simply try increasing some of the data buffer configurations in your .conf file to avoid this buffer allocation error.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Wed, 22 Nov 2023 13:45:03 GMT

Hi Simon,

Thank you for pushing this ticket. One problem was resolved!
After applying the fixes from that PR, the setting is read correctly on my side and not overwritten.

Did you have feedback on the second point where we go twice inside the azure_iot_hub_dps_init function?

[00:00:07.821,685] <wrn> azure_iot_hub_dps: azure_iot_hub_dps_init
[00:00:07.821,746] <inf> azure_iot_hub_dps: No registration ID provided, using ID from Kconfig: device-01
[00:00:07.821,777] <inf> azure_iot_hub_dps: Setting DPS registration ID: device-01
[00:00:07.821,807] <inf> azure_iot_hub_dps: No ID scope provided, using ID scope from Kconfig: ********
[00:00:07.821,838] <inf> azure_iot_hub_dps: Setting DPS ID scope: : ********
[00:00:07.822,204] <inf> azure_iot_hub_dps: Azure IoT Hub hostname length found: 29
[00:00:07.822,570] <inf> azure_iot_hub_dps: Azure IoT Hub hostname found: ********azure-devices.net
[00:00:07.822,814] <inf> azure_iot_hub_dps: Assigned device ID length found: 9
[00:00:07.822,967] <inf> azure_iot_hub_dps: Assigned device ID found: device-01
[00:00:07.823,028] <inf> azure_iot_hub_dps: Settings fully loaded
[00:00:07.823,089] <inf> azure_iot_hub_dps: State transition: DPS_STATE_UNINIT --> DPS_STATE_DISCONNECTED
[00:00:07.823,089] <inf> azure_iot_hub_sample: DPS registration status: AZURE_IOT_HUB_DPS_REG_STATUS_NOT_STARTED
[00:00:07.823,120] <wrn> azure_iot_hub_dps: azure_iot_hub_dps_start
[00:00:07.823,120] <wrn> azure_iot_hub_dps: Settings found
[00:00:07.823,181] <inf> azure_iot_hub_dps: Device "device-01" is assigned to IoT hub: ********azure-devices.net
[00:00:07.823,211] <inf> azure_iot_hub_dps: To re-register, call azure_iot_hub_dps_reset() first
[00:00:07.823,211] <inf> azure_iot_hub_sample: Already assigned to an IoT hub, skipping DPS
[00:00:07.823,242] <inf> azure_iot_hub_dps: hostname_ptr: 0x200063e4, hostname_len: 29
[00:00:07.823,242] <inf> azure_iot_hub_dps: dev_id_ptr: 0x20006364, dev_id_len: 9
[00:00:07.823,303] <inf> azure_iot_hub_sample: Device ID "device-01" assigned to IoT hub with hostname "*****.azure-devices.net"
[00:00:07.824,157] <inf> azure_fota: Current firmware version: 0.0.0-dev
[00:00:07.824,188] <inf> azure_iot_hub_sample: Azure IoT Hub library initialized
[00:00:07.824,218] <inf> azure_iot_hub_sample: AZURE_IOT_HUB_EVT_CONNECTING
[00:00:07.824,218] <wrn> azure_iot_hub_dps: azure_iot_hub_dps_init
[00:00:07.824,249] <inf> azure_iot_hub_dps: Setting DPS registration ID: device-01
[00:00:07.824,279] <inf> azure_iot_hub_dps: Setting DPS ID scope: *******
[00:00:07.824,676] <inf> azure_iot_hub_dps: Azure IoT Hub hostname length found: 29
[00:00:07.825,012] <inf> azure_iot_hub_dps: Azure IoT Hub hostname found: *******.azure-devices.net
[00:00:07.825,256] <inf> azure_iot_hub_dps: Assigned device ID length found: 9
[00:00:07.825,408] <inf> azure_iot_hub_dps: Assigned device ID found: device-01
[00:00:07.825,469] <inf> azure_iot_hub_dps: Settings fully loaded
[00:00:07.825,500] <wrn> azure_iot_hub_dps: azure_iot_hub_dps_start
[00:00:07.825,531] <wrn> azure_iot_hub_dps: Settings found
[00:00:07.825,592] <inf> azure_iot_hub_dps: Device "device-01" is assigned to IoT hub: *******..azure-devices.net
[00:00:07.825,592] <inf> azure_iot_hub_dps: To re-register, call azure_iot_hub_dps_reset() first

I also have some errors happening on my side that I would appreciate some support/explanation to assess if we can fix them. Could you explain to me what this error is really about and what is happening? Then, I will asses if I can live with it or I need to fix it.

[00:00:08.765,472] <inf> mqtt_helper: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING
[00:00:09.828,948] <err> net_pkt: Data buffer (68) allocation failed.
[00:00:09.829,040] <err> net_tcp: TCP failed to allocate buffer in retransmission
[00:00:10.385,528] <inf> net_mqtt: Connect completed
[00:00:10.385,589] <inf> mqtt_helper: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_TRANSPORT_CONNECTED
[00:00:10.385,620] <inf> mqtt_helper: State transition: MQTT_STATE_TRANSPORT_CONNECTED --> MQTT_STATE_CONNECTING

Again, thank you and your team for the support.
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Wed, 22 Nov 2023 07:21:39 GMT

Hi again Vincent

We were able to reproduce this on our end in the meantime, and one of our devs discovered that the host name and dev ID is not loaded correctly. This morning I was told a pull request has been made where the settings are changed on the backend. Please check it out and confirm whether it solves the issues on your end. https://github.com/nrfconnect/sdk-nrf/pull/13180/commits/e46e68c1cf4fdcf5e7a8d24df7e0b70561248c0b

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Mon, 20 Nov 2023 13:29:01 GMT

Hi Simonr,

Thank you for your continued support. Before proceeding with the Wireshark trace, which I am fully prepared to conduct, I would like to bring to light some key observations and concerns.

From my understanding and experience with the system, the initial communication with the Azure IoT Hub, including connecting and sending telemetry, seems to function correctly. This includes the use of the DPS for initial provisioning and connection. However, the problem becomes apparent during the reconnection phase, where the device gets reprovisioned each time due to the inability to bypass the DPS. This leads to unnecessary reprovisioning despite the device being previously provisioned and connected to the Azure IoT Hub.

The core issue seems to be rooted more in the Nordic code handling rather than a communication problem. Specifically, the application's failure to properly recognize and utilize the saved Azure IoT Hub information gives the impression of no previous information being saved, triggering the reprovisioning process. This was detailed in my previous communication, highlighting concerns about the azure_iot_hub_dps_start function in azure_iot_hub_dps.c and the apparent issues with thread synchronization or variable caching, particularly with the dps_reg_ctx global variable.

In light of these observations, I am keen to understand the specific objectives behind the request for a Wireshark trace:

What aspects or anomalies in the network communication are we looking to identify in the Wireshark trace that could contribute to the reconnection and reprovisioning issues?
How will the insights from the Wireshark trace help us address the issue of the application not recognizing the saved Azure IoT Hub information, especially considering this seems more related to the Nordic code’s handling of the process?

Understanding the reasoning behind this request will greatly assist me in ensuring that the Wireshark trace is focused and relevant and will help us collaboratively pinpoint the exact nature of the problem more effectively.

Looking forward to your insights and guidance on this matter.

Best regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Mon, 20 Nov 2023 12:52:09 GMT

Hi again Vincent

Thank you for the extra information. Would it be possible for you to provide a Wireshark trace so the Azure library developers can take a look and see what's going on over the air? To do so you can connect to the nRF7002 DK to a Wi-Fi hotspot on you computer and capture a trace from that with Wireshark. Let me know if you need any further details. That way we can make sure what is going on over the air exactly, and won't have to make any guesses to what's going on here.

Best regards,

Simon

RE: Azure IoT Hub DPS library seems to always reprovision our device

Actif — Thu, 16 Nov 2023 16:08:10 GMT

Hi Simonr,

Thank you for your prompt response and the suggestions regarding the Azure IoT Hub and DPS configuration for the nRF7002. I appreciate your assistance.

I want to clarify a few points that might have been overlooked in my initial query. From the log, it is evident that the registration ID is being correctly pulled from Kconfig, as indicated by these lines:

[00:00:07.963,134] <inf> azure_iot_hub_dps: No registration ID provided, using ID from Kconfig: device-01
[00:00:07.963,165] <inf> azure_iot_hub_dps: Setting DPS registration ID: device-01

This confirms that the DPS overlay is active and functioning. However, the crux of the issue lies elsewhere. Even when the Azure IoT Hub information is saved and retrievable from the Zephyr setting module (under the key DPS_SETTINGS_KEY "azure_iot_hub"), the device persists in processing through the DPS service, which is not the expected behaviour.

The primary concern revolves around the azure_iot_hub_dps_start function in azure_iot_hub_dps.c, where the DPS is expected to be skipped if saved Azure IoT Hub information is found. Despite this, the dps_reg_ctx global variable doesn't seem to update accordingly in the DPS thread, suggesting possible issues with thread synchronization or variable caching.

Additionally, the azure_iot_hub_connect function in azure_iot_hub.c seems to unnecessarily redo the DPS provisioning process, which should be redundant in this context.

I believe these insights point towards a more specific area of the problem. I would greatly appreciate it if you could provide guidance or suggestions on these particular aspects:

Ensuring proper synchronization and caching mechanisms for the dps_reg_ctx global variable across different threads.
Understanding why the DPS service is reinitiated despite having valid Azure IoT Hub details in the settings.
Addressing the apparent redundancy in the DPS provisioning process within the azure_iot_hub_connect function.

Your expertise and deeper insights into these areas would be beneficial.

We are looking forward to your guidance.

Best regards,
Vincent

RE: Azure IoT Hub DPS library seems to always reprovision our device

Simonr — Wed, 15 Nov 2023 13:30:42 GMT

Hi Vincent

Yes, the Azure hub IoT sample should be tested for the nRF7002 and work out of the box. I see that your log reports that no registration ID has been added, so can you confirm that you've set the required configurations in your project required to use DPS on your end?

If DPS is used, use the Kconfig fragment found in the overlay-dps.conf file and change the desired configurations there. As an example, the following should compile with DPS for nRF7002DK:

west build -p -b nrf7002dk_nrf5340_cpuapp -- -DOVERLAY_CONFIG=overlay-dps.conf

CONFIG_AZURE_IOT_HUB_DPS - Enables Azure IoT Hub DPS.
CONFIG_AZURE_IOT_HUB_DPS_REG_ID - Sets the Azure IoT Hub DPS registration ID. It can be provided at run time. By default, the sample uses the device ID as the registration ID and sets it at run time.
CONFIG_AZURE_IOT_HUB_DPS_ID_SCOPE - Sets the DPS ID scope of the Azure IoT Hub. This can be provided at run time.

Best regards,

Simon