NRF9160 Azure IoT hub DPS Certification Connection Rejected

[jczacharia]

Hi Nordic

Sample: asset_tracker_v2 (with only changes to overlay-azure.conf)
SDK: 2.5.2
Modem FW: nrf9160_1.3.5

I have followed this tutorial https://learn.microsoft.com/en-us/azure/iot-dps/tutorial-custom-hsm-enrollment-group-x509?tabs=windows&pivots=programming-language-ansi-c for getting my device connected to IOT Hub via DPS. Which I have followed and re-tried many times.

On Azure, in my DPS Certificates, I've uploaded azure-iot-test-only.root.ca.cert.pem, per tutorial.
And in my Managed Enrollments, I've added an Enrollment Group with the azure-iot-test-only.intermediate.cert.pem, per tutorial.

But, I am struggling with the certification and getting a valid connection.

The error that keeps persisting is:

[00:00:15.799,896] <err> azure_iot_hub_dps: Connection was rejected with return code 5
[00:00:15.799,926] <wrn> azure_iot_hub_dps: Is the device certificate valid?
[00:00:15.799,987] <err> mqtt_helper: Library is in the wrong state (MQTT_STATE_DISCONNECTED), MQTT_STATE_CONNECTED required
[00:00:15.800,018] <err> azure_iot_hub_dps: Failed to disconnect gracefully, error: -95
[00:00:15.800,048] <err> azure_iot_hub_dps: There was a failure during DPS registration, process is stopped
[00:00:15.801,086] <err> mqtt_helper: Cloud MQTT input error: -111

Here is what I have successfully uploaded to the modem: 

And my overlay-azure.conf looks like:

CONFIG_AZURE_IOT_HUB=y
CONFIG_AZURE_IOT_HUB_DPS=y
CONFIG_AZURE_IOT_HUB_AUTO_DEVICE_TWIN_REQUEST=y

# Increase the number of maximum message properties that can be parsed by the Azure IoT Hub library.
# Needed to be able to parse P-GPS responses.
CONFIG_AZURE_IOT_HUB_MSG_PROPERTY_RECV_MAX_COUNT=4

# Azure IoT Hub options that must be configured in order to establish a connection.
CONFIG_AZURE_IOT_HUB_DPS_HOSTNAME="global.azure-devices-provisioning.net"
CONFIG_AZURE_IOT_HUB_DPS_ID_SCOPE="0ne********" # Stars replaces with my ID scope

# MQTT helper library
CONFIG_MQTT_HELPER_RX_TX_BUFFER_SIZE=2048
CONFIG_MQTT_HELPER_STACK_SIZE=4096
CONFIG_MQTT_HELPER_SEC_TAG=11

# MQTT Transport library
# Maximum specified MQTT keepalive timeout for Azure IoT Hub is 1177 seconds.
CONFIG_MQTT_KEEPALIVE=1177

The full trace is:

*** Booting nRF Connect SDK v2.5.2 ***
[00:00:00.257,446] <inf> app_event_manager: APP_EVT_START
[00:00:00.506,896] <inf> app_event_manager: MODEM_EVT_INITIALIZED
[00:00:00.516,571] <inf> azure_fota: Current firmware version: 0.0.0-development
[00:00:00.570,159] <inf> app_event_manager: MODEM_EVT_LTE_CONNECTING
[00:00:01.293,548] <inf> app_event_manager: DATA_EVT_CONFIG_INIT
%CESQ: 48,2,10,1
+CEREG: 2,"AEFE","005C4C16",7
[00:00:02.401,519] <inf> app_event_manager: MODEM_EVT_LTE_CELL_UPDATE
+CSCON: 1
+CGEV: ME PDN ACT 0,0
[00:00:03.967,681] <inf> app_event_manager: MODEM_EVT_LTE_CONNECTED
[00:00:03.968,902] <inf> app_event_manager: CLOUD_EVT_CONNECTING
+CNEC_ESM: 50,0
%MDMEV: SEARCH STATUS 2
+CEREG: 5,"AEFE","005C4C16",7,,,"00001010","11000001"
%XTIME: "0A","4220328121620A","00"
[00:00:04.073,364] <inf> app_event_manager: MODEM_EVT_LTE_PSM_UPDATE
[00:00:04.073,913] <inf> app_event_manager: DATA_EVT_DATE_TIME_OBTAINED
%CESQ: 47,2,5,0
%CESQ: 47,2,10,1
+CSCON: 0
+CSCON: 1
[00:00:17.401,397] <err> azure_iot_hub_dps: Connection was rejected with return code 5
[00:00:17.401,397] <wrn> azure_iot_hub_dps: Is the device certificate valid?
[00:00:17.401,458] <err> mqtt_helper: Library is in the wrong state (MQTT_STATE_DISCONNECTED), MQTT_STATE_CONNECTED required
[00:00:17.401,489] <err> azure_iot_hub_dps: Failed to disconnect gracefully, error: -95
[00:00:17.401,519] <err> azure_iot_hub_dps: There was a failure during DPS registration, process is stopped
[00:00:17.402,832] <err> mqtt_helper: Cloud MQTT input error: -111
+CSCON: 0
%CESQ: 46,2,4,0
%CESQ: 255,0,255,0

Hope you can help.

Thank you!
-Jeremy

[Sigurd Hellesvik]

Hi,

Have you tried to connect your device to Azure IoT Hub without DPS?

Regards,
Sigurd Hellesvik

[jczacharia]

Yes, I have.

I followed this tutorial: developer.nordicsemi.com/.../azure_iot_hub.html

I have tried 3 different ways of creating credentials:

• https://learn.microsoft.com/en-us/azure/iot-hub/tutorial-x509-test-certs?tabs=windows
• https://learn.microsoft.com/en-us/azure/iot-dps/tutorial-custom-hsm-enrollment-group-x509?tabs=windows&pivots=programming-language-ansi-c
• https://github.com/Azure/azure-iot-sdk-c/blob/main/tools/CACertificates/CACertificateOverview.md

But have not been able to get any to work. I still always see the "Is the device certificate valid?" error.

My overlay-azure.conf for one device:

CONFIG_AZURE_IOT_HUB=y
CONFIG_AZURE_IOT_HUB_DPS=n
CONFIG_AZURE_IOT_HUB_AUTO_DEVICE_TWIN_REQUEST=y

# Increase the number of maximum message properties that can be parsed by the Azure IoT Hub library.
# Needed to be able to parse P-GPS responses.
CONFIG_AZURE_IOT_HUB_MSG_PROPERTY_RECV_MAX_COUNT=4

CONFIG_AZURE_IOT_HUB_HOSTNAME="******.azure-devices.net" # Replace with my IoT Hub hostname
CONFIG_AZURE_IOT_HUB_DEVICE_ID="device-01"

# MQTT helper library
CONFIG_MQTT_HELPER_RX_TX_BUFFER_SIZE=2048
CONFIG_MQTT_HELPER_STACK_SIZE=4096
CONFIG_MQTT_HELPER_SEC_TAG=11

# MQTT Transport library
# Maximum specified MQTT keepalive timeout for Azure IoT Hub is 1177 seconds.
CONFIG_MQTT_KEEPALIVE=1177

I have tried each combination of creating credentials from each above source with both DPS and single device using both the asset_tracker_v2 codebase and as well as the https://github.com/NordicSemiconductor/asset-tracker-cloud-firmware-azure codebase.

But no matter which combination I pick, I cannot get the certificates provisioned correctly to establish a valid communication.

[Mikael Jansson]

Did you make sure to copy the ETAG from the JSON response for each invocation of "az iot hub"? 

az iot hub certificate create --hub-name "$HUBNAME" --name "$CERTNAME" --path ca/root-ca-cert.pem

> read the ETAG, use in following command:

az iot hub certificate generate-verification-code --hub-name $HUBNAME --name MYROOTCA --etag "$ETAG"

> read the ETAG from the above command, use it in the following:

az iot hub certificate verify --hub-name $HUBNAME --name $CERTNAME --etag $ETAG2 --path certs/client-cert.pem

and so on.

[Byter]

Hello, I'm receiving the exact same issue. Followed the tutorial 1:1, copied the ETAG from the JSON response at each invocation and all. Not sure what the issue is at this point

[jczacharia]

Yes, I have done that. I can't seem to shake the -111 error (Is the device certificate valid?). I have a feeling it might be related to Azure DigiCert migration.

[Neso]

Hi. Did you find a solution to this?

[jczacharia]

Unfortunately not, I haven’t worked on the project in a few months so things might have changed on Azure since then. 

[Mikael Jansson]

I wrote this script to create modem keys, send them to Azure and prepare DPS for automatic registration.

It uses MQTT SEC TAG 42 and SECONDARY 43.  Requires signing CA and CAs previously generated.

#!/bin/bash

DEFAULT_MODEM_PORT="/dev/tty.usbserial-B002XPXB"
MODEM_PORT=
RESGROUP="<My-IoT-ResourceGroup>"
HUBNAME="<My-IoTHub>"
CERTNAME="my_root_ca"
IOTHUB_URL="<my-iot-hub>.azure-devices.net"
ENROLLMENT_NAME="<my-enrollment>"
DPS="<my-dps>"
SEC_TAG=42 # for GlobalSign G2 root CA
SECONDARY_SEC_TAG=43 # for legacy Baltimore root CA
TENANT="<my-tenant>"

check() {
    st=$1
    if [[ $st != 0 ]]; then
        echo "error occured: $st"

        echo "are you logged in? otherwise, run::"
        echo
        echo "    az login --scope https://management.core.windows.net//.default"
        exit $st
    fi
}

configure() {
    echo "********* (step 0) removing old client data: client-csr.der, client-csr.pem, client-cert.pem"
    rm -f certs/client-csr.der certs/client-csr.pem certs/client-cert.pem > /dev/null
    echo "********* (step 1) listing current data"
    nrfcredstore $MODEM_PORT list

    echo "********* (step 2) deleting old data: will create errors first time run on a device, disregard these."
    nrfcredstore $MODEM_PORT delete $SEC_TAG CLIENT_CERT > /dev/null
    nrfcredstore $MODEM_PORT delete $SEC_TAG CLIENT_KEY > /dev/null

    echo "********* (step 3) generating private key in device and output CSR in sec tag $SEC_TAG => certs/client-csr.der"
    nrfcredstore $MODEM_PORT generate $SEC_TAG certs/client-csr.der
    check $?

    #Convert the CSR from DER format to PEM format using the following command:
    echo "********* (step 4) convert CSR from DER to PEM => certs/client-csr.pem"
    openssl req -inform DER -in certs/client-csr.der -outform PEM -out certs/client-csr.pem
    check $?

    echo "********* (step 5) \"give\" the CSR client-csr.pem to sub-CA (sub-ca-key.pem, sub-ca-cert.pem) to generate client cert => certs/client-cert.pem"
    ./cert_tool.py sign 
    check $?

    DEVICE_CN=$(openssl x509 -in certs/client-cert.pem -noout -subject | sed -e 's/.*CN=\(.*\)/\1/g')
    [[ $DEVICE_CN != "" ]]
    check $?
    echo "********* (step 5.5) DEVICE_CN: $DEVICE_CN"
    az iot dps enrollment delete --enrollment-id $DEVICE_CN --resource-group $RESGROUP  --dps-name $DPS > /dev/null

    echo "********* (step 6) write signed client cert in sec tag $SEC_TAG => certs/client-cert.pem"
    nrfcredstore $MODEM_PORT write $SEC_TAG CLIENT_CERT certs/client-cert.pem
    check $?

    echo "********* (step 7) listing current credentials in sec tag $SEC_TAG"
    nrfcredstore $MODEM_PORT list | grep -w $SEC_TAG

    echo "********* (step 8) registering device-id my-device-revN-$DEVICE_CN using certs/client-cert.pem"

    az iot dps enrollment create --enrollment-id $DEVICE_CN --device-id "my-device-revN-$DEVICE_CN" \
    --provisioning-status enabled --resource-group $RESGROUP \
    --iot-hubs $IOTHUB_URL --attestation-type x509 \
    --certificate-path certs/client-cert.pem --dps-name $DPS \
    --allocation-policy static  > /dev/null
    check $?

    echo "********* (step 9) adding credentials for DigiCert and Baltimore"
    nrfcredstore $MODEM_PORT write $SEC_TAG ROOT_CA_CERT DigiCertGlobalRootG2.crt.pem
    check $?
    nrfcredstore $MODEM_PORT write $SECONDARY_SEC_TAG ROOT_CA_CERT BaltimoreCyberTrustRoot.crt.pem
    check $?

    echo "********* (step 10) listing current credentials in primary and secondary sec tags"
    nrfcredstore $MODEM_PORT list | grep -Ew '42|43'
    check $?
}

help() {
    echo "usage: $0 [--skip-install] [<modem port>]"
    echo
    echo "--skip-install:   don't install firmware"
    echo "<modem port>:     the port where the modem is connected, default $DEFAULT_MODEM_PORT"
    echo
    echo "requirements:"
    echo "- BaltimoreCyberTrustRoot.crt.pem"
    echo "- DigiCertGlobalRootG2.crt.pem"
    echo "ca/"
    echo "- root-ca-cert.pem"
    echo "- root-ca-key.pem"
    echo "- sub-ca-cert.pem"
    echo "- sub-ca-key.pem"
    echo
    echo "intermediates:"
    echo "certs/"
    echo "- client-cert.pem"
    echo "- client-csr.der"
    echo "- client-csr.pem"
    echo
    echo "Possible devices":
    ls -1 /dev/tty.*serial*
    exit 0
}

install_firmware() {
    GDB=/opt/nordic/ncs/toolchains/20d68df7e5//opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb
    $GDB -q \
         -ex "target extended-remote /opt/bmp" \
         -ex "monitor swd_scan" -ex "att 1" \
         -ex "set confirm off" \
         -ex "load at_client/merged.hex" -ex "file at_client/zephyr.elf" \
         -ex "start" -ex "quit"

}

iothub_login() {
    az group list -o table > /dev/null
    if [[ "$?" != "0" ]]; then
        az login --tenant $TENANT --scope https://management.core.windows.net//.default
    fi
}

main() {
    if [[ "$1" == "--help" ]]; then
        help
    fi

    local INSTALL_FIRMWARE=1
    if [[ "$1" == "--skip-install" ]]; then
        INSTALL_FIRMWARE=0
        shift
    fi

    MODEM_PORT=$1
    if [[ ! -e $MODEM_PORT ]]; then
        echo "no modem port specified, using default $DEFAULT_MODEM_PORT"
        MODEM_PORT=$DEFAULT_MODEM_PORT
    fi

    source ~/.zephyrtools/venv-python3.13/bin/activate

    if [[ $INSTALL_FIRMWARE == 1 ]]; then
        install_firmware
        echo "Power reset the device and enter to continue"
        read
        sleep 1
    fi

    iothub_login

    configure 

    echo ""
    echo "Azure IoTHub Device Name: $DEVICE_CN"
}

main $*