Azure FOTA download fails with errno 111 on one Azure Blob endpoint, while another Blob endpoint works

[Martin Hauser]

Hello Nordic team,

I am reporting an Azure FOTA issue where firmware download works from one Azure Blob endpoint and fails from another.

Setup summary:

• Device: nRF9160
• nRF Connect SDK: 2.7.0
• Modem firmware: 1.3.7
• IoT Hub and Blob storage are managed by another company

Observed behavior:

• Device receives the FOTA job.
• FOTA download fails during connect step for production endpoint.
• Error log:

download_client: Unable to connect, errno 111
fota_download: Download client failed to connect to server
azure_fota: FOTA download failed
azure_iot_hub: AZURE_FOTA_EVT_ERROR

What works vs what fails:

Working case:

• TLS chain observed:
  • DigiCert Global Root G2
  • Microsoft Azure RSA TLS Issuing CA 04
  • *.blob.core.windows.net
• FOTA download succeeds

Failing case:

• TLS chain observed:
  • DigiCert Global Root G2
  • Microsoft TLS RSA Root G2 (cross-signed by DigiCert)
  • Microsoft TLS G2 RSA CA OCSP 02
  • *.blob.core.windows.net
• FOTA download fails with errno 111

What I have already tested:

1. Security tag testing

I have added some debug logs to validate that the correct cert is used for FOTA, and that shows the DigiCert.

2. Certificate/chain testing:

With a suspicion that error was caused by the extra intermediate cert, i tried to load the Microsoft TLS RSA Root G2 to use for FOTA, but this gives the same error. 

4. URL accessibility:

I have confirmed by pasting the URL in my browser, that the firmware is available from the link.

I would like help identifying the root cause and what's needed to fix it.

Thanks.

[Øyvind]

Hello, are you able to provide full application logs and modem trace from the device when failing? 

Kind regards,
Øyvind

[Martin Hauser]

Hi

I am unable to provide a modem trace (not enough ram).

here is a longer redacted Log:

[00:01:29.514,770] <dbg> azure_fota: extract_fw_details: No 'desired' object found, assuming it's root object
[00:01:29.514,831] <dbg> azure_fota: extract_fw_details: Incoming firmware version: 2.0.0
[00:01:29.514,862] <dbg> azure_fota: extract_fw_details: Job ID: 18822030012340789
[00:01:29.515,197] <dbg> azure_fota: state_set: State transition: STATE_INIT --> STATE_DOWNLOADING
[00:01:29.515,350] <inf> azure_fota: Attempting to download firmware (version '2.0.0') from <URL>
[00:01:29.515,411] <dbg> fota_download: set_host_and_file: URI checksums -530769204,-1464671077,0,0

[00:01:29.515,747] <dbg> download_client: set_state: state = 1
[00:01:29.515,808] <inf> download_client: Downloading: <URL PATH> [0]
[00:01:29.516,052] <dbg> azure_fota: create_report: Created FOTA report: {"firmware":{"fwUpdateStatus":"downloading","currentFwVersion":"1.9.99","pendingFwVersion":"2.0.0","jobId":"18822030012340789"}}
[00:01:29.652,099] <dbg> download_client: client_connect: Port not specified, using default: 443
[00:01:29.846,160] <dbg> download_client: host_lookup: Failed to resolve hostname <HOSTNAME> on IPv6
[00:01:29.972,106] <dbg> download_client: client_socket_connect: family: 1, type: 1, proto: 258
[00:01:29.972,351] <wrn> download_client: sec_tag_list[0]: 10
[00:01:29.972,381] <inf> download_client: Setting up TLS credentials, sec tag count 1
[00:01:29.972,503] <inf> download_client: Connecting to <IP>
[00:01:29.972,534] <dbg> download_client: client_socket_connect: fd 4, addrlen 8, fam IPv4, port 443
[00:01:30.380,310] <err> download_client: Unable to connect, errno 111
[00:01:30.381,774] <err> fota_download: Download client failed to connect to server
[00:01:30.381,805] <dbg> download_client: set_state: state = 4
[00:01:30.381,835] <dbg> download_client: set_state: state = 0
[00:01:30.381,835] <err> azure_fota: FOTA download failed
[00:01:30.381,896] <dbg> azure_fota: state_set: State transition: STATE_DOWNLOADING --> STATE_INIT
[00:01:30.382,110] <dbg> azure_fota: create_report: Created FOTA report: {"firmware":{"fwUpdateStatus":"error","currentFwVersion":"1.9.99","pendingFwVersion":"2.0.0","jobId":"18822030012340789"}}
[00:01:30.382,110] <err> azure_iot_hub: AZURE_FOTA_EVT_ERROR
[00:01:30.514,038] <err> CloudHdlr: AZURE_IOT_HUB_EVT_FOTA_ERROR: FOTA failed

[Martin Hauser]

I have found an old post where Heidi states: The chain file for certificate chaining must be less than 8kB nrf9160: Adding intermediate certificates to the TLS engine  

In the release notes for the modem FW of the nrf9160 it states: Maximum server certificate chain size has a limit of 4kB.

I have tries to validate the length of the Cert chains presented by the two different blob storage servers by the following command:

$ openssl s_client -connect "<HOST>.blob.core.windows.net:443" -servername "<HOST>.blob.core.windows.net" -showcerts </dev/null 2>/dev/null > chain_from_server.pem

$ wc -c chain_from_server.pem

this gives:
11642 for the one that does NOT work
8287 for the one that does work

Do you think the update fails due to this limitation, and if so is there a way around it?
 

[Øyvind]

Thanks for sharing. The modem trace would allow us to actually see what is failing, but the most likely issue are certificate issues, security tag mismatches, or network/firewall blocking. 

Based on the last questions, yes, the certificate chain size might be issue here. If possible you can do a local capture in wireshark and see the length of the certificate chain. 

Or ou can stream LTE modem trace data directly to Wireshark in real time using nrfutil trace:

nrfutil trace lte --input-serialport <serial_port> --output-wireshark <path/to/wireshark>

Let me know how that works for you.

Kind regards,
Øyvind