nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

[ntgabriel]

Could you provide details on how the nrfutil toolchain-manager makes its internet requests?  Or even better, how to add certificates to be trusted in making its TLS connections? 

Our corporate internet security systems are causing the connections to fail, and we need to know what framework is used by the toolchain-manager so that we can work with our IT to resolve or workaround it.  Here are some examples of such workarounds for other development tools (Python PIP, NPM, etc.): https://help.zscaler.com/zia/adding-custom-certificate-application-specific-trust-store 

For context the nrfutil toolchain-manager gives us this error:

C:\>nrfutil toolchain-manager search
Error: Failed to retrieve toolchain-index

Caused by:
0: Failed to download index
1: Get request failed
2: developer.nordicsemi.com/.../index-windows-x86_64.json: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer
3: invalid peer certificate: UnknownIssuer

Or a similar error via the nRF Connect VS Code extension, in that case "Failed to fetch available toolchains." is the high-level error that pops up, but the accompanying console output shows same error as nrfutil toolchain-manager gives on command-line.

I have seen other DevZone posts (such as  "Failed to fetch available toolchains." on Visual Studio Code  ) with same error, but none of them explain how to add trusted certificates so the download works on a network employing SSL/TLS inspection.

Again this is not an issue with the nrfutil toolchain-manager or nRF Connect VS Code extension itself.  But do need to know more about how it works to enable us to get it to work as intended in our corporate environment.

[dejans]

Hi,

Do you use WSL or virtual machine? If so, this discussion might be relevant.

Best regards,
Dejan

[ntgabriel]

Thanks for the response.  No, this is directly in Windows.  We need to find a way to add our certificates into the network call(s) that nrfutil toolchain-manager is making.

[dejans]

Hi,

Could you try to use .pem file with your certificate chain?

Best regards,
Dejan

[ntgabriel]

Yes, I could try that, I do have a .pem file ready to try that has the needed custom certificates in it.  How do I tell nrfutil toolchain-manager about these extra certificates?

Just as one example, if it were Python pip or requests making the network call, we can do this certificate modification via a Windows environment variable `REQUESTS_CA_BUNDLE` that gives a path to the custom .pem file.

[dejans]

Hi,

If the SSL_CERT_FILE environment variable is set, certificates (in PEM format) are read from that file.
You could also use this file which has crt extension, but it is in PEM format.

Best regards,
Dejan

[ntgabriel]

Thanks for the suggestion, I did try to set that environment variable to point to a .pem file, but unfortunately it doesn't change the outcome of a 'nrfutil toolchain-manager search' attempt.

I tried a few different combinations for the .pem file: pointing to our existing custom certificate .pem bundle as we use with some other developer tools, or by appending those two GoDaddy G2 certificates from the provided link into our custom .pem file (note 1 of the 2 GoDaddy certs was already in there anyway).  For good measure I tried also pointing it to a simple .pem file with only those 2 GoDaddy certs in it even though I figured that wouldn't work.

I turned on the `--log-level trace` to try to see more details of what is going wrong.  Not immediately clear, but in all cases it does report the "Server cert" right before it fails, each time giving it as a list of what looks like 3 sets of certificates (always the same).  Each "Certificate()" seems to be a mix of hex-encoded bytes and some human readable where I see references to my company's domain, to a domain of an internet-security software we use, and to nordicsemi.com.  I'll see if I can compare those certificates to what is in our .pem file and see what might be missing...

Since 'SSL_CERT_FILE' is involved, is nrfutil toolchain-manager using OpenSSL?  If so, is there a way to ask it to use the Windows secure channel library instead?  For example, like Git can be set to use 'schannel' on Windows instead of 'openssl' (https://stackoverflow.com/questions/62456484/whats-the-difference-between-openssl-and-the-native-windows-secure-channel-libr)

[rgrr2]

Don't want to hijack this, but I'm stuck at almost the same place.  If one can trust the trace output of nrfutil, then the SSL_CERT_FILE environment variable is never accessed (NRFUTIL_REGISTRY_INDEX, NRFUTIL_SOURCE, NRFUTIL_HTTP, NRFUTIL_NET are used).

Perhaps modifiers using git-cli could be of any help?  See `nrfutil --help-extended`.  But it seems to me, that modifier handling is quite complicated because one has to clone https://github.com/nordicsemiconductor/nrfutil-package-index and then call nrfutil from the created directory.

[dejans]

Hi ntgabriel ,

The nrf toolchain manager does not use OpenSSL.
Unfortunately, there might not be a way to replace underlying libraries.

Best regards,
Dejan

[ntgabriel]

Closing this loop somewhat, as a test I did try to use the 3 certificates reported in the nrfutil toolchain-manager search trace log in a .pem file pointed to by `SSL_CERT_FILE` environment variable, since they do appear to be different certs than the "usual" ones we use for other dev tools and our internet security software.  Unfortunately it didn't change the behavior. 

And since those 3 certs do seem to be missing my company's root certificate from the chain, I also tried a variation where I appended that root certificate to the .pem file, also not effective.

FWIW I'm a Python user, so I found the Python ssl module useful in this, especially to convert the DER-encoded certificates in the nrfutil log file into ASCII/Base64 PEM certs.  https://docs.python.org/3/library/ssl.html#certificate-handling 

ssl.DER_cert_to_PEM_cert()

I also tried one other variation on this theme, using the certificate chain I can export from my web browser and combine into a .pem file, and pointed to that series of certs with SSL_CERT_FILE instead.  None of this changes the behavior either.

So it is still unclear to me whether `SSL_CERT_FILE` is doing anything or not (as rgrr2 is suggesting), and/or if I still haven't gotten the certs right.  The trace logs I get don't seem to indicate that.

A related observation I don't know what to do with is it seems like I have three different ways (3 different sets of certs) to attempt the same thing:

1. I have a series of corporate certificates that work for other developer tools (root, CA, intermediate). 
2. Or the series of (similar but different) corporate certificates that the nrfutil trace log reports to encounter before failing.
3. Or the series of (again similar yet different) corporate certificates that the web browser reports. 

Each of these fit the expected pattern of the internet security software we have in place (a corporate root cert, a corporate CA cert, a corporate intermediate cert), but it is puzzling to me that none of the certificates at each level seem to be equivalent when I would otherwise say I am trying to do the same thing in 3 different ways.  Same type of cert (issuer/subject names), but apparently not literally same cert.  This could be my lack of understanding of DER-PEM conversion, or it could make sense given multiple ways to attempt to make a TLS connection, but it at least complicates my testing as it gives me more combinations of custom certs to try with a given approach.

[ntgabriel]

Thanks for the input.  Could you say more about how you get or are interpreting the trace output of nrfutil you are referring to?  I'm not seeing any reference to any environment variable in the trace logs I'm getting.

I'm doing this (WIndows cmd):

nrfutil toolchain-manager search --log-level trace


I get notable extra info at trace level, mostly comprising a few TLS message payloads being sent to the server, but doesn't seem to say anything about local certificates or reference the names or contents of environment variables AFAIK.  I'd say the only thing useful to me in this trace log so far is a line near the end which gives me a list of 3 DER-encoded server certificates, right before it fails.

[2024-02-07T17:17:38.621Z] [nrfutil-toolchain-manager] TRACE - Server cert is [Certificate(b"..."), Certificate(b"..."), Certificate(b"...")]