nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

[ntgabriel]

Could you provide details on how the nrfutil toolchain-manager makes its internet requests?  Or even better, how to add certificates to be trusted in making its TLS connections? 

Our corporate internet security systems are causing the connections to fail, and we need to know what framework is used by the toolchain-manager so that we can work with our IT to resolve or workaround it.  Here are some examples of such workarounds for other development tools (Python PIP, NPM, etc.): https://help.zscaler.com/zia/adding-custom-certificate-application-specific-trust-store 

For context the nrfutil toolchain-manager gives us this error:

C:\>nrfutil toolchain-manager search
Error: Failed to retrieve toolchain-index

Caused by:
0: Failed to download index
1: Get request failed
2: developer.nordicsemi.com/.../index-windows-x86_64.json: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer
3: invalid peer certificate: UnknownIssuer

Or a similar error via the nRF Connect VS Code extension, in that case "Failed to fetch available toolchains." is the high-level error that pops up, but the accompanying console output shows same error as nrfutil toolchain-manager gives on command-line.

I have seen other DevZone posts (such as  "Failed to fetch available toolchains." on Visual Studio Code  ) with same error, but none of them explain how to add trusted certificates so the download works on a network employing SSL/TLS inspection.

Again this is not an issue with the nrfutil toolchain-manager or nRF Connect VS Code extension itself.  But do need to know more about how it works to enable us to get it to work as intended in our corporate environment.

[dejans]

Hi,

Do you use WSL or virtual machine? If so, this discussion might be relevant.

Best regards,
Dejan

[ntgabriel]

Thanks for the response.  No, this is directly in Windows.  We need to find a way to add our certificates into the network call(s) that nrfutil toolchain-manager is making.

[dejans]

Hi,

Could you try to use .pem file with your certificate chain?

Best regards,
Dejan

[ntgabriel]

Yes, I could try that, I do have a .pem file ready to try that has the needed custom certificates in it.  How do I tell nrfutil toolchain-manager about these extra certificates?

Just as one example, if it were Python pip or requests making the network call, we can do this certificate modification via a Windows environment variable `REQUESTS_CA_BUNDLE` that gives a path to the custom .pem file.

[dejans]

Hi,

If the SSL_CERT_FILE environment variable is set, certificates (in PEM format) are read from that file.
You could also use this file which has crt extension, but it is in PEM format.

Best regards,
Dejan

[ntgabriel]

Thanks for the suggestion, I did try to set that environment variable to point to a .pem file, but unfortunately it doesn't change the outcome of a 'nrfutil toolchain-manager search' attempt.

I tried a few different combinations for the .pem file: pointing to our existing custom certificate .pem bundle as we use with some other developer tools, or by appending those two GoDaddy G2 certificates from the provided link into our custom .pem file (note 1 of the 2 GoDaddy certs was already in there anyway).  For good measure I tried also pointing it to a simple .pem file with only those 2 GoDaddy certs in it even though I figured that wouldn't work.

I turned on the `--log-level trace` to try to see more details of what is going wrong.  Not immediately clear, but in all cases it does report the "Server cert" right before it fails, each time giving it as a list of what looks like 3 sets of certificates (always the same).  Each "Certificate()" seems to be a mix of hex-encoded bytes and some human readable where I see references to my company's domain, to a domain of an internet-security software we use, and to nordicsemi.com.  I'll see if I can compare those certificates to what is in our .pem file and see what might be missing...

Since 'SSL_CERT_FILE' is involved, is nrfutil toolchain-manager using OpenSSL?  If so, is there a way to ask it to use the Windows secure channel library instead?  For example, like Git can be set to use 'schannel' on Windows instead of 'openssl' (https://stackoverflow.com/questions/62456484/whats-the-difference-between-openssl-and-the-native-windows-secure-channel-libr)

[rgrr2]

Don't want to hijack this, but I'm stuck at almost the same place.  If one can trust the trace output of nrfutil, then the SSL_CERT_FILE environment variable is never accessed (NRFUTIL_REGISTRY_INDEX, NRFUTIL_SOURCE, NRFUTIL_HTTP, NRFUTIL_NET are used).

Perhaps modifiers using git-cli could be of any help?  See `nrfutil --help-extended`.  But it seems to me, that modifier handling is quite complicated because one has to clone https://github.com/nordicsemiconductor/nrfutil-package-index and then call nrfutil from the created directory.

[dejans]

Hi ntgabriel ,

The nrf toolchain manager does not use OpenSSL.
Unfortunately, there might not be a way to replace underlying libraries.

Best regards,
Dejan

[ntgabriel]

Thanks for the input.  Could you say more about how you get or are interpreting the trace output of nrfutil you are referring to?  I'm not seeing any reference to any environment variable in the trace logs I'm getting.

I'm doing this (WIndows cmd):

nrfutil toolchain-manager search --log-level trace


I get notable extra info at trace level, mostly comprising a few TLS message payloads being sent to the server, but doesn't seem to say anything about local certificates or reference the names or contents of environment variables AFAIK.  I'd say the only thing useful to me in this trace log so far is a line near the end which gives me a list of 3 DER-encoded server certificates, right before it fails.

[2024-02-07T17:17:38.621Z] [nrfutil-toolchain-manager] TRACE - Server cert is [Certificate(b"..."), Certificate(b"..."), Certificate(b"...")]

[dejans]

Hi,

I have asked internally. You can find exe file inside attached 3755.nrfutil-toolchain-manager.zip which you could try out. The file should work out-of-the-box. Note that the file is provided as is, for testing purpose, and without any guarantee to be working. The file has not been tested with proxy, and thus might not work with proxy. However, it seems that the file picks up on the SSL_CERT_FILE environment variable. 

Best regards,
Dejan

[rgrr2]

I can confirm, that (for me) this solves the certificate issues under Windows.  Any chance for a Linux version which also obeys proxy rules?

PS: but I'm just the "hijacker"

[ntgabriel]

dejans Indeed, yes that custom nrfutil-toolchain-manager.exe works if I point the SSL_CERT_FILE to the certificate file I use for other dev tools, thank you!

At first I called it directly, for initial testing, then I also temporarily replaced the ~\.nrfutil\bin\nrfutil-toolchain-manager.exe with it for now too, to try this within nrfutil calls, and that works now too.

What's your plan for incorporating this change into the nrfutil / toolchain-manager release? 

Or even better perhaps, to incorporate a change to allow use of the OS-native certificate store?  (The certs I'm passing into this custom exe via SSL_CERT_FILE are in the OS cert store already.)