nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Tue, 26 Mar 2024 12:31:56 GMT

Hi ntgabriel ,

Thank you for the update.

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Mon, 25 Mar 2024 14:49:53 GMT

Thanks dejans ! Appreciate this being updated in the main toolchain-manager. This does indeed work in conjunction with the SSL_CERT_FILE environment variable to use a custom certificate bundle.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Wed, 06 Mar 2024 14:13:15 GMT

Hi,

We have released new version which contains updates included in previously provided exe/binary (solving problem with native certificates) and fix for proxy networks. It can be installed using nrfutil command "nrfutil install toolchain-manager".

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

jgilles — Mon, 12 Feb 2024 16:37:08 GMT

rgrr2 The new toolchain-manager binary worked for me because I wasn't having proxy-related connections problems (this is a separate issue). I am behind a corporate TLS inspection system that requires installation of company-specific root certs for clients to trust or cert validation fails. The new binary is successfully using the locally installed trusted certs, the released one does not.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

rgrr2 — Mon, 12 Feb 2024 09:35:09 GMT

Unfortunately this version still does not obey to the proxy setup. Don't know how jgilles did it

hardy@debian-hardy:~/.nrfutil/bin$ nrfutil --detect-proxy --log-level=trace --log-output=stdout --json toolchain-manager search
{"type":"log","data":{"level":"INFO","message":"nrfutil (version = 7.7.0, platform = x86_64-unknown-linux-gnu) invoked with --detect-proxy --log-level=trace --log-output=stdout --json toolchain-manager search ","timestamp":"2024-02-12T09:33:21.296Z"}}
{"type":"log","data":{"level":"INFO","message":"Trying to detect proxy configuration automatically...","timestamp":"2024-02-12T09:33:21.296Z"}}
{"type":"log","data":{"level":"INFO","message":"Using proxy helper program from '/home/hardy/.nrfutil/proxy_utils/proxy'","timestamp":"2024-02-12T09:33:21.296Z"}}
{"type":"log","data":{"level":"INFO","message":"Running '/home/hardy/.nrfutil/proxy_utils/proxy developer.nordicsemi.com' to detect proxy configuration","timestamp":"2024-02-12T09:33:21.296Z"}}
{"type":"log","data":{"level":"INFO","message":"Proxy configuration detected","timestamp":"2024-02-12T09:33:21.297Z"}}
{"type":"log","data":{"level":"DEBUG","message":"Setting environment variable NRFUTIL_EXEC_PATH to /home/hardy/.nrfutil/bin/nrfutil for subcommand `toolchain-manager`","timestamp":"2024-02-12T09:33:21.297Z"}}
{"type":"log","data":{"level":"INFO","message":"nrfutil-toolchain-manager (version = 0.14.1, platform = x86_64-unknown-linux-gnu) invoked with --json --log-level TRACE --log-output stdout search ","timestamp":"2024-02-12T09:33:21.300Z"}}
{"type":"log","data":{"level":"DEBUG","message":"Constructing the configuration from the defaults","timestamp":"2024-02-12T09:33:21.300Z"}}
{"type":"log","data":{"level":"DEBUG","message":"Current config: Config { install_dir: \"/home/hardy/ncs\", toolchain_index: None }","timestamp":"2024-02-12T09:33:21.300Z"}}
{"type":"log","data":{"level":"INFO","message":"Downloading from "timestamp":"2024-02-12T09:33:21.300Z"}}">developer.nordicsemi.com/.../index-linux-x86_64.json","timestamp":"2024-02-12T09:33:21.300Z"}}
{"type":"log","data":{"level":"DEBUG","message":"connecting to developer.nordicsemi.com:443 at 104.20.251.111:443","timestamp":"2024-02-12T09:33:21.326Z"}}
{"type":"log","data":{"level":"INFO","message":"Failed to retrieve cached index: Failed to get cache","timestamp":"2024-02-12T09:33:51.331Z"}}
Error: Failed to retrieve toolchain-index

Caused by:
   0: Failed to download index
   1: Get request failed
   2: developer.nordicsemi.com/.../index-linux-x86_64.json: Network Error: timed out reading response
   3: timed out reading response

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Fri, 09 Feb 2024 16:13:31 GMT

dejans Thanks to you and the development team for your help with this!

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

jgilles — Fri, 09 Feb 2024 15:55:41 GMT

This linux binary works for me and doesn't require setting SSL_CERT_FILE since I have added the required certs using update-ca-certificates in Ubuntu. Thank you!

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Fri, 09 Feb 2024 09:05:57 GMT

Hi ntgabriel and rgrr2 ,

[quote user="Hardy"]Any chance for a Linux version which also obeys proxy rules?[/quote]

Please find attached Linux binary file (inside provided zip file) which should work in a similar way as previously provided file for Windows. Please note that this is not yet an official version. The same considerations apply as for the Windows file. You would need to test it and verify that it works for you.

devzone.nordicsemi.com/.../3808.nrfutil_2D00_toolchain_2D00_manager.zip

[quote user="ntgabriel"]What's your plan for incorporating this change into the nrfutil / toolchain-manager release?[/quote]

For a specific plan or timeline regarding the official release with changes, please contact your regional sales manager.

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Thu, 08 Feb 2024 22:20:00 GMT

dejans Indeed, yes that custom nrfutil-toolchain-manager.exe works if I point the SSL_CERT_FILE to the certificate file I use for other dev tools, thank you!

At first I called it directly, for initial testing, then I also temporarily replaced the ~\.nrfutil\bin\nrfutil-toolchain-manager.exe with it for now too, to try this within nrfutil calls, and that works now too.

What's your plan for incorporating this change into the nrfutil / toolchain-manager release?

Or even better perhaps, to incorporate a change to allow use of the OS-native certificate store? (The certs I'm passing into this custom exe via SSL_CERT_FILE are in the OS cert store already.)

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

rgrr2 — Thu, 08 Feb 2024 15:07:06 GMT

I can confirm, that (for me) this solves the certificate issues under Windows. Any chance for a Linux version which also obeys proxy rules?

PS: but I'm just the "hijacker"

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Thu, 08 Feb 2024 13:10:18 GMT

Hi,

I have asked internally. You can find exe file inside attached devzone.nordicsemi.com/.../3755.nrfutil_2D00_toolchain_2D00_manager.zip which you could try out. The file should work out-of-the-box. Note that the file is provided as is, for testing purpose, and without any guarantee to be working. The file has not been tested with proxy, and thus might not work with proxy. However, it seems that the file picks up on the SSL_CERT_FILE environment variable.

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

rgrr2 — Wed, 07 Feb 2024 20:54:07 GMT

You can get one step further by doing the following (at least if "git clone" works in your environment):

mkdir tmp cd tmp git clone github.com/.../nrfutil-package-index cd nrfutil-package-index NRFUTIL_NET_GIT_FETCH_WITH_CLI=true NRFUTIL_PACKAGE_INDEX_URL=github.com/.../nrfutil-package-index nrfutil +myconfig search
nrfutil +myconfig install toolchain-manager

But then "nrfutil +myconfig toolchain-manager search" is the next dead end :-/

To my opinion proxy environment has never been tested for the tools although I guess the majority of developers is sitting behind a proxy.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Wed, 07 Feb 2024 18:15:42 GMT

Thanks for the input. Could you say more about how you get or are interpreting the trace output of nrfutil you are referring to? I'm not seeing any reference to any environment variable in the trace logs I'm getting.

I'm doing this (WIndows cmd):

nrfutil toolchain-manager search --log-level trace

I get notable extra info at trace level, mostly comprising a few TLS message payloads being sent to the server, but doesn't seem to say anything about local certificates or reference the names or contents of environment variables AFAIK. I'd say the only thing useful to me in this trace log so far is a line near the end which gives me a list of 3 DER-encoded server certificates, right before it fails.

[2024-02-07T17:17:38.621Z] [nrfutil-toolchain-manager] TRACE - Server cert is [Certificate(b"..."), Certificate(b"..."), Certificate(b"...")]

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Wed, 07 Feb 2024 18:02:44 GMT

Closing this loop somewhat, as a test I did try to use the 3 certificates reported in the nrfutil toolchain-manager search trace log in a .pem file pointed to by `SSL_CERT_FILE` environment variable, since they do appear to be different certs than the "usual" ones we use for other dev tools and our internet security software. Unfortunately it didn't change the behavior.

And since those 3 certs do seem to be missing my company's root certificate from the chain, I also tried a variation where I appended that root certificate to the .pem file, also not effective.

FWIW I'm a Python user, so I found the Python ssl module useful in this, especially to convert the DER-encoded certificates in the nrfutil log file into ASCII/Base64 PEM certs. https://docs.python.org/3/library/ssl.html#certificate-handling

ssl.DER_cert_to_PEM_cert()

I also tried one other variation on this theme, using the certificate chain I can export from my web browser and combine into a .pem file, and pointed to that series of certs with SSL_CERT_FILE instead. None of this changes the behavior either.

So it is still unclear to me whether `SSL_CERT_FILE` is doing anything or not (as rgrr2 is suggesting), and/or if I still haven't gotten the certs right. The trace logs I get don't seem to indicate that.

A related observation I don't know what to do with is it seems like I have three different ways (3 different sets of certs) to attempt the same thing:

I have a series of corporate certificates that work for other developer tools (root, CA, intermediate).
Or the series of (similar but different) corporate certificates that the nrfutil trace log reports to encounter before failing.
Or the series of (again similar yet different) corporate certificates that the web browser reports.

Each of these fit the expected pattern of the internet security software we have in place (a corporate root cert, a corporate CA cert, a corporate intermediate cert), but it is puzzling to me that none of the certificates at each level seem to be equivalent when I would otherwise say I am trying to do the same thing in 3 different ways. Same type of cert (issuer/subject names), but apparently not literally same cert. This could be my lack of understanding of DER-PEM conversion, or it could make sense given multiple ways to attempt to make a TLS connection, but it at least complicates my testing as it gives me more combinations of custom certs to try with a given approach.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Wed, 07 Feb 2024 11:58:39 GMT

Hi ntgabriel ,

The nrf toolchain manager does not use OpenSSL.
Unfortunately, there might not be a way to replace underlying libraries.

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

rgrr2 — Wed, 07 Feb 2024 09:02:15 GMT

Don't want to hijack this, but I'm stuck at almost the same place. If one can trust the trace output of nrfutil, then the SSL_CERT_FILE environment variable is never accessed (NRFUTIL_REGISTRY_INDEX, NRFUTIL_SOURCE, NRFUTIL_HTTP, NRFUTIL_NET are used).

Perhaps modifiers using git-cli could be of any help? See `nrfutil --help-extended`. But it seems to me, that modifier handling is quite complicated because one has to clone https://github.com/nordicsemiconductor/nrfutil-package-index and then call nrfutil from the created directory.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Tue, 06 Feb 2024 20:41:27 GMT

Thanks for the suggestion, I did try to set that environment variable to point to a .pem file, but unfortunately it doesn't change the outcome of a 'nrfutil toolchain-manager search' attempt.

I tried a few different combinations for the .pem file: pointing to our existing custom certificate .pem bundle as we use with some other developer tools, or by appending those two GoDaddy G2 certificates from the provided link into our custom .pem file (note 1 of the 2 GoDaddy certs was already in there anyway). For good measure I tried also pointing it to a simple .pem file with only those 2 GoDaddy certs in it even though I figured that wouldn't work.

I turned on the `--log-level trace` to try to see more details of what is going wrong. Not immediately clear, but in all cases it does report the "Server cert" right before it fails, each time giving it as a list of what looks like 3 sets of certificates (always the same). Each "Certificate()" seems to be a mix of hex-encoded bytes and some human readable where I see references to my company's domain, to a domain of an internet-security software we use, and to nordicsemi.com. I'll see if I can compare those certificates to what is in our .pem file and see what might be missing...

Since 'SSL_CERT_FILE' is involved, is nrfutil toolchain-manager using OpenSSL? If so, is there a way to ask it to use the Windows secure channel library instead? For example, like Git can be set to use 'schannel' on Windows instead of 'openssl' (https://stackoverflow.com/questions/62456484/whats-the-difference-between-openssl-and-the-native-windows-secure-channel-libr)

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Tue, 06 Feb 2024 12:41:37 GMT

Hi,

If the SSL_CERT_FILE environment variable is set, certificates (in PEM format) are read from that file.
You could also use this file which has crt extension, but it is in PEM format.

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Mon, 05 Feb 2024 18:26:30 GMT

Yes, I could try that, I do have a .pem file ready to try that has the needed custom certificates in it. How do I tell nrfutil toolchain-manager about these extra certificates?

Just as one example, if it were Python pip or requests making the network call, we can do this certificate modification via a Windows environment variable `REQUESTS_CA_BUNDLE` that gives a path to the custom .pem file.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Mon, 05 Feb 2024 11:41:49 GMT

Hi,

Could you try to use .pem file with your certificate chain?

Best regards,
Dejan

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

ntgabriel — Fri, 02 Feb 2024 16:25:14 GMT

Thanks for the response. No, this is directly in Windows. We need to find a way to add our certificates into the network call(s) that nrfutil toolchain-manager is making.

RE: nrfutil toolchain-manager "Error: Failed to retrieve toolchain-index" due to corporate SSL/TLS inspection

dejans — Fri, 02 Feb 2024 14:41:05 GMT

Hi,

Do you use WSL or virtual machine? If so, this discussion might be relevant.

Best regards,
Dejan