Toolchain very slow on Windows machines

RE: Toolchain very slow on Windows machines

Peter Haas — Fri, 31 Jan 2025 07:28:55 GMT

Hi Simon,

thank you for your reply.

I can verify that this indeed circumvents the issue.
My buildchain though required to also set that flag for the mcuboot subimage I also had to set: -Dmcuboot_DTC=False

We don't use sysbuild atm so I cannot verify the latter.

Best regards,
Peter Haas

RE: Toolchain very slow on Windows machines

Simonr — Thu, 30 Jan 2025 14:30:31 GMT

Hi Peter

So, it seems we're out of ways to potentially fix this, and rather have to move towards avoiding the issue instead. Could you try skipping the dtc.exe check altogether and see if that at least unblocks you from these incredibly long build times:

State that DTC is not there by doing -DDTC=False when invoking CMake (or west).

This will do two things:

Tell the system that DTC isn't available and just not perform the checks
Avoid FindDtc.cmake to lookup Dtc.

Please note that this will only work for builds running without sysbuild (introduced in NCS 2.7.0) so you would need to run something like the following on newer SDK versions:

❯ west build --no-sysbuild ../zephyr/samples/hello_world -- -DDTC=False

I hope this will get you around the issue at least.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Peter Haas — Mon, 20 Jan 2025 10:36:17 GMT

Hi Carles,

thank you for your reply.

I just had the chance to try the toolchain by manually installing the version of dtc as you suggested.
Copying all the binaries from chocolatey into their respective location in the ncs toolchain works.

However it seems that doesn't fix our problem.
The build is still taking a very long time and I have to swap networks to shorten that. It does seem to be a bit faster than our initial version (~8 minutes instead of 15m)

Best
Peter

RE: Toolchain very slow on Windows machines

Carles — Wed, 08 Jan 2025 14:05:50 GMT

Hi there!

Any chance you could give the latest package a try?

https://community.chocolatey.org/packages/dtc-msys2/1.6.1

Not sure how you replace the one that is bundled with NCS, but perhaps you could try 1.6.1 separately to see if it helps?

You can use `choco install dtc-msys2` to install this version.

RE: Toolchain very slow on Windows machines

Peter Haas — Wed, 04 Sep 2024 07:21:01 GMT

By stuck I mean just the command line program doesn't return.
No output, no return code.
Seemingly equivalent to just an empty infinite loop :)

Git gc didn't help here.
To reiterate: Our temporal fix is just disconnecting the build machine from the network when building/flashing firmware.

Thanks!
Best,
Peter

RE: Toolchain very slow on Windows machines

Simonr — Wed, 04 Sep 2024 05:49:01 GMT

Hmm, okay. Thank you. By stuck, does the terminal just pop up empty or does nothing happen at all when running dtc.exe?

One thing to try, as a colleague had an issue with slow build times as well, although I don't really think it's related, since I'm pretty sure your issue is firewall related. On their end it helped running git gc in .../<NCS_VERSION>/zephyr

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Peter Haas — Tue, 03 Sep 2024 06:53:59 GMT

Hi Simon,

i just tried running dtc.exe on various network settings (firewalled, non-firewalled, no connection) and terminals (git bash, powershell).
I just executed dtc.exe without any arguments.
All of them seem to be stuck. (>15 min).

No return value yet and it had it running for a while now.

I hope this helps in any way,
Best regards,
Peter

RE: Toolchain very slow on Windows machines

Simonr — Tue, 03 Sep 2024 06:08:44 GMT

I'm still waiting on a reply from the devs on the steps to change the msys version. On your end, can you confirm this is the issue by trying to execute dtc.exe and see how long it takes to load.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Simonr — Mon, 26 Aug 2024 07:52:00 GMT

Hi Alex

Just wanted to let you know that I'm currently looking into how to do this myself, as I assumed it was more straightforward than what it seems to be. I'll get back to you when I figure it out.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Alex-Sony — Thu, 22 Aug 2024 07:16:55 GMT

Hi Simon,

We are not exactly sure how this should be done. Could you please give a us some small instructions on how to proceed to try this?

Thank you.

RE: Toolchain very slow on Windows machines

Simonr — Tue, 20 Aug 2024 13:12:16 GMT

Hovering his name in the bottom of the original ticket should take you to his profile.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

pixbroker — Mon, 19 Aug 2024 19:38:42 GMT

Thank you. I dont have his username.

RE: Toolchain very slow on Windows machines

Alex-Sony — Mon, 19 Aug 2024 14:32:31 GMT

He said to send him a direct message if you want to ask him anything.

RE: Toolchain very slow on Windows machines

Alex-Sony — Mon, 19 Aug 2024 14:32:05 GMT

Thanks but we don't use OpenVPN

RE: Toolchain very slow on Windows machines

bicanoff — Mon, 19 Aug 2024 12:34:08 GMT

I encountered a similar issue. One day, building my project took approximately 30 minutes. I had to close the OpenVPN client, and now it builds as quickly as before. I hope this helps someone.

RE: Toolchain very slow on Windows machines

pixbroker — Sun, 18 Aug 2024 19:48:26 GMT

HiAlex-Sony , im also trying to use CLion with the nrf-sdk and I would love to connect with your colleague Peter.

RE: Toolchain very slow on Windows machines

Simonr — Thu, 15 Aug 2024 11:58:16 GMT

Hi Alex

So, after some digging, it seems like this might be caused by msys-2.0.dll used by the dtc.exe (devicetree compiler) making lots of network requests and attempting to acess LDAP. Changing the version to msys1 might solve the build time issue entirely. Can you give that a try first?

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Alex-Sony — Wed, 14 Aug 2024 07:02:05 GMT

Hi Simon,

We have tried temporally fully disabling our antivirus with no success. So we will need the hostnames/IPs/ports that we need to whitelist in the firewall to be able to build as fast as outside of our network.

Best regards,

Alex

RE: Toolchain very slow on Windows machines

Simonr — Mon, 12 Aug 2024 13:50:42 GMT

Hi Alex

You can begin by whitelisting the source folders and SDK toolchain install locations in your antivirus software, and whitelist the ones I mentioned already, as well as west.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Alex-Sony — Fri, 09 Aug 2024 07:08:19 GMT

Hi Simon,

I will update the nRF VS Code extension to the pre-release version.

Still, we don't think this problem comes from VS Code. My colleague Peter who started this thread, uses CLion with CMake to build the project (Always worked fast before like my setup with VS Code) and we started to get the slow down builds at the same time so we both faced the same issue with and without VS Code so we are pretty sure the build issue doesn't come from VS Code or the extension.

If you can tell us what would need to be whitelisted would be great but still we think that this is probably a timeout that should be updated or an option to disable the wait since we are not sure if IT team here at Sony and probably in other companies would allow to change the firewall configuration.

In any case, we are grateful for your help.

Best regards,

Alex

RE: Toolchain very slow on Windows machines

Simonr — Thu, 08 Aug 2024 07:33:59 GMT

Hi Alex

Okay, glad to hear we found the root of the issue at least. I think in order to better the performance on the internal network the firewall will need to whitelist some of the resources VS Code is pulling from. I'll get back to you on what exactly these are. I believe you can check the Online Services in VS code to see what is enabled to fetch straight from URLs. I think JSON, Git and CMake at least are some of these.

Also, we see that you're using the stable version of the VS code extension. In NCS v2.7.0 you need to use the pre-release version of the nRF Connect VS Code extension.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Alex-Sony — Thu, 08 Aug 2024 05:33:12 GMT

Hi Simon,

Good news, your shot in the dark hit the bullseye. It seems to be a firewall issue, when I remove the connection to our local network or connect through my phone's hotspot it builds as fast as it is supposed to.

We assume that since we can build without being connected, the firewall shouldn't be a problem.

Can you explain to us why is this a problem? Is there something we can do to avoid having to disconnect from our local network to build as fast as possible?

Here you have the log you requested

{

  "platform": {

    "os": "win32",

    "osVersion": "Windows 10 Enterprise",

    "osKernel": "10.0.19045",

    "vscode": "1.92.0",

    "electron": "30.1.2",

    "node": "v20.14.0"

  },

  "system": {

    "date": "2024-08-08T05:23:14.842Z",

    "vscodeRoot": "c:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Microsoft VS Code\\resources\\app",

    "nrfConnectForDesktopInstalled": true,

    "vscodeUptime": "00:13:21",

    "osUptime": "00:22:21",

    "cpu": "13th Gen Intel(R) Core(TM) i7-1370P"

  },

  "workspace": {

    "name": "blinky_3",

    "workspaceFile": null,

    "folders": [

      "c:\\temp\\blinky_3"

    ]

  },

  "sdks": [

    {

      "version": "v2.7.0",

      "path": "c:\\ncs\\v2.7.0"

    }

  ],

  "activeBuild": "c:\\temp\\blinky_3\\build",

  "apps": [

    {

      "workspace": "c:\\temp\\blinky_3",

      "uri": "c:\\temp\\blinky_3",

      "buildConfigurations": [

        {

          "id": "c:\\temp\\blinky_3\\build",

          "name": "build",

          "boardId": "nrf5340dk_nrf5340_cpuapp",

          "type": "Zephyr",

          "isChild": false,

          "isStale": false,

          "taskBindings": {

            "build": [],

            "pristineBuild": [],

            "flash": [],

            "eraseAndFlash": []

          }

        }

      ]

    }

  ],

  "topdir": "c:\\ncs\\v2.7.0",

  "workspaceState": "freestanding-ready",

  "toolchains": [

    {

      "version": "2.7.0",

      "path": "c:\\ncs\\toolchains\\ce3b5ff664"

    }

  ],

  "connectedDevices": [],

  "deviceProvider": "nrfutil",

  "tools": {

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\arm-zephyr-eabi\\bin\\arm-zephyr-eabi-gcc.exe": "0.16.5-1",

    "C:\\Program Files\\SEGGER\\JLink_V794e\\JLink.exe": "7.94e",

    "C:\\Program Files\\Nordic Semiconductor\\nrf-command-line-tools\\bin\\nrfjprog.exe": null,

    "nrfutil": "Not found",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\cmake.exe": "3.21.0",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\Scripts\\west.exe": "1.2.0",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\python.exe": "3.9.13",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\ninja.exe": "1.10.2",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\gperf.exe": "3.1",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\dtc.exe": "1.4.7",

    "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\gn.exe": "2172",

    "C:\\ncs\\toolchains\\ce3b5ff664\\mingw64\\bin\\git.exe": "2.37.3.windows.1",

    "gdbPath": "c:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\arm-zephyr-eabi\\bin\\arm-zephyr-eabi-gdb.exe"

  },

  "nrfutil": {

    "nrfutil-device": {

      "version": "2.0.0",

      "binPath": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-connect-2024.7.13-win32-x64\\platform\\nrfutil\\bin\\nrfutil-device"

    },

    "nrfutil-toolchain-manager": {

      "version": "0.14.4",

      "binPath": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-connect-2024.7.13-win32-x64\\platform\\nrfutil\\bin\\nrfutil-toolchain-manager"

    }

  },

  "environment": {

    "westEnv": {

      "ELECTRON_RUN_AS_NODE": "1",

      "HOME": "C:\\Users\\CHMendeA",

      "HOMEDRIVE": "C:",

      "HOMEPATH": "\\Users\\CHMendeA",

      "PATH": "C:\\ncs\\toolchains\\ce3b5ff664;C:\\ncs\\toolchains\\ce3b5ff664\\mingw64\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\Scripts;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\nanopb\\generator-bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\aarch64-zephyr-elf\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\x86_64-zephyr-elf\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\arm-zephyr-eabi\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\riscv64-zephyr-elf\\bin;c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-connect-2024.7.13-win32-x64\\platform\\nrfutil\\lib\\nrfutil-toolchain-manager;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program
 Files\\WindowsApps\\Microsoft.DesktopAppInstaller_1.22.10661.0_x64__8wekyb3d8bbwe\";C:\\Program Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;C:\\Program Files\\Nordic Semiconductor\\nrf-command-line-tools\\bin\\;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Python\\Launcher\\;C:\\Users\\CHMendeA\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Microsoft
 VS Code\\bin;C:\\Program Files\\SEGGER\\JLink;;C:\\Users\\CHMendeA\\AppData\\Roaming\\Programs\\Zero Install",

      "ZEPHYR_BASE": "c:\\ncs\\v2.7.0\\zephyr",

      "ZEPHYR_TOOLCHAIN_VARIANT": "zephyr",

      "ZEPHYR_SDK_INSTALL_DIR": "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk",

      "PYTHONPATH": "C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\Lib;C:\\ncs\\toolchains\\ce3b5ff664\\opt\\bin\\Lib\\site-packages"

    },

    "inherited": {

      "ELECTRON_RUN_AS_NODE": "1",

      "HOME": "C:\\Users\\CHMendeA",

      "HOMEDRIVE": "C:",

      "HOMEPATH": "\\Users\\CHMendeA",

      "Path": "C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\WindowsApps\\Microsoft.DesktopAppInstaller_1.22.10661.0_x64__8wekyb3d8bbwe\";C:\\Program
 Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;C:\\Program Files\\Nordic Semiconductor\\nrf-command-line-tools\\bin\\;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Python\\Launcher\\;C:\\Users\\CHMendeA\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Microsoft
 VS Code\\bin;C:\\Program Files\\SEGGER\\JLink;;C:\\Users\\CHMendeA\\AppData\\Roaming\\Programs\\Zero Install",

      "PATH": "C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\WindowsApps\\Microsoft.DesktopAppInstaller_1.22.10661.0_x64__8wekyb3d8bbwe\";C:\\Program
 Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;C:\\Program Files\\Nordic Semiconductor\\nrf-command-line-tools\\bin\\;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Python\\Launcher\\;C:\\Users\\CHMendeA\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\CHMendeA\\AppData\\Local\\Programs\\Microsoft
 VS Code\\bin;C:\\Program Files\\SEGGER\\JLink;;C:\\Users\\CHMendeA\\AppData\\Roaming\\Programs\\Zero Install"

    },

    "toolchainPath": "c:\\ncs\\toolchains\\ce3b5ff664",

    "toolchainBinPath": "c:\\ncs\\toolchains\\ce3b5ff664\\opt\\zephyr-sdk\\arm-zephyr-eabi\\bin",

    "toolchainVersion": "2.7.0"

  },

  "terminal": {

    "defaultProfile": null

  },

  "config": {

    "nordic-semiconductor.nrf-connect": {

      "topdir": "C:\\ncs\\v2.6.1",

      "toolchain": {

        "path": "C:\\ncs\\toolchains"

      },

      "ozonePath": "",

      "applications": [],

      "applicationOptions": {},

      "kconfig": {

        "interface": "kconfig"

      },

      "west": {

        "env": {

          "$base": "terminal"

        }

      },

      "boardRoots": [],

      "taskBindings": {},

      "buildTerminal": {

        "condensedProgress": true

      },

      "debugging": {

        "flash": true,

        "bindings": {},

        "justMyCode": true

      },

      "activeAppFollowActiveEditor": true,

      "deviceProvider": "",

      "flash": {

        "softreset": false,

        "erase": false,

        "recover": false

      },

      "enableTelemetry": true,

      "thirdpartyIntegration": {},

      "toolchainManager": {

        "indexURL": null,

        "installDirectory": null

      },

      "nrfutil": {

        "home": null

      },

      "defaultOpenAction": "ask"

    },

    "nordic-semiconductor.nrf-terminal": {

      "terminalMode": "character"

    }

  },

  "extensions": {

    "internal": {

      "nordic-semiconductor.nrf-connect": {

        "version": "2024.7.13",

        "path": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-connect-2024.7.13-win32-x64",

        "isActive": true

      },

      "nordic-semiconductor.nrf-terminal": {

        "version": "2024.3.15",

        "path": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-terminal-2024.3.15",

        "isActive": true

      },

      "nordic-semiconductor.nrf-devicetree": {

        "version": "2024.6.20",

        "path": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-devicetree-2024.6.20",

        "isActive": true

      },

      "nordic-semiconductor.nrf-kconfig": {

        "version": "2024.6.20",

        "path": "c:\\Users\\CHMendeA\\.vscode\\extensions\\nordic-semiconductor.nrf-kconfig-2024.6.20",

        "isActive": true

      }

    },

    "external": {

      "marus25.cortex-debug": null,

      "ms-vscode.cpptools": "1.21.5",

      "ms-vscode.js-debug": "1.92.0",

      "ms-vscode.js-debug-companion": "1.1.3",

      "ms-vscode.vscode-js-profile-table": "1.0.9",

      "ms-python.debugpy": "2024.10.0",

      "ms-python.python": "2024.12.1",

      "ms-python.vscode-pylance": "2024.7.1",

      "ms-vscode.cmake-tools": "1.18.44",

      "ms-vscode.cpptools-extension-pack": "1.3.0",

      "ms-vscode.cpptools-themes": "2.0.0",

      "njqdev.vscode-python-typehint": "1.5.1",

      "oderwat.indent-rainbow": "8.3.1",

      "seanwu.vscode-qt-for-python": "7.6.0",

      "trond-snekvik.gnu-mapfiles": "1.1.0",

      "twxs.cmake": "0.0.17",

      "VisualStudioExptTeam.intellicode-api-usage-examples": "0.2.8",

      "VisualStudioExptTeam.vscodeintellicode": "1.3.1"

    }

  }

}

Thank you very much.

Best regards,

Alex

RE: Toolchain very slow on Windows machines

Simonr — Wed, 07 Aug 2024 11:13:32 GMT

Can you also provide the support log for us?

Here are the steps to generate support information:

Open the nRF Connect for VS Code extension.
Navigate to the Welcome View in the nRF Connect Sidebar Views.
Find and click on the "Generate Support Information" action.

This will generate a log in the Output Panel View.

Next, some suggestions from our devs:

On Windows, try disabling AV, or whitelist source folders and the SDK/toolchain install locations from AV.
We've seen some bad performance from west calls due to git garbage collection, but that can be helped by doing an aggressive prune of the git.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Simonr — Wed, 07 Aug 2024 09:03:59 GMT

Hi Alex

No worries. I have forwarded this to our VS code team and we'll take a look. The things I can think of that you can try: Can you try building the sample within the NCS instead of in your /temp folder, just to see if that makes a difference. Could this be firewall related and only happen on devices on the internal network on your end perhaps? This is just a shot in the dark though. I'll get back to you when I hear from the devs on this.

Best regards,

Simon

RE: Toolchain very slow on Windows machines

Alex-Sony — Mon, 05 Aug 2024 15:26:35 GMT

Hi Simon,

Sorry for the delay, we were on holidays.

I just tried in another computer to install the newest NCS version (2.7.0) and still can take 10 minutes to build the blinky sample.

Most of the time is spend on this line:

Including generated dts.cmake file: C:/temp/blinky/build/zephyr/dts.cmake

Here is the full log from the terminal. The build is fast when there is no change to the DTS or proj.conf. If it is strictly just compiling c code that has changed, it is fast. I'm using among others an AMD 7950x3d with 64GB of RAM so the PC performance is not the bottle neck.

Here is the log: I could not paste it here because it would throw an error when posting: https://justpaste.it/8abzg

Thank you.

Alex