NCS: CMake unable to execute Python

[timonsku]

When I try to build any application with west I always get a CMake error complaining that it is unable to run the Python version included with the NCS toolchain.
The error log:
CMake Error at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Interpreter) (Required is at least version
  "3.8")

      Reason given by package:
          Interpreter: Cannot run the interpreter "/home/timon/ncs/toolchains/7795df4459/usr/local/bin/python3"

I do not have any issues building application with mainline Zephyr targeting other SoCs. This is unique to NCS.
I followed the instructions for command line use over at https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/installation/install_ncs.html

The python installation does exist at the mentioned path but if you run it prints "error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory"
So something seems very broken here.
I source the Zephyr environment via source ncs/zephyr/zephyr-env.sh before trying to build with west as I build from the command line.
Building using the shell launched by nrfutil toolchain-manager launch --shell does work but working from that shell is absolutely not an option. I would like to work in my own shell environment as I do with mainline Zephyr.
This feels like the environment loaded with zephyr-env.sh is incomplete.

NCS version: 2.5.0
OS: Ubuntu 22.04

[Burt]

I have been investigating this problem in a specific environment, using nrfutil, and I think I can give a pretty good explanation, although I may not have all the detailed answers.

I have installed nrfutil, its toolchain-manager, and toolchains v2.5.0 and v2.7.0. I also installed the corresponding SDKs using the nRF Connect for Desktop toolchain-manager (actually, that's how I installed both the SDK and toolchain, as they are installed in one step). Presumably I installed v2.5.0 and then v2.7.0 at a later date.

Now look at this command and it's output:

burt@T420s:~$ nrfutil toolchain-manager list
Version Toolchain
  v2.5.0 /home/burt/ncs/toolchains/7795df4459
* v2.7.0 /home/burt/ncs/toolchains/e9dba88316

Notice the asterisk is at v2.7.0.

Suppose I try this build

burt@T420s:~$ cd ~/ncs/v2.5.0/nrf/applications/asset_tracker_v2/
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ rm -rf build
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ west build -b nrf9160dk_nrf9160_ns

I could have added -p (pristine) to the build command rather than removing the build directory; either way is fine.

The build succeeds. What most of us won't notice is that the build is being done with the v2.7.0 toolchain; the latest one I have installed. Depending on your philosophy, using the latest toolchain is a good thing, or you will believe that the v2.5.0 toolchain should be used for building v2.5.0 SDK code.

One thing I will mention here is that the v2.5.0 toolchain has python 3.8 included in it, whereas the v2.7.0 toolchain has python 3.9 included.

I probably had been building this way for a while when one day I listed the toolchains and noticed the asterisk at v2.7.0. Devzone pointed out that I could change this using

burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager install --ncs-version v2.5.0
[00:00:00] ###### 100% [Install toolchain] Toolchain with version v2.5.0 already installed
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager list
  Version  Toolchain
* v2.5.0   /home/burt/ncs/toolchains/7795df4459
  v2.7.0   /home/burt/ncs/toolchains/e9dba88316

You see that the asterisk was moved to v2.5.0. You can also look at some environment variables

burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager env
GIT_EXEC_PATH            : /home/burt/ncs/toolchains/7795df4459/usr/local/libexec/git-core
GIT_TEMPLATE_DIR         : /home/burt/ncs/toolchains/7795df4459/usr/local/share/git-core/templates
LD_LIBRARY_PATH          : /home/burt/ncs/toolchains/7795df4459/usr/lib:/home/burt/ncs/toolchains/7795df4459/usr/lib/x86_64-linux-gnu:/home/burt/ncs/toolchains/7795df4459/usr/local/lib:/home/burt/.nrfutil/lib/nrfutil-toolchain-manager
PATH                     : /home/burt/ncs/toolchains/7795df4459/usr/bin:/home/burt/ncs/toolchains/7795df4459/usr/bin:/home/burt/ncs/toolchains/7795df4459/usr/local/bin:/home/burt/ncs/toolchains/7795df4459/opt/bin:/home/burt/ncs/toolchains/7795df4459/opt/nanopb/generator-bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/aarch64-zephyr-elf/bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/x86_64-zephyr-elf/bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/arm-zephyr-eabi/bin:/home/burt/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/burt/.nrfutil/bin
PYTHONHOME               : /home/burt/ncs/toolchains/7795df4459/usr/local
PYTHONPATH               : /home/burt/ncs/toolchains/7795df4459/usr/local/lib/python3.8:/home/burt/ncs/toolchains/7795df4459/usr/local/lib/python3.8/site-packages
ZEPHYR_SDK_INSTALL_DIR   : /home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk
ZEPHYR_TOOLCHAIN_VARIANT : zephyr

You see that 7795df4459 shows up frequently in the environment, including in the two PYTHON related variables.

I added some debug messages to the cmake files (cmake is called by the west wrapper) and I see this:

BMS NcsToolchainConfig.cmake entered...
BMS NCS_TOOLCHAIN_PATH=
BMS NCS_TOOLCHAIN_PATH=/home/burt/ncs/toolchains/e9dba88316
-- Using NCS Toolchain 2.6.20240605.1004412633878 for building. (/home/burt/ncs/toolchains/e9dba88316/cmake)
BMS just set Python3_EXECUTABLE and PYTHON_EXECUTABLE=/home/burt/ncs/toolchains/e9dba88316/usr/local/bin/python3
BMS zephyr python.cmake entered...
BMS python.cmake about to find_package Python3
BMS fn FIND_PACKAGE_HANDLE_STANDARD_ARGS entered _NAME=Python3 _FIRST_ARG=REQUIRED_VARS
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line508
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line 568
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS NAME not found
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line 599, Could NOT find Python3 missing  Interpreter
CMake Error at /home/burt/ncs/toolchains/7795df4459/usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Interpreter) (Required is at least version
  "3.8")

      Reason given by package:
          Interpreter: Cannot run the interpreter "/home/burt/ncs/toolchains/e9dba88316/usr/local/bin/python3"

You see what's happening: part of the build system is intent on using the latest toolchain available, despite the asterisk by v2.5.0. (e9dba88316 is the v2.7.0 toolchain). But part of the build system is looking for stuff in the v2.5.0 toolchain. So the build tries to use a python 3.9 executable with python 3.8 libraries, etc. The cmake find_package code catches the mistake and exits with an error.

To do a build with v2.5.0 and nrfutil, you must do two things:

1) reinstall the v2.5.0 toolchain so the asterisk points to it.
2) add -- -DNCS_TOOLCHAIN_VERSION=NONE to the command line. This is what is done in VS Code.

Step 2 is also useful when using nRF Connect for Desktop's toolchain-manager.