NCS: CMake unable to execute Python

[timonsku]

When I try to build any application with west I always get a CMake error complaining that it is unable to run the Python version included with the NCS toolchain.
The error log:
CMake Error at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Interpreter) (Required is at least version
  "3.8")

      Reason given by package:
          Interpreter: Cannot run the interpreter "/home/timon/ncs/toolchains/7795df4459/usr/local/bin/python3"

I do not have any issues building application with mainline Zephyr targeting other SoCs. This is unique to NCS.
I followed the instructions for command line use over at https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/installation/install_ncs.html

The python installation does exist at the mentioned path but if you run it prints "error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory"
So something seems very broken here.
I source the Zephyr environment via source ncs/zephyr/zephyr-env.sh before trying to build with west as I build from the command line.
Building using the shell launched by nrfutil toolchain-manager launch --shell does work but working from that shell is absolutely not an option. I would like to work in my own shell environment as I do with mainline Zephyr.
This feels like the environment loaded with zephyr-env.sh is incomplete.

NCS version: 2.5.0
OS: Ubuntu 22.04

[Jørgen Holmefjord]

Hi,

I tried to follow the same instructions on Ubuntu 22.04, but I'm not able to reproduce the behavior you are seeing.

Can you check that shared library is available? On my setup I see the following:

bash-5.1$ find /* -name "libpython3.8.so.1.0" -print 2>/dev/null
/home/joh2/ncs/toolchains/7795df4459/opt/nanopb/generator-bin/libpython3.8.so.1.0
/home/joh2/ncs/toolchains/7795df4459/usr/local/lib/libpython3.8.so.1.0

If you do not see this file, please try to reinstall the toolchain.

Best regards,
Jørgen

[timonsku]

That library is available at that location.
I assume the environment isn't loading correctly.
Could you tell me what steps you take from a terminal session to compiling one of the samples?

[JRPC]

I can reproduce the exact same issue. 

If I use west -build directly from my console, I get the following output:

-- west build: generating a build system
Loading Zephyr default modules (Zephyr base (cached)).
-- Application: /nrf/applications/Riverrock
-- CMake version: 3.22.2
-- Using NCS Toolchain 2.4.0 for building. (/root/ncs/toolchains/1f9b40e71a/cmake)
CMake Error at /usr/local/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Python3 (missing: Interpreter) (Required is exact version
"3.8")

Reason given by package:
Interpreter: Cannot run the interpreter "/root/ncs/toolchains/1f9b40e71a/usr/local/bin/python3"

Call Stack (most recent call first):
/usr/local/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
/usr/local/share/cmake-3.22/Modules/FindPython/Support.cmake:3166 (find_package_handle_standard_args)
/usr/local/share/cmake-3.22/Modules/FindPython3.cmake:490 (include)
/zephyr/cmake/modules/python.cmake:46 (find_package)
/zephyr/cmake/modules/zephyr_default.cmake:115 (include)
/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:97 (include_boilerplate)
CMakeLists.txt:11 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /usr/local/bin/cmake -DWEST_PYTHON=/opt/venv/bin/python3.8 -B/nrf/applications/Riverrock/build -GNinja -DBOARD=iotex_pebble_hw30ns -S/nrf/applications/Riverrock

if I try to use the nrfutil toolchain manager console, I get this:

-- west build: generating a build system
Loading Zephyr default modules (Zephyr base (cached)).
-- Application: /nrf/applications/Riverrock
-- CMake version: 3.20.5
-- Using NCS Toolchain 2.4.0 for building. (/root/ncs/toolchains/1f9b40e71a/cmake)
-- Found Python3: /root/ncs/toolchains/1f9b40e71a/usr/local/bin/python3 (found suitable exact version "3.8.2") found components: Interpreter
-- Cache files will be written to: /root/.cache/zephyr
-- Zephyr version: 3.3.99 (/zephyr)
-- Found west (found suitable version "1.0.0", minimum required is "0.7.1")
-- Board: iotex_pebble_hw30ns
-- Found host-tools: zephyr 0.16.0 (/root/ncs/toolchains/1f9b40e71a/opt/zephyr-sdk)
-- Found toolchain: zephyr 0.16.0 (/root/ncs/toolchains/1f9b40e71a/opt/zephyr-sdk)
-- Found Dtc: /root/ncs/toolchains/1f9b40e71a/usr/local/bin/dtc (found suitable version "1.4.7", minimum required is "1.4.6")
-- Found BOARD.dts: /nrf/boards/arm/iotex_pebble_hw30/iotex_pebble_hw30ns.dts
-- Generated zephyr.dts: /nrf/applications/Riverrock/build/zephyr/zephyr.dts
-- Generated devicetree_generated.h: /nrf/applications/Riverrock/build/zephyr/include/generated/devicetree_generated.h
-- Including generated dts.cmake file: /nrf/applications/Riverrock/build/zephyr/dts.cmake
Parsing /nrf/applications/Riverrock/Kconfig
/zephyr/scripts/kconfig/kconfig.py: /nrf/applications/Riverrock/Kconfig:64: '/nrf/applications/Riverrock/src/watchdog/Kconfig' not found (in 'rsource "src/watchdog/Kconfig"'). Check that environment variables are set correctly (e.g. $srctree, which is set to '/zephyr'). Also note that unset environment variables expand to the empty string.
CMake Error at /zephyr/cmake/modules/kconfig.cmake:339 (message):
command failed with return code: 1
Call Stack (most recent call first):
/nrf/cmake/modules/kconfig.cmake:29 (include)
/zephyr/cmake/modules/zephyr_default.cmake:115 (include)
/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:97 (include_boilerplate)
CMakeLists.txt:11 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /root/ncs/toolchains/1f9b40e71a/usr/local/bin/cmake -DWEST_PYTHON=/root/ncs/toolchains/1f9b40e71a/usr/local/bin/python3.8 -B/nrf/applications/Riverrock/build -GNinja -DBOARD=iotex_pebble_hw30ns -S/nrf/applications/Riverrock

At first glance, I think the reason is that the console sets its own WEST_PYTHON path, and uses it's own CMAKE. 

The errors you see in the compilation using the shell are not important, it's my app's coding error, but the point is that this issue is valid - west build fails since by default it's looking in the toolchain by default for Python, which I think should not be the case, unless we are using the toolchain console

[timonsku]

thanks for reproducing that is very helpful to know that its not specific to my system

[JRPC]

I found that if I turn off the installation of the ncs toolchain, it works perfectly with the default builder using python with virtual environments. if I install NCS via toolchain manager, west defaults to using python installed by the toolchain manager, even if the console is not initiated. - and if you don't initialize the console before building, like `nrfutil toolchain-manager launch --shell`, then the build will fail, since the "default" West build still relies on the toolchain manager specified versions, even though the shell was not launched; and since the shell is not launched, it's missing some parameters set by the custom shell.

Instead of using the ncs shell, I use a Docker based solution, where I wrote a specific dockerfile for each individual type of custom firmware I want to build:

# Prepare the environment variables for building the firmware
FROM ubuntu:mantic
ENV DEBIAN_FRONTEND noninteractive
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Update image and install necessary packages
RUN apt update && apt install -y apt-utils
RUN apt upgrade -y
RUN apt install -y --no-install-recommends \
    openssl \
    wget \
    xz-utils \ 
    file \
    ninja-build \
    make \
    gcc \
    gcc-multilib \
    g++-multilib \
    libssl-dev \
    libsdl2-dev \
    dfu-util \
    git \
    flex \
    bison \
    bzip2 \
    udev \
    libxcb-render0 \
    libxcb-render-util0 \
    libxcb-shape0 \
    libxcb-icccm4 \
    libxcb-keysyms1 \
    libxcb-image0 \
    libxkbcommon-x11-0 \
    libfontconfig1 \
    libfreetype6 \
    ca-certificates

# Install packages manually, since they are not provided pre-packaged by the ubuntu repositories
RUN wget https://github.com/Kitware/CMake/releases/download/v3.22.2/cmake-3.22.2.tar.gz && \
    tar -zxvf cmake-3.22.2.tar.gz && \
    mv cmake-3.22.2 /opt/cmake && \
    cd /opt/cmake && \
    ./bootstrap && \
    make -j`nproc` && \
    make install && \
    rm /cmake-3.22.2.tar.gz

# Install GNU Arm Embedded Toolchain
RUN wget https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu-rm/9-2020q2/gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2 && \
    tar -jxvf gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2 && \
    mv gcc-arm-none-eabi-9-2020-q2-update /opt/gnuarmemb && \
    rm /gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2

# Install dtc
RUN wget https://git.kernel.org/pub/scm/utils/dtc/dtc.git/snapshot/dtc-1.6.1.tar.gz && \
    tar -zxvf dtc-1.6.1.tar.gz && \
    mv dtc-1.6.1 /opt/dtc-1.6.1 && \
    cd /opt/dtc-1.6.1 && \
    make -j`nproc` && \
    make install && \
    rm /dtc-1.6.1.tar.gz 

# Install Python 3.8
RUN wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tgz && \
    tar -zxvf Python-3.8.0.tgz && \
    cd Python-3.8.0 && \
    ./configure --enable-optimizations && \
    make -j`nproc` && \
    make install && \
    rm /Python-3.8.0.tgz

# Prepare Python virtual environment
RUN python3.8 -m venv $VIRTUAL_ENV
RUN python3.8 -m pip install --upgrade pip wheel
RUN python3.8 -m pip install --upgrade west tk setuptools setuptools-rust intelhex pyelftools docopt click cryptography cbor

# Install zephyr-sdk
RUN wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.1/zephyr-sdk-0.16.1_linux-x86_64.tar.xz && \
    tar xvf zephyr-sdk-0.16.1_linux-x86_64.tar.xz && \
    mv zephyr-sdk-0.16.1 /opt/zephyr-sdk && \
    cd /opt/zephyr-sdk && \
    ./setup.sh -t all -h -c && \
    rm /zephyr-sdk-0.16.1_linux-x86_64.tar.xz

#initialize west
RUN west init -m https://github.com/nrfconnect/sdk-nrf --mr v2.4.0 && \
    west update

Then afterwards in the build script, before I do anything, I do 

west update

west zephyr-export

<---- do whatever here, like get code from git or whatever --->

west build

Previously I had these commands before the "west update" command, but they caused my default builder to go to waste, as I needed to use the NCS shell to get anything done. These lines break the builder

# RUN wget https://nsscprodmedia.blob.core.windows.net/prod/software-and-other-downloads/desktop-software/nrf-command-line-tools/sw/versions-10-x-x/10-23-2/nrf-command-line-tools_10.23.2_amd64.deb && \
#  dpkg -i nrf-command-line-tools_10.23.2_amd64.deb && \
#  rm /nrf-command-line-tools_10.23.2_amd64.deb

# RUN dpkg --unpack /opt/nrf-command-line-tools/share/JLink_Linux_V788j_x86_64.deb \
#  && rm -f /var/lib/dpkg/info/jlink.postinst \
#  && dpkg --configure jlink \
#  && apt install -yf \
#  && rm -f /opt/nrf-command-line-tools/share/JLink_Linux_V788j_x86_64.deb

# RUN wget https://developer.nordicsemi.com/.pc-tools/nrfutil/x64-linux/nrfutil -O /usr/local/bin/nrfutil && \
#  chmod +x /usr/local/bin/nrfutil

# RUN nrfutil install toolchain-manager && \
#  nrfutil toolchain-manager install --ncs-version v2.4.0 && \
#  nrfutil toolchain-manager launch --shell && \

I think this issue is related in somewhat to this legacy issue NanoPB and Zephyr SDK toolchain in NCS v2.0.x - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com) - I guess it was never really fixed

[Burt]

I have been investigating this problem in a specific environment, using nrfutil, and I think I can give a pretty good explanation, although I may not have all the detailed answers.

I have installed nrfutil, its toolchain-manager, and toolchains v2.5.0 and v2.7.0. I also installed the corresponding SDKs using the nRF Connect for Desktop toolchain-manager (actually, that's how I installed both the SDK and toolchain, as they are installed in one step). Presumably I installed v2.5.0 and then v2.7.0 at a later date.

Now look at this command and it's output:

burt@T420s:~$ nrfutil toolchain-manager list
Version Toolchain
  v2.5.0 /home/burt/ncs/toolchains/7795df4459
* v2.7.0 /home/burt/ncs/toolchains/e9dba88316

Notice the asterisk is at v2.7.0.

Suppose I try this build

burt@T420s:~$ cd ~/ncs/v2.5.0/nrf/applications/asset_tracker_v2/
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ rm -rf build
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ west build -b nrf9160dk_nrf9160_ns

I could have added -p (pristine) to the build command rather than removing the build directory; either way is fine.

The build succeeds. What most of us won't notice is that the build is being done with the v2.7.0 toolchain; the latest one I have installed. Depending on your philosophy, using the latest toolchain is a good thing, or you will believe that the v2.5.0 toolchain should be used for building v2.5.0 SDK code.

One thing I will mention here is that the v2.5.0 toolchain has python 3.8 included in it, whereas the v2.7.0 toolchain has python 3.9 included.

I probably had been building this way for a while when one day I listed the toolchains and noticed the asterisk at v2.7.0. Devzone pointed out that I could change this using

burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager install --ncs-version v2.5.0
[00:00:00] ###### 100% [Install toolchain] Toolchain with version v2.5.0 already installed
burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager list
  Version  Toolchain
* v2.5.0   /home/burt/ncs/toolchains/7795df4459
  v2.7.0   /home/burt/ncs/toolchains/e9dba88316

You see that the asterisk was moved to v2.5.0. You can also look at some environment variables

burt@T420s:~/ncs/v2.5.0/nrf/applications/asset_tracker_v2$ nrfutil toolchain-manager env
GIT_EXEC_PATH            : /home/burt/ncs/toolchains/7795df4459/usr/local/libexec/git-core
GIT_TEMPLATE_DIR         : /home/burt/ncs/toolchains/7795df4459/usr/local/share/git-core/templates
LD_LIBRARY_PATH          : /home/burt/ncs/toolchains/7795df4459/usr/lib:/home/burt/ncs/toolchains/7795df4459/usr/lib/x86_64-linux-gnu:/home/burt/ncs/toolchains/7795df4459/usr/local/lib:/home/burt/.nrfutil/lib/nrfutil-toolchain-manager
PATH                     : /home/burt/ncs/toolchains/7795df4459/usr/bin:/home/burt/ncs/toolchains/7795df4459/usr/bin:/home/burt/ncs/toolchains/7795df4459/usr/local/bin:/home/burt/ncs/toolchains/7795df4459/opt/bin:/home/burt/ncs/toolchains/7795df4459/opt/nanopb/generator-bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/aarch64-zephyr-elf/bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/x86_64-zephyr-elf/bin:/home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk/arm-zephyr-eabi/bin:/home/burt/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/burt/.nrfutil/bin
PYTHONHOME               : /home/burt/ncs/toolchains/7795df4459/usr/local
PYTHONPATH               : /home/burt/ncs/toolchains/7795df4459/usr/local/lib/python3.8:/home/burt/ncs/toolchains/7795df4459/usr/local/lib/python3.8/site-packages
ZEPHYR_SDK_INSTALL_DIR   : /home/burt/ncs/toolchains/7795df4459/opt/zephyr-sdk
ZEPHYR_TOOLCHAIN_VARIANT : zephyr

You see that 7795df4459 shows up frequently in the environment, including in the two PYTHON related variables.

I added some debug messages to the cmake files (cmake is called by the west wrapper) and I see this:

BMS NcsToolchainConfig.cmake entered...
BMS NCS_TOOLCHAIN_PATH=
BMS NCS_TOOLCHAIN_PATH=/home/burt/ncs/toolchains/e9dba88316
-- Using NCS Toolchain 2.6.20240605.1004412633878 for building. (/home/burt/ncs/toolchains/e9dba88316/cmake)
BMS just set Python3_EXECUTABLE and PYTHON_EXECUTABLE=/home/burt/ncs/toolchains/e9dba88316/usr/local/bin/python3
BMS zephyr python.cmake entered...
BMS python.cmake about to find_package Python3
BMS fn FIND_PACKAGE_HANDLE_STANDARD_ARGS entered _NAME=Python3 _FIRST_ARG=REQUIRED_VARS
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line508
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line 568
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS NAME not found
BMS FIND_PACKAGE_HANDLE_STANDARD_ARGS line 599, Could NOT find Python3 missing  Interpreter
CMake Error at /home/burt/ncs/toolchains/7795df4459/usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Interpreter) (Required is at least version
  "3.8")

      Reason given by package:
          Interpreter: Cannot run the interpreter "/home/burt/ncs/toolchains/e9dba88316/usr/local/bin/python3"

You see what's happening: part of the build system is intent on using the latest toolchain available, despite the asterisk by v2.5.0. (e9dba88316 is the v2.7.0 toolchain). But part of the build system is looking for stuff in the v2.5.0 toolchain. So the build tries to use a python 3.9 executable with python 3.8 libraries, etc. The cmake find_package code catches the mistake and exits with an error.

To do a build with v2.5.0 and nrfutil, you must do two things:

1) reinstall the v2.5.0 toolchain so the asterisk points to it.
2) add -- -DNCS_TOOLCHAIN_VERSION=NONE to the command line. This is what is done in VS Code.

Step 2 is also useful when using nRF Connect for Desktop's toolchain-manager.