NCS Uses wrong python version when built with bootloader

RE: NCS Uses wrong python version when built with bootloader

warcomeb-am — Fri, 11 Nov 2022 16:46:59 GMT

I have the same problem mentioned above but in my case, the issue is present both in VS and Terminal opened by the Toolchain manager.

The log is (I have just opened a ticket):

Building prova_multicore
west build --build-dir /Users/warcomeb/Projects/x/prova_multicore/build_nonna /Users/warcomeb/Projects/x/prova_multicore --pristine --board nrf5340dk_nrf5340_cpuapp -- -DNCS_TOOLCHAIN_VERSION:STRING="NONE" -DDTC_OVERLAY_FILE:STRING="/Users/warcomeb/Projects/x/prova_multicore/nrf5340dk_nrf5340_cpuapp.overlay" -DCONF_FILE:STRING="/Users/warcomeb/Projects/x/prova_multicore/prj.conf"

-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /Users/warcomeb/Projects/x/prova_multicore
-- Found Python3: /opt/nordic/ncs/toolchains/v2.1.0/bin/python3.9 (found suitable exact version "3.9.6") found components: Interpreter 
-- Cache files will be written to: /Users/warcomeb/Library/Caches/zephyr
-- Zephyr version: 3.1.99 (/opt/nordic/ncs/v2.1.0/zephyr)
-- Found west (found suitable version "0.14.0", minimum required is "0.7.1")
-- Board: nrf5340dk_nrf5340_cpuapp
-- Found host-tools: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)
-- Found dtc: /opt/nordic/ncs/toolchains/v2.1.0/bin/dtc (found suitable version "1.6.1", minimum required is "1.4.6")
-- Found toolchain: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)
-- Found BOARD.dts: /opt/nordic/ncs/v2.1.0/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp.dts
-- Found devicetree overlay: /Users/warcomeb/Projects/x/prova_multicore/nrf5340dk_nrf5340_cpuapp.overlay
-- Generated zephyr.dts: /Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: /Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: /Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: /Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/dts.cmake
Parsing /Users/warcomeb/Projects/x/prova_multicore/Kconfig
Loaded configuration '/opt/nordic/ncs/v2.1.0/zephyr/boards/arm/nrf5340dk_nrf5340/nrf5340dk_nrf5340_cpuapp_defconfig'
Merged configuration '/Users/warcomeb/Projects/x/prova_multicore/prj.conf'
Configuration saved to '/Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/.config'
Kconfig header saved to '/Users/warcomeb/Projects/x/prova_multicore/build_nonna/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc

=== child image b0 -  begin ===
loading initial cache file /Users/warcomeb/Projects/x/prova_multicore/build_nonna/b0/child_image_preload.cmake
Loading Zephyr default modules (Zephyr base).
-- Application: /opt/nordic/ncs/v2.1.0/nrf/samples/bootloader
CMake Error at /opt/nordic/ncs/toolchains/v2.1.0/Cellar/cmake/3.20.5/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3: Found unsuitable version "3.9.7", but required is
  exact version "3.7" (found /usr/local/bin/python3, found components:
  Interpreter)
Call Stack (most recent call first):
  /opt/nordic/ncs/toolchains/v2.1.0/Cellar/cmake/3.20.5/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:592 (_FPHSA_FAILURE_MESSAGE)
  /opt/nordic/ncs/toolchains/v2.1.0/Cellar/cmake/3.20.5/share/cmake/Modules/FindPython/Support.cmake:3165 (find_package_handle_standard_args)
  /opt/nordic/ncs/toolchains/v2.1.0/Cellar/cmake/3.20.5/share/cmake/Modules/FindPython3.cmake:485 (include)
  /opt/nordic/ncs/v2.1.0/zephyr/cmake/modules/python.cmake:46 (find_package)
  /opt/nordic/ncs/v2.1.0/zephyr/cmake/modules/zephyr_default.cmake:121 (include)
  /opt/nordic/ncs/v2.1.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /opt/nordic/ncs/v2.1.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:9 (find_package)


-- Configuring incomplete, errors occurred!
CMake Error at /opt/nordic/ncs/v2.1.0/nrf/cmake/multi_image.cmake:409 (message):
  CMake generation for b0 failed, aborting.  Command: 1
Call Stack (most recent call first):
  /opt/nordic/ncs/v2.1.0/nrf/cmake/multi_image.cmake:150 (add_child_image_from_source)
  /opt/nordic/ncs/v2.1.0/nrf/samples/CMakeLists.txt:68 (add_child_image)


-- Configuring incomplete, errors occurred!
See also "/Users/warcomeb/Projects/x/prova_multicore/build_nonna/CMakeFiles/CMakeOutput.log".
See also "/Users/warcomeb/Projects/x/prova_multicore/build_nonna/CMakeFiles/CMakeError.log".
FATAL ERROR: command exited with status 1: /opt/nordic/ncs/toolchains/v2.1.0/bin/cmake -DWEST_PYTHON=/opt/nordic/ncs/toolchains/v2.1.0/bin/python3 -B/Users/warcomeb/Projects/x/prova_multicore/build_nonna -GNinja -DBOARD=nrf5340dk_nrf5340_cpuapp -DNCS_TOOLCHAIN_VERSION:STRING=NONE -DDTC_OVERLAY_FILE:STRING=/Users/warcomeb/Projects/x/prova_multicore/nrf5340dk_nrf5340_cpuapp.overlay -DCONF_FILE:STRING=/Users/warcomeb/Projects/x/prova_multicore/prj.conf -S/Users/warcomeb/Projects/x/prova_multicore

Any ideas?

Best

Marco

RE: NCS Uses wrong python version when built with bootloader

jtrueb — Fri, 21 Oct 2022 21:15:39 GMT

Since I opened the issue, Apple's documentation on how to install python to use with tensorflow changed to a different pattern: https://developer.apple.com/metal/tensorflow-plugin/

This resolves the surface issue that wrong version of python was picked from my path because the old Apple installation script would put itself at the beginning of your PATH when you opened a new shell.

The caveat, which I think is more important and a potential attack vector, is that the build process includes unexpected changes to the environment.

I think it is a reasonable expectation for the build to only use environment variables, tools, or files from inside the toolchain or my freestanding application. I did not have my toolchain configured to use PATH in VS Code, it was set to 2.1.0.

RE: NCS Uses wrong python version when built with bootloader

Marte Myrvold — Thu, 20 Oct 2022 07:23:14 GMT

Hi,

I am glad to hear that you managed to fix the issue with Toolchain Manager. If the issue show up again, please open a new ticket to report the bug.

Thank you for testing with command line as well, and for sharing the build log. I have reported the issue to the development team, and will keep you updated.

In the meantime, a workaround is to remove the Python 3.10 installation from path as you said, but since Zephyr selects the first Python installation in path it should also work to add the correct Python installation (3.9 installed by Toolchain Manager) before 3.10 in path.

Best regards,

Marte

RE: NCS Uses wrong python version when built with bootloader

jtrueb — Tue, 18 Oct 2022 16:49:36 GMT

Good news for staying on topic on this issue. After opening and closing the Toolchain Manager several times and clicking Open Terminal for every toolchain versions several times, eventually Terminal windows would open whenever I clicked Open Terminal, so Terminal launch not working reliably on macOS is a separate Toolchain Manager issue. It works for me now.

The same environment error in VS Code reproduces in a terminal with the v2.1.0 toolchain.

(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor % west build --build-dir /Users/jwtrueb/Desktop/workspace/nirs-monitor/build /Users/jwtrueb/Desktop/workspace/nirs-monitor --pristine --board nrf52840dk_nrf52840 -- -DNCS_TOOLCHAIN_VERSION:STRING="NONE" -DBOARD_ROOT:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor;/Users/jwtrueb/Desktop/workspace/nirs-monitor/" -DCONFIG_DEBUG_OPTIMIZATIONS:STRING="y" -DCONFIG_DEBUG_THREAD_INFO:STRING="y" -DCONF_FILE:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor/prj.conf" -DDTC_OVERLAY_FILE:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor/boards/nrf52840dk_nrf52840.overlay"

-- west build: making build dir /Users/jwtrueb/Desktop/workspace/nirs-monitor/build pristine

-- west build: generating a build system

Loading Zephyr default modules (Zephyr base).

-- Application: /Users/jwtrueb/Desktop/workspace/nirs-monitor

-- Found Python3: /opt/nordic/ncs/toolchains/v2.1.0/bin/python3.9 (found suitable exact version "3.9.6") found components: Interpreter

-- Cache files will be written to: /Users/jwtrueb/Library/Caches/zephyr

-- Zephyr version: 3.1.99 (/opt/nordic/ncs/v2.1.0/zephyr)

-- Found west (found suitable version "0.14.0", minimum required is "0.7.1")

-- Board: nrf52840dk_nrf52840

-- Found host-tools: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)

-- Found dtc: /opt/nordic/ncs/toolchains/v2.1.0/bin/dtc (found suitable version "1.6.1", minimum required is "1.4.6")

-- Found toolchain: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)

-- Found BOARD.dts: /opt/nordic/ncs/v2.1.0/zephyr/boards/arm/nrf52840dk_nrf52840/nrf52840dk_nrf52840.dts

-- Found devicetree overlay: /Users/jwtrueb/Desktop/workspace/nirs-monitor/boards/nrf52840dk_nrf52840.overlay

-- Generated zephyr.dts: /Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/zephyr.dts

-- Generated devicetree_unfixed.h: /Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/include/generated/devicetree_unfixed.h

-- Generated device_extern.h: /Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/include/generated/device_extern.h

-- Including generated dts.cmake file: /Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/dts.cmake

Parsing /Users/jwtrueb/Desktop/workspace/nirs-monitor/Kconfig

Loaded configuration '/opt/nordic/ncs/v2.1.0/zephyr/boards/arm/nrf52840dk_nrf52840/nrf52840dk_nrf52840_defconfig'

Merged configuration '/Users/jwtrueb/Desktop/workspace/nirs-monitor/prj.conf'

Merged configuration '/Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/misc/generated/extra_kconfig_options.conf'

Configuration saved to '/Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/.config'

Kconfig header saved to '/Users/jwtrueb/Desktop/workspace/nirs-monitor/build/zephyr/include/generated/autoconf.h'

-- The C compiler identification is GNU 10.3.0

-- The CXX compiler identification is GNU 10.3.0

-- The ASM compiler identification is GNU

-- Found assembler: /opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc

-- Found Python3: /opt/nordic/ncs/toolchains/v2.1.0/bin/python3.9 (found version "3.9.6") found components: Interpreter

=== child image mcuboot - begin ===

loading initial cache file /Users/jwtrueb/Desktop/workspace/nirs-monitor/build/mcuboot/child_image_preload.cmake

Loading Zephyr default modules (Zephyr base).

-- Application: /opt/nordic/ncs/v2.1.0/bootloader/mcuboot/boot/zephyr

-- Found Python3: /Users/jwtrueb/miniforge3/bin/python3.10 (found suitable exact version "3.10.6") found components: Interpreter

-- Cache files will be written to: /Users/jwtrueb/Library/Caches/zephyr

-- Zephyr version: 3.1.99 (/opt/nordic/ncs/v2.1.0/zephyr)

-- Board: nrf52840dk_nrf52840

-- Found host-tools: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)

-- Found dtc: /opt/nordic/ncs/toolchains/v2.1.0/bin/dtc (found suitable version "1.6.1", minimum required is "1.4.6")

-- Found toolchain: zephyr 0.14.1 (/opt/nordic/ncs/toolchains/v2.1.0/opt/zephyr-sdk)

-- Found BOARD.dts: /opt/nordic/ncs/v2.1.0/zephyr/boards/arm/nrf52840dk_nrf52840/nrf52840dk_nrf52840.dts

-- Found devicetree overlay: /opt/nordic/ncs/v2.1.0/bootloader/mcuboot/boot/zephyr/dts.overlay

Traceback (most recent call last):

File "/opt/nordic/ncs/v2.1.0/zephyr/scripts/dts/gen_defines.py", line 33, in <module>

from devicetree import edtlib

File "/opt/nordic/ncs/v2.1.0/zephyr/scripts/dts/python-devicetree/src/devicetree/edtlib.py", line 77, in <module>

import yaml

ModuleNotFoundError: No module named 'yaml'

RE: NCS Uses wrong python version when built with bootloader

jtrueb — Tue, 18 Oct 2022 16:22:13 GMT

I can only build with the NCS toolchain via VS Code.

Nothing happens when I click Open Terminal in the NRF Connect for Desktop (v3.12.1) Toolchain Manager (v1.2.0).

There is an error in the logs of the Toolchain Manager

10:46:11.474

Initializing environments...

10:46:11.546

nrfutil-toolchain-manager 0.8.0 (92e632145e1f406f57a364e4214d7a4528af5a3a 2022-08-31)

10:46:11.550

Locally exsisting environment found at /opt/nordic/ncs/v1.9.1/toolchain

10:46:11.550

With version: v1.9.1

10:46:11.550

With west found: yes

10:46:11.637

Toolchain v2.1.0 has been added to the list

10:46:11.637

Toolchain v2.0.2 has been added to the list

10:46:12.416

Toolchain v2.0.0 has been added to the list

10:46:12.417

Toolchain v2.0.0-rc2 has been added to the list

10:46:12.417

Toolchain v2.0.1 has been added to the list

10:46:12.419

Toolchain v2.0.2 has been added to the list

10:46:12.419

Toolchain v2.1.0 has been added to the list

10:46:12.420

Toolchain v2.1.0-rc1 has been added to the list

10:46:12.420

Toolchain v2.1.0-rc2 has been added to the list

10:46:12.937

Failed to get the library versions: RangeError: Maximum call stack size exceeded

10:46:13.501

Toolchain v1.4.99-dev1 has been added to the list

10:46:13.513

Toolchain v1.4.1 has been added to the list

10:46:13.519

Toolchain v1.4.0 has been added to the list

10:46:13.524

Toolchain v1.3.2 has been added to the list

10:46:13.622

Toolchain v1.3.1 has been added to the list

10:46:13.669

Toolchain v1.3.0 has been added to the list

10:46:13.687

Toolchain v1.3.0-rc1 has been added to the list

10:46:13.803

Toolchain v1.4.2 has been added to the list

10:46:13.821

Toolchain v1.5.0-rc1 has been added to the list

10:46:13.829

Toolchain v1.5.0 has been added to the list

10:46:13.832

Toolchain v1.5.1-rc1 has been added to the list

10:46:13.841

Toolchain v1.5.1 has been added to the list

10:46:13.847

Toolchain v1.6.0 has been added to the list

10:46:13.855

Toolchain v1.6.1 has been added to the list

10:46:13.909

Toolchain v1.7.0 has been added to the list

10:46:13.931

Toolchain v1.7.1 has been added to the list

10:46:13.934

Toolchain v1.8.0 has been added to the list

10:46:13.937

Toolchain v1.9.0 has been added to the list

10:46:13.941

Toolchain v1.9.1 has been added to the list

If I manually open a terminal, add the v2.1.0 toolchain to my PATH, source the zephyr-env.sh, set the SDK toolchain variant, and build with the build command from VS Code, the build fails due to CMAKE_PREFIX_PATH issues.

(base) jwtrueb@dhcp-10-101-37-8 bin % which python3
/opt/nordic/ncs/toolchains/v2.1.0/bin/python3
(base) jwtrueb@dhcp-10-101-37-8 bin % which west /opt/nordic/ncs/toolchains/v2.1.0/bin/west

(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor % echo $ZEPHYR_BASE
/opt/nordic/ncs/v2.1.0/zephyr
(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor % echo $ZEPHYR_SDK_INSTALL_DIR
/opt/nordic/ncs/toolchains/v2.1.0
(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor % echo $ZEPHYR_TOOLCHAIN_VARIANT
zephyr
(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor % west build --build-dir /Users/jwtrueb/Desktop/workspace/nirs-monitor/build /Users/jwtrueb/Desktop/workspace/nirs-monitor --pristine --board nrf52840dk_nrf52840 -- -DNCS_TOOLCHAIN_VERSION:STRING="NONE" -DBOARD_ROOT:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor;/Users/jwtrueb/Desktop/workspace/nirs-monitor/" -DCONFIG_DEBUG_OPTIMIZATIONS:STRING="y" -DCONFIG_DEBUG_THREAD_INFO:STRING="y" -DCONF_FILE:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor/prj.conf" -DDTC_OVERLAY_FILE:STRING="/Users/jwtrueb/Desktop/workspace/nirs-monitor/boards/nrf52840dk_nrf52840.overlay"
-- west build: making build dir /Users/jwtrueb/Desktop/workspace/nirs-monitor/build pristine
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /Users/jwtrueb/Desktop/workspace/nirs-monitor
-- Found Python3: /opt/nordic/ncs/toolchains/v2.1.0/bin/python3.9 (found suitable exact version "3.9.6") found components: Interpreter
-- Cache files will be written to: /Users/jwtrueb/Library/Caches/zephyr
-- Zephyr version: 3.1.99 (/opt/nordic/ncs/v2.1.0/zephyr)
-- Found west (found suitable version "0.14.0", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
CMake Error at /opt/nordic/ncs/v2.1.0/zephyr/cmake/modules/verify-toolchain.cmake:79 (find_package):
Could not find a package configuration file provided by "Zephyr-sdk"
(requested version 0.13.1) with any of the following names:

Zephyr-sdkConfig.cmake
zephyr-sdk-config.cmake

Add the installation prefix of "Zephyr-sdk" to CMAKE_PREFIX_PATH or set
"Zephyr-sdk_DIR" to a directory containing one of the above files. If
"Zephyr-sdk" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
/opt/nordic/ncs/v2.1.0/zephyr/cmake/modules/zephyr_default.cmake:121 (include)
/opt/nordic/ncs/v2.1.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
/opt/nordic/ncs/v2.1.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
CMakeLists.txt:12 (find_package)

-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /opt/nordic/ncs/toolchains/v2.1.0/bin/cmake -DWEST_PYTHON=/opt/nordic/ncs/toolchains/v2.1.0/bin/python3 -B/Users/jwtrueb/Desktop/workspace/nirs-monitor/build -GNinja -DBOARD=nrf52840dk_nrf52840 -DNCS_TOOLCHAIN_VERSION:STRING=NONE '-DBOARD_ROOT:STRING=/Users/jwtrueb/Desktop/workspace/nirs-monitor;/Users/jwtrueb/Desktop/workspace/nirs-monitor/' -DCONFIG_DEBUG_OPTIMIZATIONS:STRING=y -DCONFIG_DEBUG_THREAD_INFO:STRING=y -DCONF_FILE:STRING=/Users/jwtrueb/Desktop/workspace/nirs-monitor/prj.conf -DDTC_OVERLAY_FILE:STRING=/Users/jwtrueb/Desktop/workspace/nirs-monitor/boards/nrf52840dk_nrf52840.overlay -S/Users/jwtrueb/Desktop/workspace/nirs-monitor
(base) jwtrueb@dhcp-10-101-37-8 nirs-monitor %

Is there documentation on how to use the NRF Connect SDK in a standalone terminal setting like this? To avoid CMake issues, is Open Terminal or VS Code the only supported way to build with the toolchain with the automatic installation? I know there is documentation on how to use west and whatnot, but I am more so referencing how to make sure that the NRF Connect SDK environment is installed and the terminal session has the shell environment setup to point to the right toolchain utilities. There is documentation on how to manually set everything up, but to my understanding some things were automatically installed by the Toolchain Manager in a way that doesn't match the manual installation documentation.

RE: NCS Uses wrong python version when built with bootloader

Marte Myrvold — Tue, 18 Oct 2022 12:26:51 GMT

Hi,

Thank you for the detailed explanation and logs.

Do you get the same if you build with a terminal opened via Toolchain Manager, or does this only happen in VS Code?

The issue is the Python version selected by Zephyr, which is used by imgtool among others. Zephyr selects the first Python installation in path, unless the environment is set up correctly. This should be done automatically, but there seems to be a bug in VS Code or some of the other tools causing this to not be set correctly. If you can test with terminal it would be of great help in order to narrow down where the issue is.

Best regards,

Marte