Apple M1, SDK v2.5.2, toolchain-supplied Python uses system-installed site-packages

[Mikael Jansson]

MacBook M1. Latest macOS. SDK v2.5.2. Building blinky w/ CONFIG_BOOTLOADER_MCUBOOT=1 enabled.

Doesn't matter if I start an nrfutil shell* or my normal settings.

* nrfutil toolchain-manager launch --shell --ncs-version v2.5.0

$ west build  -b circuitdojo_feather_nrf9160_ns
[0/222] Performing build step for 'tfm'
[4/8] Linking C executable bin/tfm_s.axf
Memory region         Used Size  Region Size  %age Used
           FLASH:       31544 B      32256 B     97.79%
             RAM:       18832 B        32 KB     57.47%
[8/8] Generating ../bin/tfm_s.elf
[1/222] Performing build step for 'mcuboot_subimage'
[18/124] Generating zephyr/autogen-pubkey.c
FAILED: zephyr/autogen-pubkey.c
cd .../nrf-sdk-test-2024-02-28/blinky/build/mcuboot && /opt/nordic/ncs/toolchains/20d68df7e5/bin/python3 /opt/nordic/ncs/v2.5.2/bootloader/mcuboot/scripts/imgtool.py getpub -k /opt/nordic/ncs/v2.5.2/bootloader/mcuboot/root-ec-p256.pem > .../nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/autogen-pubkey.c
Traceback (most recent call last):
  File "/opt/nordic/ncs/v2.5.2/bootloader/mcuboot/scripts/imgtool.py", line 19, in <module>
    from imgtool import main
  File "/opt/nordic/ncs/v2.5.2/bootloader/mcuboot/scripts/imgtool/main.py", line 26, in <module>
    from imgtool import image, imgtool_version
  File "/opt/nordic/ncs/v2.5.2/bootloader/mcuboot/scripts/imgtool/image.py", line 35, in <module>
    from cryptography.hazmat.primitives.kdf.hkdf import HKDF
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/cryptography/hazmat/primitives/kdf/hkdf.py", line 13, in <module>
    from cryptography.hazmat.primitives import constant_time, hashes, hmac
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/cryptography/hazmat/primitives/hmac.py", line 12, in <module>
    from cryptography.hazmat.backends.openssl.hmac import _HMACContext
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 63, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/Users/me/Library/Python/3.9/lib/python/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/Users/me/Library/Python/3.9/lib/python/site-packages/_cffi_backend.cpython-39-darwin.so, 0x0002): tried: '/Users/me/Library/Python/3.9/lib/python/site-packages/_cffi_backend.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/me/Library/Python/3.9/lib/python/site-packages/_cffi_backend.cpython-39-darwin.so' (no such file), '/Users/me/Library/Python/3.9/lib/python/site-packages/_cffi_backend.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
[29/124] Building C object zephyr/CMakeFiles/zephyr.dir/lib/os/cbprintf_nano.c.obj
ninja: build stopped: subcommand failed.
[2/222] Performing install step for 'tfm'
-- Install configuration: "RelWithDebInfo"
[36/222] Building C object modules/nrf/subsys/nrf_security/src/CMakeFiles/mbedcrypto.dir/opt/nordic/ncs/v2.5.2/modules/crypto/mbedtls/library/gcm.c.obj
FAILED: modules/mcuboot/mcuboot_subimage-prefix/src/mcuboot_subimage-stamp/mcuboot_subimage-build mcuboot/zephyr/zephyr.hex mcuboot/zephyr/zephyr.elf
cd .../nrf-sdk-test-2024-02-28/blinky/build/mcuboot && /opt/nordic/ncs/toolchains/20d68df7e5/Cellar/cmake/3.20.5/bin/cmake --build . --
[48/222] Building C object modules/nrf/subsys/nrf_security/src/CMakeFiles/mbedcrypto.dir/opt/nordic/ncs/v2.5.2/modules/crypto/mbedtls/library/gcm.c.obj
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: /opt/nordic/ncs/toolchains/20d68df7e5/bin/cmake --build .../nrf-sdk-test-2024-02-28/blinky/build
$

The provided Python is x86_64, my system is aarch64.

As you can see, the build system uses its own Python, but links to my system files.

The provided toolchain Python should use its own site-packges.

This is a clean install of v2.5.2 from nRF Connect Desktop.

Thanks in advance,
- Mikael

[Susheel Nuguru]

Thanks for the report Michael,

I do not have Mac to test, but I can forward this to the toolchain team and get you some updates on what can be done. Be back to you when I hear from them.

[Mikael Jansson]

Thanks Sushi.

For other readers, there is a *workaround* set PYTHONPATH=/opt/nordic/ncs/toolchains/20d68df7e5/lib/python3/site-packages before running west build. You do _not_ want to have this globally always set, because it'll mess all your other system apps using Python code up.

Obviously, this is not an acceptable solution, but it allows you to resume your work.

[Susheel Nuguru]

Mikael,

Your workaround shows that your toolchain installation somehow using the global PYTHONPATH which it ideally should not.

Here are at least some observarions: 

• The error trace from Python shows that the system installation of some python libraries are being used at one point
• Ideally, our python installation should never need to interact with the system installation of Python for anything. 
• We have had issues with the cmake cache before, so trying to clear up interactions with cmake may help, but this might not be related to the issue you are seeing.

But it could be helpful to see what environment variables there are and if you are affecting the system. Can you run

 

echo "$PATH"
echo "$PYTHONHOME"
echo "$PYTHONPATH"

We would also suggest upgrading to version 2.5.2, as some bugs may have been fixed by that. 

Finally, running 

• which -a python should show all python installations and if one form inside the toolchain bundle is currently being used
• pip list -vvv should show where the different pip packages are installed, and therefore which ones may cause issues
  • Here, you will see some issues if you have globally installed packages that you try to use that have not been installed inside the toolchain bundle

[Mikael Jansson]

Hi,

This is on v2.5.2.

PYTHONPATH and PYTHONHOME are empty. The default is for Python to construct it based on the location of the binary.

PATH=/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin

which -a python3 gives /usr/bin/python3, i.e. the macOS system-installed python3. Note that its python3, not python. That's reserved for python2.x, which NCS doesnt use.

It's really easy to test this. Just install the offending crypto libraries that use cffi on your system using pip3 install <libraries>, then try building mcuboot using the ncs toolchain.

[Susheel Nuguru]

If the shell is to use the toolchain then this PATH is wrong, python is in the toolchain and the toolchain should be in PATH. Also I see no reason to install anything with pip, because all the modules needed for NCS development should be contained in the toolchain.

If PYTHONPATH and PYTHONHOME are empty, and the path is that simple, then spawning the shell should be sufficient. The modules should already be installed, and the libraries relating to the failure in the toolchain bundle. 

So our only current guess is to make sure all operations are done in the shell, and do the entire setup for the project from scratch to make sure that no old state accidentally points to the system installation. 

We have made an internal ticket to test this later. NRFU-932 , but we don't know when we can prioritize this investigation.. 

you can use  pip list -vvv and python -c "import sys; print(*sys.path, sep='\\n')"  to check where the potentially bad packages may originate from