Problem building with application west.yml in ncs-v1.5.1

[Jens Cameron]

We have developed an application in ncs-1.3.0 where we created a west.yml file for the application that lists nrf, zephyr and other dependent repositories. We have downstream clones of all relevant repos to be able to put our corporate fixes into the source without clobbering public repos.

When upgrading to ncs-1.5.1 this apparently does not work, build fails.

Building the formware appears to require the nrf/west.yml to be the initialized west.

Replacing our west file with the nrf/west builds but changing the self:path: from nrf to something else fails to build.

We have tried a simplistic application west file that just imports the nrf/west.yml. This fails to build.

Using our custom application/west.yml fails with different error, semms to indicate that boilerplate.cmake is imported twice.

Apparently the west file is not only for synchronizing repositories but also part of the build process.

We need is to have downstream clones of all repos for our own fixes. How do we accomplish this, preferrably with a minimum of commits into the downstream repositories and with the application sync'd with commits in other repos?

[Heidi]

Hi!

Which version of west are you using?

Please note that due to the summer holidays in Norway, a large portion of the staff are on vacation in the next few weeks and delays are to be expected.

Best regards,

Heidi

[Jens Cameron]

We are using west vrsoin 0.9 as in the toolchain for ncs-v1.5.1.
Technically we are fetching the contents of 
https://developer.nordicsemi.com/.pc-tools/toolchain/ncs-toolchain-v1.5.1-20210416-96ab6aa.zip
and unzip these into the Toolchain subfolder.

We have tried it in a freshly installed HyperV VM so we are pretty sure nothing else in the system clobbers the behaviour.

The repository related commands "west init -l" and "west update" works without problem. It is in the build proces we get errors.

It turns out we can use a west file in our application directory as long as it ends with the following lines:

  # West-related configuration for the nrf repository.
  self:
    # This repository should be cloned to ncs/nrf.
    path: nrf
    # This line configures west extensions which are currently only
    # for internal use by NCS maintainers.
    west-commands: scripts/west-commands.yml

First log lines from build looks like this:

  1 bash scripts/nrf_version_check.sh
  2 cmake -B_build_uma_nrf52840_uma_ep2_Debug -GNinja -DBOARD=nrf52840_uma_ep2 -S applications/uma -DCMAKE_BUILD_TYPE=Debug
  3 CMake setup for debug build
  4 Including boilerplate (Zephyr base): C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake
  5 -- Application: C:/workspace/ncs2/uma_nrf52840_app/applications/uma
  6 -- Using NCS Toolchain 1.5.1 for building. (C:/workspace/ncs3/toolchain/cmake)
  7 -- Zephyr version: 2.4.99 (C:/workspace/ncs2/zephyr)
  8 -- Found Python3: C:/workspace/ncs3/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter 
  9 -- Found west (found suitable version "0.9.0", minimum required is "0.7.1")
 10 -- Board: nrf52840_uma_ep2
 11 -- Cache files will be written to: C:/workspace/ncs2/zephyr/.cache
 12 -- Found dtc: C:/workspace/ncs3/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
 13 -- Found toolchain: gnuarmemb (C:/workspace/ncs3/toolchain/opt)
 14 -- Found BOARD.dts: C:/workspace/ncs2/uma_nrf52840_app/boards/arm/nrf52840_uma/nrf52840_uma_ep2.dts
 15 -- Generated zephyr.dts: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/zephyr.dts
 16 -- Generated devicetree_unfixed.h: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/include/generated/devicetree_unfixed.h
 17 -- Generated device_extern.h: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/include/generated/device_extern.h
 18 Parsing C:/workspace/ncs2/uma_nrf52840_app/applications/uma/Kconfig
 19 Loaded configuration 'C:/workspace/ncs2/uma_nrf52840_app/boards/arm/nrf52840_uma/nrf52840_uma_ep2_defconfig'
 20 Merged configuration 'prj.conf'
 21 Merged configuration 'debug.conf'
 22 Configuration saved to 'C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/.config'
 23 Kconfig header saved to 'C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/include/generated/autoconf.h'
 24 -- The C compiler identification is GNU 9.2.1
 25 -- The CXX compiler identification is GNU 9.2.1
 26 -- The ASM compiler identification is GNU
 27 -- Found assembler: C:/workspace/ncs3/toolchain/opt/bin/arm-none-eabi-gcc.exe
 28 
 29 === child image mcuboot -  begin ===
 30 Including boilerplate (Zephyr base): C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake
 31 -- Application: C:/workspace/ncs2/bootloader/mcuboot/boot/zephyr

If we replace the last few lines of west.yml with 

  # West-related configuration for the nrf repository.
  self:
    # This repository should be cloned to ncs/nrf.
    path: app
    # This line configures west extensions which are currently only
    # for internal use by NCS maintainers.
    west-commands: scripts/west-commands.yml

The west commands still work but the build fails. Log in this case becomes:

 1 bash scripts/nrf_version_check.sh
 2 cmake -B_build_uma_nrf52840_uma_ep2_Debug -GNinja -DBOARD=nrf52840_uma_ep2 -S applications/uma -DCMAKE_BUILD_TYPE=Debug
 3 CMake setup for debug build
 4 Including boilerplate (Zephyr base): C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake
 5 -- Application: C:/workspace/ncs2/uma_nrf52840_app/applications/uma
 6 -- Using NCS Toolchain 1.5.1 for building. (C:/workspace/ncs3/toolchain/cmake)
 7 -- Zephyr version: 2.4.99 (C:/workspace/ncs2/zephyr)
 8 -- Found Python3: C:/workspace/ncs3/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter 
 9 -- Found west (found suitable version "0.9.0", minimum required is "0.7.1")
10 -- Board: nrf52840_uma_ep2
11 -- Cache files will be written to: C:/workspace/ncs2/zephyr/.cache
12 -- Found dtc: C:/workspace/ncs3/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
13 -- Found toolchain: gnuarmemb (C:/workspace/ncs3/toolchain/opt)
14 -- Found BOARD.dts: C:/workspace/ncs2/uma_nrf52840_app/boards/arm/nrf52840_uma/nrf52840_uma_ep2.dts
15 -- Generated zephyr.dts: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/zephyr.dts
16 -- Generated devicetree_unfixed.h: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/include/generated/devicetree_unfixed.h
17 -- Generated device_extern.h: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/zephyr/include/generated/device_extern.h
18 Parsing C:/workspace/ncs2/uma_nrf52840_app/applications/uma/Kconfig
19 C:/workspace/ncs2/zephyr/scripts/kconfig/kconfig.py: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/Kconfig/Kconfig.modules:2: Could not open 'C:\workspace\ncs2\zephyr\' (in 'osource "$(ZEPHYR_MCUBOOT_KCONFIG)"') (ENOENT: No such file or directory)
20 CMake Error at C:/workspace/ncs2/zephyr/cmake/kconfig.cmake:262 (message):
21   command failed with return code: 1
22 Call Stack (most recent call first):
23   C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake:534 (include)
24   C:/workspace/ncs2/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
25   C:/workspace/ncs2/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:35 (include_boilerplate)
26   CMakeLists.txt:71 (find_package)
27 
28 
29 -- Configuring incomplete, errors occurred!
30 make: *** [Makefile:86: uma_nrf52840_uma_ep2_Debug] Error 1

To me it appears that it doesn't have with hte repositories to do but with some other use of the west file in the build process.

We can work with a custom west.yml file in the application folder like this but it is error prone to not have repository management for the nrf folder but have to synchronize application and nrf repositories manually.
We will eventually hadnd over the code to our customer that will maintain it and to have repository management streamlined would definitely be an advantage.

[Torsten Rasmussen]

Hi Jens,

Thanks for your detailed report.
I have not been able to reproduce the error you are facing, so let me first try to provide links / info on how you should be able to achieve what you want.

Based on your description, I understand it so that your prefered way is to have the application itself as the manifest repo.
That is described here: developer.nordicsemi.com/.../dm_adding_code.html

with a detailed example.

Could you try to go through that description and verify that you are doing things according to the guide.

Besides doing so, then let me try to reply some of your earlier questions  and observations.

Using our custom application/west.yml fails with different error, semms to indicate that boilerplate.cmake is imported twice.

are your referring to those messages ?

Including boilerplate (Zephyr base): C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake
....
=== child image mcuboot -  begin ===
Including boilerplate (Zephyr base): C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake

then that is expected behavior because each child image is built as an independent NCS / Zephyr application and thus each of them will source the boilerplate code for their own build.

Apparently the west file is not only for synchronizing repositories but also part of the build process.

when west is used for repository management, then the build system will use west to gather a list of projects / repositories handled by west to ensure those repositories are included into the build.

Each of the repositories returned by west is examined to see if it contains a zephyr/module.yml file, and if so, the repo is considered a Zephyr module and included in the build.
See more details here: developer.nordicsemi.com/.../modules.html

C:/workspace/ncs2/zephyr/scripts/kconfig/kconfig.py: C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug/Kconfig/Kconfig.modules:2: Could not open 'C:\workspace\ncs2\zephyr\' (in 'osource "$(ZEPHYR_MCUBOOT_KCONFIG)"') (ENOENT: No such file or directory)
20 CMake Error at C:/workspace/ncs2/zephyr/cmake/kconfig.cmake:262 (message):

This error is puzzling to me.
The presence of ZEPHYR_MCUBOOT_KCONFIG indicates that Mcuboot is found as a Zephyr module, so the repo is present and having a `zephyr/module.yml file, with `kconfig-ext: True` as content.

But the `No such file or directory` indicates that either the repo is not located at the expected location, the `modules/mcuboot/Kconfig` file is missing in your fork (github.com/.../Kconfig), which I doubt, or that modules/modules.cmake has not been processed (github.com/.../modules.cmake).

Could you try to re-run CMake with the `--trace-expand` option and attach the output ?

cd C:/workspace/ncs2/uma_nrf52840_app/_build_uma_nrf52840_uma_ep2_Debug
cmake --trace-expand . 2> cmake.trace

This will provide valuable info on what is actually happening in the system.

If we replace the last few lines of west.yml with

Just replacing `path: nrf` with `path: app` and not change the folder layout is bound to give unexpected behavior. So let's try to get your manifest file in a shape where the import of sdk-nrf is actually working.

Remember, it's not enough to update just sdk-nrf to 1.5.1
You must also update your mcuboot fork to be based on the version used in 1.5.1, and similar for other repos you have forked.
I assume you have already done so, just trying to make it safe here ;-)

[Jens Cameron]

Hi Torsten

Thanks fo ryour answer.

Attached our $/uma_nrf52480_app/west.yml file and cmake.log as you suggested.

# The west manifest file (west.yml) for the nRF Connect SDK (NCS).
#
# The per-workspace west configuration file, ncs/.west/config,
# specifies the location of this manifest file like this:
#
#     [manifest]
#     path = nrf
#
# See the west documentation for more information:
#
# https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/guides/west/index.html

# This is an application west file but ncs requires west to have nrf as base
# Update this reference to point at a specific ncs version including svep fixes
# nrf-revision: bcbc02cb833105acbb35b9ffc83ad3735fa15342

manifest:
  # This west.yml requires west 0.9 or later, because the "submodules"
  # feature used below was introduced then.
  version: 0.9

  # "remotes" is a list of locations where git repositories are cloned
  # and fetched from.
  remotes:
    # nRF Connect SDK GitHub organization.
    # NCS repositories are hosted here.
    # - name: ncs
      # url-base: https://github.com/nrfconnect
    # # Third-party repository sources:
    # - name: zephyrproject
      # url-base: https://github.com/zephyrproject-rtos
    # - name: throwtheswitch
      # url-base: https://github.com/ThrowTheSwitch
    # - name: armmbed
      # url-base: https://github.com/ARMmbed
    # - name: nanopb
      # url-base: https://github.com/nanopb
    # - name: alexa
      # url-base: https://github.com/alexa
    # - name: nordicsemi
      # url-base: https://github.com/NordicSemiconductor
    - name: svep
      url-base: [email protected]:1852-08

  # If not otherwise specified, the projects below should be obtained
  # from the ncs remote.
  defaults:
    remote: svep

  # "projects" is a list of git repositories which make up the NCS
  # source code.
  projects:

    # The Zephyr RTOS fork in the NCS, along with the subset of its
    # modules which NCS imports directly.
    #
    # https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/introduction/index.html
    # https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/guides/modules.html
    - name: nrf
      repo-path: fw-nrfconnect-nrf
      #official repo-path: sdk-nrf
      path: nrf
      revision: bcbc02cb833105acbb35b9ffc83ad3735fa15342 # ncs-v1.5.1-svep
      west-commands: scripts/west-commands.yml
    - name: fw-nrfconnect-zephyr
      path: zephyr
      west-commands: scripts/west-commands.yml
      revision: 484f5a58c9ce60fc06aa464359cfd04b61390bd1 # v2.4.99-ncs2-svep
    - name: cmsis
      revision: 421dcf358fa420e9721a8452c647f0d42af8d68c
      path: modules/hal/cmsis
    - name: segger
      revision: 38c79a447e4a47d413b4e8d34448316a5cece77c
      path: modules/debug/segger
      # remote: zephyrproject
    - name: should_be_modules_dash_crypto_dash_mbedtls # crypto_mbedtls
      path: modules/crypto/mbedtls
      revision: 13cf2e52024a144ecee9f37680681760a85febab
      # remote: zephyrproject
    - name: mcuboot
      repo-path: sdk-mcuboot
      revision: 11aa5f08cf9dd5815f5cf51ecf6e21f9c1ab65e6 # svep-v1.5.1
      path: bootloader/mcuboot
    - name: mcumgr
      repo-path: fw-nrfconnect-mcumgr
      revision: 43845e883ff3a6cdaae22e23f3e60b5fcf78c6ba
      path: modules/lib/mcumgr
    - name: tinycbor
      repo-path: fw-nrfconnect-tinycbor
      path: modules/lib/tinycbor
      revision: 40daca97b478989884bffb5226e9ab73ca54b8c4
      # remote: zephyrproject
    - name: ci-tools
      path: tools/ci-tools
      # remote: zephyrproject
      revision: da9a2df574094f52d87a03f6393928bdc7dce17c
    - name: net-tools
      path: tools/net-tools
      # remote: zephyrproject
      revision: 41132e9220f8bc1223084975350c5e5f3b492afe
    - name: nrfxlib
      repo-path: nrfxlib
      path: nrfxlib
      revision: 3cff48e890b037a0fe2964091702cb1c1bb59926 # v1.5.1
    - name: cmock
      path: test/cmock
      revision: 9d092898ef26ece140d9225e037274b64d4f851e
      # remote: throwtheswitch
    - name: unity
      path: test/cmock/vendor/unity
      revision: 74cde089e65c3435ce9aa87d5c69f4f16b7f6ade
      # remote: throwtheswitch
    - name: mbedtls-nrf
      path: mbedtls
      repo-path: fw-nrfconnect-mbedtls
      revision: 523f0554b6cdc7ace5d360885c3f5bbcc73ec0e8 # mbedtls-2.24.0
      # remote: armmbed
#    - name: civetweb
#      path: modules/lib/civetweb
#      remote: civetweb
#      revision: 99129c5efc907ea613c4b73ccff07581feb58a7a
#    - name: edtt
#      path: tools/edtt
#      remote: zephyrproject
#      revision: dd4dd502ef2fbeced6ef7faaba562a7ddca45632
    - name: fatfs
      path: modules/fs/fatfs
      # remote: zephyrproject
      revision: 1d1fcc725aa1cb3c32f366e0c53d7490d0fe1109
    - name: hal_nordic
      path: modules/hal/nordic
      # remote: zephyrproject
      revision: b96360f2446b772c2495ca49d4137b9b8f4bb2a8 # v1.5.1
    - name: hal_st
      path: modules/hal/st
      # remote: zephyrproject
      revision: b52fdbf4b62439be9fab9bb4bae9690a42d2fb14
    - name: libmetal
      path: modules/hal/libmetal
      # remote: zephyrproject
      revision: 9d4ee2c3cfd5f49861939447990f3b7d7bf9bf94
#    - name: loramac-node
#      path: modules/lib/loramac-node
#      remote: zephyrproject
#      revision: 29e516ec585b1a909af2b5f1c60d83e7d4d563e3
#    - name: lvgl
#      path: modules/lib/gui/lvgl
#      remote: zephyrproject
#      revision: 74fc2e753a997bd71cefa34dd9c56dcb954b42e2
    - name: nrf_hw_models
      path: modules/bsim_hw_models/nrf_hw_models
      # remote: zephyrproject
      revision: a47e326ca772ddd14cc3b9d4ca30a9ab44ecca16
    - name: open-amp
      path: modules/lib/open-amp
      # remote: zephyrproject
      revision: de1b85a13032a2de1d8b6695ae5f800b613e739d
    - name: openthread
      path: modules/lib/openthread
      # remote: zephyrproject
      revision: aad6518ee30d3840411b21437d396f0405e947c3
    - name: littlefs
      path: modules/fs/littlefs
      # remote: zephyrproject
      revision: 9e4498d1c73009acd84bb36036ee5e2869112a6c
    - name: mipi-sys-t
      path: modules/debug/mipi-sys-t
      # remote: zephyrproject
      revision: 75e671550ac1acb502f315fe4952514dc73f7bfb
#    - name: canopennode
#      path: modules/lib/canopennode
# #      remote: zephyrproject
#      revision: 5c6b0566d56264efd4bf23ed58bc7cb8b32fe063
    - name: tinycrypt
      path: modules/crypto/tinycrypt
      # remote: zephyrproject
      revision: 3e9a49d2672ec01435ffbf0d788db6d95ef28de0

  # West-related configuration for the nrf repository.
  self:
    # This repository should be cloned to ncs/nrf.
    path: uma_nrf52840_app
    # This line configures west extensions which are currently only
    # for internal use by NCS maintainers.
    # west-commands: scripts/west-commands.yml

cmake.log

You are probably right that there is some misconfig in the mcuboot setup. Al repos are updated (a few of them are one or two commits past the official v1.5.1  but nothing I could imagine affects this).

mcuboot is checked ou tin the folder $/bootloader/mcuboot, just as it is in the $/nrf/west.yml. There is no mcuboot in $/modules. We had to add a prj.conf into th e$/nrf/modules/mcuboot to make it build but other than that we haven't manipulated anything in $/nrf folder.

My suspicion with boilerplate included twice is cmake errors like

CMake Error at C:/workspace/ncs2/zephyr/cmake/app/boilerplate.cmake:94 (add_custom_target):
  add_custom_target cannot create target "code_data_relocation_target"
  because another target with the same name already exists.  The existing
  target is a custom target created in source directory
  "C:/workspace/ncs2/uma_nrf52840_app/applications/uma".  See documentation
  for policy CMP0002 for more details.
Call Stack (most recent call first):
  C:/workspace/ncs2/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  C:/workspace/ncs2/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:40 (include_boilerplate)
  ../../CMakeLists.txt:2 (find_package)

but as the project is building when the $/uma_nrf52840_app/west.yml claims that nrf is the "self" repo it is probably irrelevant.

We use a slightly modified version of the mcuboot main located in $/bootloader/mcuboot/boot/zephyr.

If I understand you correctly we also need to put something under $/nrf/modules/mcuboot (other than prj.conf), or configure the system to look directly in $/bootloader/mcuboot for files?

What is puzzeling to me is that the project builds if I change the name in the west file to nrf (and remove the nrf subproject). 

[Jens Cameron]

Hi Torsten. 

It appears Chrome eats some "Reply" buttons so I accidentally responded to Heidi instead. Could you have a look?

Regards

--Jens