Trouble migrating from NCS 2.6 to 2.8

[BrianW]

Due to issues with the wifi stack, I have to move my project from 2.6 to 2.8.

I downloaded the new tools set (toolchains\cf2149caf2) and new nordic connect version 2.8.0 as per

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/installation/updating.html

I read the migration guides for 2.7 and 2.8, and the 'using sysbuild for multi-image builds guides:

https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/build/sysbuild/index.html#sysbuild

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_sysbuild.html

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_guide_2.8.html

Mostly this gave me a headache.

I created sysbuild.conf (at the root where I run west - is this right?), and filled in the stuff from the migration guides for mcuboot and nrf70 support.

# MCU boot config now handled by sysbuild
SB_CONFIG_BOOTLOADER_MCUBOOT=y
SB_CONFIG_MCUBOOT_BOOTLOADER_MODE_SWAP_WITHOUT_SCRATCH=y
SB_CONFIG_MCUBOOT_GENERATE_UNSIGNED_IMAGE=y

SB_CONFIG_NETCORE_HCI_IPC=y
SB_CONFIG_NETCORE_APP_UPDATE=y
SB_CONFIG_BOOT_SIGNATURE_TYPE_ECDSA_P256=y
SB_CONFIG_BOOT_SIGNATURE_KEY_FILE="/work/dev/if-device-nrf53/keys/bootloader_priv-ecdsa256.pem"
#CONFIG_MCUBOOT_SIGNATURE_KEY_FILE="/work/dev/if-device-nrf53/keys/bootloader_priv-ecdsa256.pem"

SB_CONFIG_MCUBOOT_BOOTLOADER_MODE_SWAP_WITHOUT_SCRATCH=y

# nrf7002 firmware handled by sysbuild
SB_CONFIG_WIFI_NRF70=y
SB_CONFIG_WIFI_NRF70_SYSTEM_MODE=y
#SB_CONFIG_WIFI_PATCHES_EXT_FLASH_XIP=y     later

# build of dfu packages
SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_BUILD=y
SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_APP=y
SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_NET=y
SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_WIFI_FW_PATCH=y

SB_CONFIG_DFU_ZIP=y
SB_CONFIG_DFU_ZIP_APP=y
SB_CONFIG_DFU_ZIP_NET=y
SB_CONFIG_DFU_ZIP_WIFI_FW_PATCH=y

# put slot2 in external flash
SB_CONFIG_PARTITION_MANAGER=y
SB_CONFIG_PM_MCUBOOT_PAD=y
SB_CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y

I updated my prj.conf for the changes to nrf70 config option names

I created sysbuild/mcuboot and copied in the prj.conf from the previous child_image/mcuboot.conf

There was already a sysbuild/hci_ipc/prj.conf, I updated it with changes I had made to child_image/hci_ipc.conf 

Then I set west to use my board and sysbuild and try to build:

> west config build.board cc1medv1_nrf5340_cpuapp

> west config build.sysbuild True

> west build --build-dir cc1-med/build cc1-med --board cc1medv1_nrf5340_cpuapp --pristine -DBOARD_ROOT=%CD%

This got me cmake failing.

-- west build: making build dir C:\work\dev\if-device-nrf53\cc1-med\build pristine
-- west build: generating a build system
Loading Zephyr module(s) (Zephyr base): sysbuild_default
-- Found Python3: C:/ncs/toolchains/2d382dcd92/opt/bin/python.exe (found suitable version "3.12.4", minimum required is "3.8") found components: Interpreter
-- Cache files will be written to: C:/ncs/v2.8.0/zephyr/.cache
-- Found west (found suitable version "1.2.0", minimum required is "0.14.0")
-- Board: cc1medv1_nrf5340_cpuapp
Parsing C:/work/dev/if-device-nrf53/cc1-med/Kconfig.sysbuild
Loaded configuration 'C:/work/dev/if-device-nrf53/cc1-med/build/_sysbuild/empty.conf'
Merged configuration 'C:/work/dev/if-device-nrf53/cc1-med/build/_sysbuild/empty.conf'
Configuration saved to 'C:/work/dev/if-device-nrf53/cc1-med/build/zephyr/.config'
Kconfig header saved to 'C:/work/dev/if-device-nrf53/cc1-med/build/_sysbuild/autoconf.h'
CMake Error at C:/ncs/v2.8.0/nrf/sysbuild/CMakeLists.txt:117 (list):
  list GET given empty list
Call Stack (most recent call first):
  cmake/modules/sysbuild_extensions.cmake:583 (nrf_PRE_CMAKE)
  cmake/modules/sysbuild_extensions.cmake:583 (cmake_language)
  cmake/modules/sysbuild_images.cmake:16 (sysbuild_module_call)
  cmake/modules/sysbuild_default.cmake:20 (include)
  C:/ncs/v2.8.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:75 (include)
  C:/ncs/v2.8.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  C:/ncs/v2.8.0/zephyr/share/sysbuild-package/cmake/SysbuildConfig.cmake:8 (include)
  template/CMakeLists.txt:10 (find_package)


--
   *****************************
   * Running CMake for cc1-med *
   *****************************

Loading Zephyr default modules (Zephyr base).
-- Application: C:/work/dev/if-device-nrf53/cc1-med
-- CMake version: 3.21.0
-- Using NCS Toolchain 2.8.20241106.194216054162 for building. (C:/ncs/toolchains/2d382dcd92/cmake)
-- Found Python3: C:/ncs/toolchains/2d382dcd92/opt/bin/python.exe (found suitable version "3.12.4", minimum required is "3.8") found components: Interpreter
-- Cache files will be written to: C:/ncs/v2.8.0/zephyr/.cache
-- Zephyr version: 3.7.99 (C:/ncs/v2.8.0/zephyr)
-- Found west (found suitable version "1.2.0", minimum required is "0.14.0")
-- Board: cc1medv1_nrf5340_cpuapp
-- Found host-tools: zephyr 0.16.8 (C:/ncs/toolchains/2d382dcd92/opt/zephyr-sdk)
-- Found toolchain: zephyr 0.16.8 (C:/ncs/toolchains/2d382dcd92/opt/zephyr-sdk)
-- Found Dtc: C:/ncs/toolchains/2d382dcd92/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")

-- Found BOARD.dts: C:/work/dev/if-device-nrf53/boards/arm/cc1medv1_nrf5340/cc1medv1_nrf5340_cpuapp.dts
'label' is marked as deprecated in 'properties:' in C:/ncs/v2.8.0/zephyr/dts/bindings\audio\nordic,nrf-pdm.yaml for node /soc/peripheral@50000000/pdm@26000.
devicetree error: gpio controller <Node /soc/peripheral@50000000/i2c@9000/mcp23017@20 in 'C:/ncs/v2.8.0/zephyr/misc/empty_file.c'> for <Node /soc/peripheral@50000000/pdm@26000/en in 'C:/ncs/v2.8.0/zephyr/misc/empty_file.c'> lacks binding
CMake Error at C:/ncs/v2.8.0/zephyr/cmake/modules/dts.cmake:295 (execute_process):
  execute_process failed command indexes:

    1: "Child return code: 1"

Call Stack (most recent call first):
  C:/ncs/v2.8.0/zephyr/cmake/modules/zephyr_default.cmake:133 (include)
  C:/ncs/v2.8.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  C:/ncs/v2.8.0/zep-- Configuring incomplete, errors occurred!
hyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:8 (find_package)


CMake Error at cmake/modules/sysbuild_extensions.cmake:514 (message):
  CMake configure failed for Zephyr project: cc1-med

  Location: C:/work/dev/if-device-nrf53/cc1-med
Call Stack (most recent call first):
  cmake/modules/sysbuild_images.cmake:20 (ExternalZephyrProject_Cmake)
  cmake/modules/sysbuild_default.cmake:20 (include)
  C:/ncs/v2.8.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:75 (include)
  C:/ncs/v2.8.0/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  C:/ncs/v2.8.0/zephyr/share/sysbuild-package/cmake/SysbuildConfig.cmake:8 (include)
  template/CMakeLists.txt:10 (find_package)


-- Configuring incomplete, errors occurred!
See also "C:/work/dev/if-device-nrf53/cc1-med/build/CMakeFiles/CMakeOutput.log".
←[91mFATAL ERROR: command exited with status 1: 'C:\ncs\toolchains\2d382dcd92\opt\bin\cmake.EXE' -DWEST_PYTHON=C:/ncs/toolchains/2d382dcd92/opt/bin/python.exe '-BC:\work\dev\if-device-nrf53\cc1-med\build' -GNinja -DBOARD=cc1medv1_nrf5340_cpuapp '-DBOARD_ROOT=C:\work\dev\if-device-nrf53' '-SC:\ncs\v2.8.0\zephyr\share\sysbuild' '-DAPP_DIR:PATH=C:\work\dev\if-device-nrf53\cc1-med'

The DTS also seems to now fail (didn't with 2.6) with:

devicetree error: gpio controller <Node /soc/peripheral@50000000/i2c@9000/mcp23017@20 in 'C:/ncs/v2.8.0/zephyr/misc/empty_file.c'> for <Node /soc/peripheral@50000000/pdm@26000/en in 'C:/ncs/v2.8.0/zephyr/misc/empty_file.c'> lacks binding

 

Now I'm stuck....I'll look at the DTS thing tomorrow... I think I already knew that the DTS stuff had changed 2.6 to 2.8... but the migration guide doesn't reference it as far as I can see?

[BrianW]

I have posted this as a seperate case together with my 'fix', to check if the change I made is reasonable and reliable or not. Please see:

 nrf5340 Anomaly 159 Workaround : not a workaround? Breaks QPSI Flash use on NCS 2.8.0 

and give me an opinion.

[BrianW]

Blocking issue for now is that I cannot get the BLE HCI to start without this error:

[00:00:09.128,051] <wrn> app: NetBLE starting....
[00:00:09.134,338] <err> os: ***** USAGE FAULT *****
[00:00:09.139,923] <err> os:   Unaligned memory access
[00:00:09.145,751] <err> os: r0/a1:  0xc8c3b562  r1/a2:  0x601d8767  r2/a3:  0x9939399b
[00:00:09.154,418] <err> os: r3/a4:  0x0002dff5 r12/ip:  0x01010101 r14/lr:  0x0002e01d
[00:00:09.163,085] <err> os:  xpsr:  0x21000200
[00:00:09.168,304] <err> os: Faulting instruction address (r15/pc): 0x00073cde
[00:00:09.176,208] <err> os: >>> ZEPHYR FATAL ERROR 31: Unknown error on CPU 0
[00:00:09.184,082] <err> os: Current thread: 0x20007d60 (sysworkq)
[00:00:09.190,948] <err> os: Halting system

This is as soon as the application calls bt_enable(NULL); The wifi stack is not yet started BTW so shouldn't be causing an issue....

Call stack from the debugger:

arch_system_halt@0x000b59ee (c:\ncs\v2.8.0\zephyr\kernel\fatal.c:30)
k_sys_fatal_error_handler@0x000742d6 (c:\ncs\v2.8.0\zephyr\kernel\fatal.c:44)
z_fatal_error@0x00074378 (c:\ncs\v2.8.0\zephyr\kernel\fatal.c:119)
z_arm_fatal_error@0x0004b594 (c:\ncs\v2.8.0\zephyr\arch\arm\core\fatal.c:86)
z_arm_fault@0x0004bad8 (c:\ncs\v2.8.0\zephyr\arch\arm\core\cortex_m\fault.c:1157)
z_arm_usage_fault@0x0004bbbc (c:\ncs\v2.8.0\zephyr\arch\arm\core\cortex_m\fault_s.S:102)
<signal handler called>@0xfffffffd (Unknown Source:0)
rpmsg_send_offchannel_raw@0x00073cde (c:\ncs\v2.8.0\modules\lib\open-amp\open-amp\lib\rpmsg\rpmsg.c:131)
rpmsg_send@0x0002e01c (c:\ncs\v2.8.0\modules\lib\open-amp\open-amp\lib\include\openamp\rpmsg.h:204)
send@0x0002e01c (c:\ncs\v2.8.0\zephyr\subsys\ipc\ipc_service\backends\ipc_rpmsg_static_vrings.c:525)
hci_core_send_cmd@0x0004ef36 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:3012)
process_pending_cmd@0x0004ef36 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:4722)
bt_hci_cmd_send_sync@0x0004f016 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:422)
common_init@0x0004f0d0 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:3222)
hci_init@0x0004f0d0 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:3974)
bt_init@0x0004f0d0 (c:\ncs\v2.8.0\zephyr\subsys\bluetooth\host\hci_core.c:4187)
_netble_start@0x0001396e (c:\work\dev\if-device-nrf53\cc1-med\src\net\netble.c:1338)
_app_event_handler@0x0001396e (c:\work\dev\if-device-nrf53\cc1-med\src\net\netble.c:1446)
event_processor_fn@0x000312b8 (c:\ncs\v2.8.0\nrf\subsys\app_event_manager\app_event_manager.c:190)
work_queue_main@0x00074de8 (c:\ncs\v2.8.0\zephyr\kernel\work.c:688)

I have tried with the prj.conf setup reduced to the minimum:

# BT/BLE config

CONFIG_BT=y

CONFIG_BT_PERIPHERAL=y

 

but still the same error.

This happens whether using the CPU-NET image from previous 2.6 build, from the current  project's hci_ipc v2.8 build, or from the sample 'extended_adv/advertiser' built with v2.8.

hci_ipc.conf:

#
# Copyright (c) 2021 Nordic Semiconductor
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#

CONFIG_BT_CTLR_ADV_SET=2
CONFIG_BT_CTLR_ADV_EXT=y
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_MAX_CONN=1

CONFIG_BT_EXT_ADV=y
CONFIG_BT_EXT_ADV_MAX_ADV_SET=2
CONFIG_BT_DATA_LEN_UPDATE=y
CONFIG_BT_USER_DATA_LEN_UPDATE=y
# added to try to get 2.8 working
CONFIG_IPC_SERVICE=y
CONFIG_MBOX=y
CONFIG_BT_HCI_RAW=y
CONFIG_BT_BUF_CMD_TX_COUNT=10

(I copied in the lines from the advertiser sample to see if that helped, but no...)

The advertiser sample runs OK with the hci_ipc CPU-NET build from its project, but gives:

Bluetooth init failed (err -5)

with either the v2.6 project build CPU-NET, or the v2.8 main project CPU-NET...

(but no memory fault in either case)

Any ideas on what I can try?

[BrianW]

Running the debugger through the bt_enable code (not fun with the different in-line and macro encapsulations btw...) led me to think that the function hooks for the hci ipc driver are not initialised correctly.

The code has a lot of macros of type DT_HAS_DRIVERxxx, searching the forum found me this post

 nrf5340 custom board:  bt_hci_core: No HCI driver registered - Bluetooth init failed (err -19) 

Although I had a crash rather than a error code, it turned out to be a similar issue.

My DTS from 2.6 had in the chosen section the correct chosen name:

   chosen {

...

      zephyr,bt-hci = &ipc0;      // BAD

      zephyr,bt-hci-ipc = &ipc0;  // IRRELEVANT

...

};

 

Turns out that in NCS 2.8, you MUST have the chosen point to the specific bt_hci_ipc0 node:

        zephyr,bt-hci = &bt_hci_ipc0;

If you don't set it at all you get the init failed log, if you set it to &ipc0 you will get the core dump (thanks for that).

btw, the chosen name "zephyr,bt-hci-ipc" now appears to be ignored and can be omitted.

The definition of bt_hci_ipc0 is found in zephyr/dts/arm/nordic/nrf5340_cpuapp_ipc.dtsi:

ipc0: ipc0 {

    compatible = "zephyr,ipc-openamp-static-vrings";

    memory-region = <&sram0_shared>;

    mboxes = <&mbox 0>, <&mbox 1>;

    mbox-names = "tx", "rx";

    role = "host";

    status = "okay";

    bt_hci_ipc0: bt_hci_ipc0 {

        compatible = "zephyr,bt-hci-ipc";

        status = "okay";

    };

};

Why the chosen has to point to bt_hci_ipc0 and not ipc0 is completely opaque to me, I can find NO documentation about this in the nordic or zephyr docs. 

Also, just for fun, note that the CPUNET dts requires this chosen line:

        zephyr,bt-hci-ipc = &ipc0;

and cannot refer to bt_hci_ipc0 as this node does not exist for the CPUNET image build (in nrf5340_cpunet.dtsi there is just ipc0)...

On tyhe positive side, it seems that the NCS 2.6 CPUNET build image will work with the 2.8 CPUAPP image, although I haven't fully tested everything yet due to the need to clean up all the various changes made over the last 2 weeks trying to get this to run....

[willdean]

Please accept my gratitude for working through this!  I have just moved a project from 2.7 to 2.9 and had exactly this problem (among others).  Your two weeks (of undoubted misery) was a great benefit to me.

What a shambles this project is.  Not sure who the target audience is, but I don't feel like I'm it.