nRF Connect SDK v2.8.0 - Experimental option for multiple domains causes flashing issues on Thingy:53, resulting in bt_hci_driver failure

[zpm1066]

Hello,

I have a Thingy:53 sensor hub application (extended the Nordic Thingy53 Sensor Hub blog to include IAQ sensor)  built under NCS v2.7.0 that runs fine.

When rebuilding the same application under NCS v2.8.0 without any code or build changes, the build is successful with no Bluetooth driver issues. However, when flashed to Thingy:53 I get the following errors.

00> rtt:~$ [00:00:00.007,934] <inf> BH1749: BH1749 initialized
00> rtt:~$ *** Booting nRF Connect SDK v2.8.0-a2386bfc8401 ***
00> rtt:~$ *** Using Zephyr OS v3.7.99-0bc3393fb112 ***
00> rtt:~$ [00:00:00.035,430] <inf> main: main
00> rtt:~$ [00:00:00.035,491] <inf> main: Zephyr RTOS: 3.7.99
00> 
00> rtt:~$ [00:00:00.035,491] <inf> main: thingy53
00> rtt:~$ Starting Sensor Hub application
00> rtt:~$ [00:00:01.036,773] <err> bt_hci_driver: Endpoint binding failed with -11
00> rtt:~$ [00:00:01.036,804] <err> bt_hci_core: HCI driver open failed (-11)
00> rtt:~$ Bluetooth init failed (err -11)
00> rtt:~$ rtt:~$ 

In nRF Connect for VSCode, I noticed the following for NCS v.2.8.0 flashing stage.

"WARNING: Specifying runner options for multiple domains is experimental.
If problems are experienced, please specify a single domain using '--domain <domain>'" 

The above warning doesn't exist in previous NCS version.

Flashing build_vsc_ncs280/thingy53_ns to <jlink sn masked>
west flash -d /Volumes/QuarrkD/Current_Dev_IoT/2024_dev/thingy53_projects/thingy53_sensor_hub_v3/build_vsc_ncs280/thingy53_ns --skip-rebuild --dev-id 801017276

WARNING: Specifying runner options for multiple domains is experimental.
If problems are experienced, please specify a single domain using '--domain <domain>'
-- west flash: using runner nrfjprog
-- runners.nrfjprog: reset after flashing requested
-- runners.nrfjprog: Flashing file: /Volumes/QuarrkD/Current_Dev_IoT/2024_dev/thingy53_projects/thingy53_sensor_hub_v3/build_vsc_ncs280/thingy53_ns/merged_CPUNET.hex
[ #################### ] 2.565s | Erase file - Done erasing 
[ #################### ] 0.308s | Program file - Done programming 
[ #################### ] 0.302s | Verify file - Done verifying 
Applying pin reset.
-- runners.nrfjprog: Board with serial number <jlink sn masked> flashed successfully.
-- west flash: using runner nrfjprog
-- runners.nrfjprog: reset after flashing requested
-- runners.nrfjprog: Flashing file: /Volumes/QuarrkD/Current_Dev_IoT/2024_dev/thingy53_projects/thingy53_sensor_hub_v3/build_vsc_ncs280/thingy53_ns/merged.hex
[ #################### ] 11.471s | Erase file - Done erasing 
[ #################### ] 2.648s | Program file - Done programming 
[ #################### ] 2.734s | Verify file - Done verifying 
Applying pin reset.
-- runners.nrfjprog: Board with serial number <jlink sn masked> flashed successfully.
* Terminal will be reused by tasks, press any key to close it.

The above flashing with the experimental multiple domains appears to cause Bluetooth driver issues, possibly as there is a build step for 'empty_net_core".

To verify that the application domain thingy53_sensor_hub_v3 built correctly for the NCS v2.8.0 build, after flashing the application with NCS v2.7.0 build, I flashed only the NCS v2.8.0 thingy53_sensor_hub_v3 domain from a command line session.

[16/18] cd /Volumes/QuarrkD/Current_Dev_IoT/2024_dev/thingy...toolchains/15b490767d/Cellar/cmake/3.21.0/bin/cmake -E true

-- west flash: using runner nrfjprog

-- runners.nrfjprog: reset after flashing requested

Using board <masked>

-- runners.nrfjprog: Flashing file: /Volumes/QuarrkD/Current_Dev_IoT/2024_dev/thingy53_projects/thingy53_sensor_hub_v3/build_vsc_ncs280/thingy53_ns/merged.hex

[ #################### ] 11.087s | Erase file - Done erasing

[ #################### ]  2.667s | Program file - Done programming 

[ #################### ]  2.748s | Verify file - Done verifying

Applying pin reset.

-- runners.nrfjprog: Board with serial number <masked> flashed successfully.

After above, the Thingy:53 application ran fine. There was no bt_hci_driver endpoint binding failure and Bluetooth initialized correctly, as shown below.

00> rtt:~$ [00:00:00.007,873] <inf> BH1749: BH1749 initialized
00> rtt:~$ *** Booting nRF Connect SDK v2.8.0-a2386bfc8401 ***
00> rtt:~$ *** Using Zephyr OS v3.7.99-0bc3393fb112 ***
00> rtt:~$ [00:00:00.035,369] <inf> main: main
00> rtt:~$ [00:00:00.035,430] <inf> main: Zephyr RTOS: 3.7.99
00> 
00> rtt:~$ [00:00:00.035,430] <inf> main: thingy53
00> rtt:~$ Starting Sensor Hub application
00> rtt:~$ [00:00:00.051,422] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
00> rtt:~$ [00:00:00.051,452] <inf> bt_hci_core: HW Variant: nRF53x (0x0003)
00> rtt:~$ [00:00:00.051,483] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 214.51162 Build 1926957230
00> rtt:~$ [00:00:00.053,710] <inf> bt_hci_core: Identity: C3:A6:C8:C4:25:42 (random)
00> rtt:~$ [00:00:00.053,741] <inf> bt_hci_core: HCI: version 5.4 (0x0d) revision 0x21fb, manufacturer 0x0059
00> rtt:~$ [00:00:00.053,771] <inf> bt_hci_core: LMP: version 5.4 (0x0d) subver 0x21fb
00> rtt:~$ Bluetooth initialized
00> rtt:~$ Advertising successfully started
00> rtt:~$ rtt:~$ 

In the nRF Connect SDK for VSCode IDE, How can I disable the experimental runner options for multiple domain introduced in NCS v2.8.0 so that it is the same as in NCS v2.7.0?

If the above is not possible, What is the recommended fix in this instance for NCS v2.8.0 when building with nRF Connect SDK?  Thank you.

Regards,
Ravi

[Simonr]

Hi Ravi

I think this is similar to the issue discussed in this ticket, where sysbuild is to blame as there are some differences in how sysbuild and the old "basic" west build does things during build. Please read the explanation by Marte and see if that gets you anywhere. It's also worth noting that the Thingy:53 sensor hub sample hasn't been maintained/tested since NCS 2.4.0, so if you're using it with newer NCS versions, make sure you follow the migration notes onto newer NCS versions, especially the ones for 2.7.0 and 2.8.0 where sysbuild was introduced, since there are quite a few changes. Especially "Migrating from multi-image builds to sysbuild should be useful here: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_sysbuild.html 

Best regards,

Simon

[zpm1066]

Hi Simon,

Thank you for picking up the issue. It's much appreciated. I reviewed the ticket that you linked. 

Unfortunately, there was no solution presented to the "experimental runner options for multiple domain" issue.

A couple of folks in the ticket reported the same issue with NCS v2.70 sysbuild. 

I can confirm that the "Thingy:53 sensor hub" application, built as NCS v2.8.0 (and v2.7.0) "no-sysbuild" runs fine with bluetooth connectivity. There are no issues at all.

For NCS v2.8.0 (and 2.7.0) sysbuild, the  "Thingy:53 sensor hub" application generates the same warning as per the ticket you linked.  Unlike that ticket, I'm not using nRF700x WiFi but BT.

"WARNING: Specifying runner options for multiple domains is experimental.
If problems are experienced, please specify a single domain using '--domain <domain>'" 

and  bluetooth driver failure.

00> rtt:~$ [00:00:01.036,773] <err> bt_hci_driver: Endpoint binding failed with -11
00> rtt:~$ [00:00:01.036,804] <err> bt_hci_core: HCI driver open failed (-11)

As per my earlier message, for now, I do have a "workaround".

I’m able to specify a single domain using the "--domain" option and use the "sysbuild" binary generated by “nRF Connect for VSCode” IDE from a command line session. 

west flash -d build_vsc_ncs280/thingy53_ns --domain thingy53_sensor_hub_v3

I had to update the NCS v2.8.0 nRF5340 application domain only and retain the NCS v2.7.0 nRF5340 Net core domain.

For some reason, when building the application with NCS v2.8.0 (or NCS 2.7.0)  “sysbuild”, the build creates an empty Net Core image. I don't if any of the BT code is in the Net Core. If it is, that may explain the BT failure. 

Here are the build tree for both the "sysbuild".

For the "workaround", the NCS v2.8.0 nRF5340 application domain only used "thingy53_ns/merged.hex" from the "sysbuild" below.

I don't understand why sysbuild generates an empty_net_core.

quark14@quark14 build_vsc_ncs280 % tree -L 2

.

├── thingy53_ns

│  ├── CMakeCache.txt

│ ├── CMakeFiles

│ ├── GENERATED_NON_SECURE_SIGN_KEY_PRIVATE.pem

│ ├── Kconfig

│   ├── _sysbuild

│   ├── b0n

│   ├── b0n_container.hex

│   ├── b0n_sysbuild_cache.txt

│   ├── b0n_sysbuild_cache.txt.tmp

│   ├── build.ninja

│   ├── build_info.yml

│   ├── build_info.yml.bak

│   ├── cmake_install.cmake

│   ├── dfu_application.zip

│   ├── dfu_application.zip_manifest.json

│   ├── domains.yaml

│   ├── empty_net_core

│   ├── empty_net_core_sysbuild_cache.txt

│   ├── empty_net_core_sysbuild_cache.txt.tmp

│   ├── mcuboot

│   ├── mcuboot_sysbuild_cache.txt

│   ├── mcuboot_sysbuild_cache.txt.tmp

│   ├── merged.hex

│   ├── merged_CPUNET.hex

│   ├── modules

│   ├── net_provision.hex

│   ├── nrf

│   ├── partitions.yml

│   ├── partitions_CPUNET.yml

│   ├── pm.config

│   ├── pm_CPUNET.config

│   ├── regions.yml

│   ├── regions_CPUNET.yml

│   ├── signed_by_b0_empty_net_core.bin

│   ├── signed_by_b0_empty_net_core.hex

│   ├── signed_by_mcuboot_and_b0_empty_net_core.bin

│   ├── signed_by_mcuboot_and_b0_empty_net_core.hex

│   ├── sysbuild_modules.txt

│   ├── thingy53_sensor_hub_v3

│   ├── thingy53_sensor_hub_v3_sysbuild_cache.txt

│   ├── thingy53_sensor_hub_v3_sysbuild_cache.txt.tmp

│   ├── zephyr

│   ├── zephyr_modules.txt

│   └── zephyr_settings.txt

and "no-sysbuild":

└── thingy53_ns_nosysbuild

    ├── CMakeCache.txt

    ├── CMakeFiles

    ├── Kconfig

    ├── app

    ├── build.ninja

    ├── build_info.yml

    ├── build_info.yml.bak

    ├── cmake_install.cmake

    ├── compile_commands.json

    ├── generated

    ├── hci_ipc

    ├── image_preload.cmake

    ├── mcuboot

    ├── modules

    ├── partitions.yml

    ├── pm.config

    ├── regions.yml

    ├── sysbuild_modules.txt

    ├── tfm

    ├── zephyr

    ├── zephyr_modules.txt

    └── zephyr_settings.txt

I have migrated the majority of our NCS v2.7.0 and earlier multi-image applications to v2.8.0 without any issues. I did again review the ""Migrating from multi-image builds to sysbuild" docs again but didn't find anything odd in the configuration for the Thingy53 build.

Could you please review to see if there is anything here that requires to be moved to sysbuild.conf or any other sysbuild related configuration? Thank you.

Here s the prj.conf.

# Enable Bluetooth
CONFIG_BT=y
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_DEVICE_NAME="Thingy53_sensor_hub"

# Enable buttons and LEDs
CONFIG_DK_LIBRARY=y

# Enable logs and set stacks
CONFIG_LOG=y
CONFIG_PRINTK=y
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

# Configure Thingy:53 sensors
CONFIG_I2C=y
CONFIG_SENSOR=y
CONFIG_SENSOR_INFO=y
CONFIG_BH1749=y

# Disable Zephyr BME680 driver and use Bosch BSEC driver
CONFIG_BME680=n
CONFIG_BME68X_IAQ=y
CONFIG_SETTINGS_NONE=y

# configure ADC for battery check
CONFIG_ADC=y

# Settings - Begin - Used to store real-time device configuration to flash.
# Required for BME68X_IAQ
CONFIG_SETTINGS=y
CONFIG_SETTINGS_FCB=y
CONFIG_FCB=y
# These have been disabled but were used in the BME688X_IAQ sample
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_STREAM_FLASH=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_SCANF=y
CONFIG_ASSERT=y
CONFIG_REBOOT=y
CONFIG_FPU=y
# Settings - End

Regards,
Ravi

[Simonr]

Thank you for your thorough feedback Ravi. I will need to ask the developers about this behavior, and what the correct way to add runner options with sysbuild will be, as I can't find this in our documentation as is.

One thing you can try is to add the "domain" for the NET core separately int he command to make sure the NET core image is in fact build with the Bluetooth driver. Sysbuild is still new, and we're working on improving it continuously.

Best regards,

Simon

[zpm1066]

Hi Simon,

Thank you. You’re most welcome. This issue is most likely affecting a good portion of the customers trying to use sysbuild. It’s completely understandable that there are a few “hiccups“ on the way for a robust sysbuild implementation. It certainly is a much better approach than the old “child-parent image” approach.

btw - I did add a note in the ticket you linked so that folks there are aware of this issue’s progress.

How do I specify adding the domain for NET core as part of the build process from within nRF Connect for VSCode or the command line so that BT driver gets included in the build? Thank you.

Regards,
Ravi

[zpm1066]

Hi Simon,

I expect the following sysbuild configuration addition is the solution.

SB_CONFIG_NETCORE_HCI_IPC

The link below states that sysbuild Kconfig options are also available for nRF53-based devices.
https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/app_dev/config_and_build/sysbuild/sysbuild_images.html

it looks like currently sysbuild is defaulting to using the following sysbuild configuration and generating an empty Net Core Image.

SB_CONFIG_NETCORE_EMPTY

i’ll try out the sysbuild configuration addition later today and update you. 

Regards’
Ravi