Enabling CONFIG_BOOTLOADER_MCUBOOT=y and/or CONFIG_BT=y breaks unrelated modules

[janb37896796]

We are attempting to add various modules to our project. For simplicity we've reduced our system to a toy setup closely mimicking the structure of our project and are trying to run the tensorflow lite micro module. 

The state of the code here: flite_main_proj compiles without error and runs the tflite hello world code on the nRF5340 dev kit board as expected. 

Problems arise when we try to enable either the bootloader or bluetooth configs, currently commented out in the file proj.conf

CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_BT=y

Either config produces the same errors on build. The following is from enabling MCUBOOT:

151/299] Building C object CMakeFiles/app.dir/main.c.obj
[152/299] Building CXX object modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/main_functions.cpp.obj
FAILED: modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/main_functions.cpp.obj 
ccache /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/arm-zephyr-eabi-g++ -DKERNEL -DMBEDTLS_CONFIG_FILE=\"mcuboot-mbedtls-cfg.h\" -DNRF5340_XXAA_APPLICATION -DNRF_SKIP_FICR_NS_COPY_TO_RAM -DUSE_PARTITION_MANAGER=1 -D__PROGRAM_START -D__ZEPHYR__=1 -I/home/user/Code/tf-ws/zephyr/include -I/home/user/Code/tf-ws/tflite_main_proj/app/build/mcuboot/zephyr/include/generated -I/home/user/Code/tf-ws/zephyr/soc/arm/nordic_nrf/nrf53 -I/home/user/Code/tf-ws/zephyr/soc/arm/nordic_nrf/common/. -I/home/user/Code/tf-ws/nrf/include -I/home/user/Code/tf-ws/nrf/tests/include -I/home/user/Code/tf-ws/tflite_sub_proj/inc -I/home/user/Code/tf-ws/modules/hal/cmsis/CMSIS/Core/Include -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx/drivers/include -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx/mdk -I/home/user/Code/tf-ws/zephyr/modules/hal_nordic/nrfx/. -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/zephyr/include -I/home/user/Code/tf-ws/modules/crypto/mbedtls/include -I/home/user/Code/tf-ws/modules/crypto/mbedtls/library -I/home/user/Code/tf-ws/zephyr/modules/mbedtls/configs -I/home/user/Code/tf-ws/zephyr/modules/mbedtls/include -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/.. -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/../include -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/../../zephyr/include -isystem /home/user/Code/tf-ws/zephyr/lib/libc/minimal/include -isystem /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/include -isystem /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/include-fixed -isystem /home/user/Code/tf-ws/nrfxlib/crypto/nrf_cc312_platform/include -fno-strict-aliasing -Os -fcheck-new -fno-exceptions -fno-rtti -imacros /home/user/Code/tf-ws/tflite_main_proj/app/build/mcuboot/zephyr/include/generated/autoconf.h -fno-common -g -gdwarf-4 -fdiagnostics-color=always -mcpu=cortex-m33 -mthumb -mabi=aapcs --sysroot=/home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/arm-zephyr-eabi -imacros /home/user/Code/tf-ws/zephyr/include/zephyr/toolchain/zephyr_stdint.h -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=/home/user/Code/tf-ws/bootloader/mcuboot/boot/zephyr=CMAKE_SOURCE_DIR -fmacro-prefix-map=/home/user/Code/tf-ws/zephyr=ZEPHYR_BASE -fmacro-prefix-map=/home/user/Code/tf-ws=WEST_TOPDIR -ffunction-sections -fdata-sections -nostdinc -MD -MT modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/main_functions.cpp.obj -MF modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/main_functions.cpp.obj.d -o modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/main_functions.cpp.obj -c /home/user/Code/tf-ws/tflite_sub_proj/lib/main_functions.cpp
/home/user/Code/tf-ws/tflite_sub_proj/lib/main_functions.cpp:19:10: fatal error: tensorflow/lite/micro/all_ops_resolver.h: No such file or directory
   19 | #include <tensorflow/lite/micro/all_ops_resolver.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[153/299] Linking C static library modules/nrf/lib/fatal_error/lib..__nrf__lib__fatal_error.a
[154/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/bignum_mod.c.obj
[155/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/camellia.c.obj
[156/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/ccm.c.obj
[157/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/chacha20.c.obj
[158/299] Linking C static library modules/nrf/drivers/hw_cc310/lib..__nrf__drivers__hw_cc310.a
[159/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/chachapoly.c.obj
[160/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/cipher.c.obj
[161/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/cipher_wrap.c.obj
[162/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/cmac.c.obj
[163/299] Building C object modules/mbedtls/CMakeFiles/modules__mbedtls.dir/home/user/Code/tf-ws/modules/crypto/mbedtls/library/code_share.c.obj
[164/299] Linking C static library modules/mcuboot/boot/bootutil/zephyr/libmcuboot_util.a
[165/299] Building CXX object modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/output_handler.cpp.obj
FAILED: modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/output_handler.cpp.obj 
ccache /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/arm-zephyr-eabi-g++ -DKERNEL -DMBEDTLS_CONFIG_FILE=\"mcuboot-mbedtls-cfg.h\" -DNRF5340_XXAA_APPLICATION -DNRF_SKIP_FICR_NS_COPY_TO_RAM -DUSE_PARTITION_MANAGER=1 -D__PROGRAM_START -D__ZEPHYR__=1 -I/home/user/Code/tf-ws/zephyr/include -I/home/user/Code/tf-ws/tflite_main_proj/app/build/mcuboot/zephyr/include/generated -I/home/user/Code/tf-ws/zephyr/soc/arm/nordic_nrf/nrf53 -I/home/user/Code/tf-ws/zephyr/soc/arm/nordic_nrf/common/. -I/home/user/Code/tf-ws/nrf/include -I/home/user/Code/tf-ws/nrf/tests/include -I/home/user/Code/tf-ws/tflite_sub_proj/inc -I/home/user/Code/tf-ws/modules/hal/cmsis/CMSIS/Core/Include -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx/drivers/include -I/home/user/Code/tf-ws/modules/hal/nordic/nrfx/mdk -I/home/user/Code/tf-ws/zephyr/modules/hal_nordic/nrfx/. -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/zephyr/include -I/home/user/Code/tf-ws/modules/crypto/mbedtls/include -I/home/user/Code/tf-ws/modules/crypto/mbedtls/library -I/home/user/Code/tf-ws/zephyr/modules/mbedtls/configs -I/home/user/Code/tf-ws/zephyr/modules/mbedtls/include -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/.. -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/../include -I/home/user/Code/tf-ws/bootloader/mcuboot/boot/bootutil/zephyr/../../zephyr/include -isystem /home/user/Code/tf-ws/zephyr/lib/libc/minimal/include -isystem /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/include -isystem /home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/include-fixed -isystem /home/user/Code/tf-ws/nrfxlib/crypto/nrf_cc312_platform/include -fno-strict-aliasing -Os -fcheck-new -fno-exceptions -fno-rtti -imacros /home/user/Code/tf-ws/tflite_main_proj/app/build/mcuboot/zephyr/include/generated/autoconf.h -fno-common -g -gdwarf-4 -fdiagnostics-color=always -mcpu=cortex-m33 -mthumb -mabi=aapcs --sysroot=/home/user/zephyr-sdk-0.16.1/arm-zephyr-eabi/arm-zephyr-eabi -imacros /home/user/Code/tf-ws/zephyr/include/zephyr/toolchain/zephyr_stdint.h -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=/home/user/Code/tf-ws/bootloader/mcuboot/boot/zephyr=CMAKE_SOURCE_DIR -fmacro-prefix-map=/home/user/Code/tf-ws/zephyr=ZEPHYR_BASE -fmacro-prefix-map=/home/user/Code/tf-ws=WEST_TOPDIR -ffunction-sections -fdata-sections -nostdinc -MD -MT modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/output_handler.cpp.obj -MF modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/output_handler.cpp.obj.d -o modules/tflite_sub_proj/lib/CMakeFiles/..__tflite_sub_proj__lib.dir/output_handler.cpp.obj -c /home/user/Code/tf-ws/tflite_sub_proj/lib/output_handler.cpp
In file included from /home/user/Code/tf-ws/tflite_sub_proj/lib/output_handler.cpp:17:
/home/user/Code/tf-ws/tflite_sub_proj/inc/output_handler.hpp:20:10: fatal error: tensorflow/lite/c/common.h: No such file or directory
   20 | #include <tensorflow/lite/c/common.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Based on this discussion Setting CONFIG_BOOTLOADER_MCUBOOT=y in prj.config on custom board prevents code working  an attempt was made to include the tf module in the child_image. We added the folder tflite_main_proj/app/child_image which includes a .conf duplicating our proj.conf. Unfortunately this has had no impact on our build and we are still seeing the same errors as before. 

This may be a similar issues to (Missing Forum Thread)

Any advice is appreciated!

[Sigurd Hellesvik]

Hi,

The common factor for CONFIG_BT (specifically for 5340) and CONFIG_BOOTLOADER_MCUBOOT is that they both enable multi imahe builds  which also enable the partition manager.

From your error message, I can not say exactly how this is related yet.

Is it an option for you to send me your minimal project to reproduce this error?
If not, I will give you some tips on how you can debug this yourself.

Regards,
Sigurd Hellesvik

[Guido]

Hi Sigurd,

Thank you for your support. I'm janb37896796 's co-worker, they asked me to follow the issue for this week.

The original post provides a github link with a minimal project. I'll re submit: https://github.com/janfari/tflite_main_proj
We are trying to mimic our current configuration where we use west to pull individual repositories with drivers and libraries. The readme explains the setup, it's a zephyr workspace (workspace folder, clone/west init and west update).
Under tflite_main_proj/app/prj.conf look for:

# # # # BAD
# CONFIG_BOOTLOADER_MCUBOOT=y
# CONFIG_BT=y 
# # # # BAD

Uncomment any of those to replicate.

We are aware of the multi image build and partition manager being enabled when using this with the ncs/zephy. We have used them for the project and they worked without problems until now.

To give you another angle: what's new is that we are looking on using zephyr's tensorflow-lite(tflite) module, which requires C++ to be enabled.
What we have tried:
- tflite and C++ work fine on their own with the nRF5340.
- Enabling tflite and C++ with our current project which resulted in compile time errors.
- After reconstructing the configs from scratchjanb37896796 was able to single out those 2 configs to be the breaking ones when in combination with tflite and C++.
- The original post shows that we tried some solutions that we found on the devzone.

I appreciate you taking the time to review this.

Best regards,
Guido

[janb37896796]

Hi Siguard,
Thank you for looking in to this issue. And thank you Guido for helping support.

Siguard, can you confirm that you've followed the step in the readme to update the .west/config file to point to the west.yml in tflite_main_proj? 

The contents of the config should look like this:

[manifest]
path = tflite_main_proj
file = west.yml

[zephyr]
base = zephyr

If you run `west update` after updating this it should pull down the folder tflite_sub_proj repo which contains the inc and src directories 

[Sigurd Hellesvik]

Ah, I have not. That is probably why I can not build.

I wll do this tomorrow and test the project then.
Sorry for the delay

[Sigurd Hellesvik]

I got some more unfortunate news.

I am traveling now, and I am somehow not able to run west update insde the folder on my current PC.
It is just stuck on getting sdk-nrf.

I will try this again on Monday when I am back in the office.

Normally, I would ask a collegua for help to reproduce this, but due to the summer vacation we are low on staff, so longer waiting times are expected

[Sigurd Hellesvik]

We are low on staffing due to the summer vacation.
Due to this, it takes longer than normal to answer cases.
This is especially true for reproduction of issues.

Thank you for your patience.

[Sigurd Hellesvik]

I have been able to build your project now.

Maybe the easiest solution could be to just remove

#include <tensorflow/lite/micro/all_ops_resolver.h>

from tflite_sub_proj/lib/main_functions.cpp

Do you need this file to be included in your project?

[janb37896796]

Hi Siguard,
Thank you for following up and attempting to build our project. The goal of our project here is to include and use the tensorflow module so removing it is pointless. 


We've been digging in to this problem more while you were away. The problem seems to be stemming from the fact that the three children images (mcuboot, hci_rpmsg hcirpmsg/b0n) are looking for the tflite code. We've tried two approaches here:
A) including the tflite configs in the three children image configs - this compiles but the code does not actually run (we would need to open a private ticket to share this particular failing code). From looking at the debugger, we get stuck in a bootloop in the mcuboot portion of the code. We can only assume this is due to the odd addition of the tflite configs. This seems like the wrong approach.


and B) attempting to exclude the tflite code from the children images. We've been unsuccessful in this. even with all function calls to tflite removed from our code, the children images are still searching for the tflite module as it is included in the west.yml as a module. Perhaps this approach of manually including tflite from the west.yml is the root of our issues but otherwise we have found to way to make the tflite module appear in the zephyr modules directory as it is not part of the "name-allowlist" in the nrf/west.yml. 


I understand many folks are on summer vacation, but perhaps there is a build system developer available we could escalate this to?

[Sigurd Hellesvik]

Looking into why the child images try to use tflite, I see that you are using sysbuild. Is that correct?
Do you have anything you need sysbuild for at this point?

janb37896796 said:

I understand many folks are on summer vacation, but perhaps there is a build system developer available we could escalate this to?

It is generally hard for me to get help from our developers with complex projects like this.
That is more for specific problems. For example: "Is sysbuild ready yet?" or "Why would sysbuild inherit configurations from the app?"

Alternatively, you have to ask sales if you want to escalate stuff.

[janb37896796]

We have tried both with and without sysbuild. With sysbuild the children inherit the configs without any additional configurations on our side. This compiles but causes the bootlooping issue. Without sysbuild we have tried manually adding the configs to the children, again this compiles but causes the bootlooping issue. We've also tried B) described above without sysbuild. Is there something to sysbuild you think may help or hurt the issue?

Two great specific question for the developers could be 1) Why does enabling the tflite configs not automatically import the module? 2) Why would the child images be looking for files in the tflite module? Having insight into those would help us have some more direction in debugging.

[Sigurd Hellesvik]

janb37896796 said:

Is there something to sysbuild you think may help or hurt the issue?

For now, I would suggest not using sysbuild, for two reasons:

1. I have heard that sysbuild does not yet fully support the Partition Manager yet, so there may be some issues there.
2. We in techincal support are not yet as familiar with sysbuild, as it is a new feature. We should in theory provide the same support, as sysbuild is being introduced, but it takes time to build experience in practice.

janb37896796 said:

1) Why does enabling the tflite configs not automatically import the module?

I do not know this one. Could sound like a design choice from whomever added tflite.

janb37896796 said:

2) Why would the child images be looking for files in the tflite module?

I do not know how sysbuild would handle this yet.

Without sysbuild the child images should not inherit the configuration from the application.
There are two possibilities to why this happens as I see it:

1. Either you have enabled tflite in your board files, as these apply to all child images as well as the app.
2. Part of the build system detects tflite configurations and apply it in a way that makes it effect child images as well.
   I know some other configurations in the application can make changes to child images, such as CONFIG_SB_SIGNING_KEY_FILE.

So that we are on the same page, can you tell me what you do to disable/enable sysbuild in your project?