nrf5340 MCUBOOT fails with multi image build at 1.8.0

[mnvcsUDT]

Hi all,

I observed an issue after upgrading to 1.8.0 and to Zephyr 2.7.0-ncs1 respectively. When having Zigbee (and NET core) enabled, all images are built as before. The external flash is used to store the secondary MCUBOOT image, as defined in the partition file. The app does not seem to start tough.

I could not find much in the release notes beside moving external flash definition to the overlay file.

*** Booting Zephyr OS build v2.7.0-ncs1  ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
I: Bootloader chainload address offset: 0xc000
[00:00:00.001,251] <err> os: ***** BUS FAULT *****
[00:00:00.001,251] <err> os:   Precise data bus error
[00:00:00.001,251] <err> os:   BFAR Address: 0x2007fc00
[00:00:00.001,281] <err> os: r0/a1:  0x2000006c  r1/a2:  0x00000000  r2/a3:  0x00000000
[00:00:00.001,281] <err> os: r3/a4:  0x000001ae r12/ip:  0x0000f800 r14/lr:  0x000182a7
[00:00:00.001,281] <err> os:  xpsr:  0x29100000
[00:00:00.001,281] <err> os: s[ 0]:  0x00000000  s[ 1]:  0x00000000  s[ 2]:  0x00000000  s[ 3]:  0x00000000
[00:00:00.001,312] <err> os: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[00:00:00.001,312] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00:00:00.001,312] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00:00:00.001,342] <err> os: fpscr:  0x20000ed8
[00:00:00.001,342] <err> os: Faulting instruction address (r15/pc): 0x0004bc22
[00:00:00.001,342] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:00.001,342] <err> os: Current thread: 0x20001490 (unknown)

 I attached the project (can be compiled with nrf5340dk_nrf5340_cpuapp board).  Any help is appriciated.

hello-world.zip

Regards,

Milan

[Simon]

Hello Milan,

This issue tricked me. I used quite long time debugging the mcuboot, thinking the issue happened there. But eventually I figured out that it actually successfully boots the main image. I ran the command addr2line -e zephyr.elf 0x4bdba (I got this log output: "<err> os: Faulting instruction address (r15/pc): 0x0004bdba") from <your project>/build/zephyr and got the following output:

/home/simon/ncs/modules/hal/libmetal/libmetal/lib/io.c:134

So I guess this happens during the Zephyr kernel initialization phase. I will continue to investigate this next week.

Best regards,

Simon

[mnvcsUDT]

Thank you Simon. Let me know if I can assist you.

Regards,

Milan

[Simon]

Yes, I will let you know. Sorry for the delay on this, I've been quite busy lately. I have asked about this internally and currently waiting for a reply. I will keep you updated.

Best regards,

Simon

[Simon]

I got some tips internally and did some investigations myself and got some progress.

During the zephyr kernel initialization process (before main), rpmsg_service_init() runs, which will eventually end up in /modules/hal/libmetal/libmetal/lib/io.c:134 and try to write to the address 0x2007fc00, which is a part of the pcd_sram, according to the ninja pm report:

+--------------------------------------------+
| 0x20000000: sram_primary (0x7e000 - 504kB) |
| 0x2007e000: pcd_sram (0x2000 - 8kB)        |
+--------------------------------------------+

The pcd_sram area is read-only after mcuboot has booted, see https://github.com/nrfconnect/sdk-mcuboot/blob/4e4e979c1998e6dad7e47011e6e076b861921645/boot/zephyr/main.c#L580. So that is why you're getting a bus fault

I looked in the file <sample>/build/zephyr/.config and saw these values

CONFIG_RPMSG_SERVICE_SHM_BASE_ADDRESS=0x20070000
CONFIG_RPMSG_SERVICE_SHM_SIZE=0x10000

which spans over the pcd_sram area.

I compared this to the peripheral_uart sample (with mcuboot enabled) and saw that it has the same values for RPMSG_SERVICE_SHM_BASE_ADDRESS and RPMSG_SERVICE_SHM_SIZE, but the ninja pm report looked like this:

+-----------------------------------------------+
| 0x20000000: sram_primary (0x6e000 - 440kB)    |
| 0x2006e000: pcd_sram (0x2000 - 8kB)           |
| 0x20070000: rpmsg_nrf53_sram (0x10000 - 64kB) |
+-----------------------------------------------+

and you can see it spans over the rpmsg_nrf53_sram (which is not present in your case) instead

I will continue the investigation on Monday and see how to fix this.

[Simon]

Seems like this is a Zigbee-specific issue, since the rpmsg_nrf53_sram area only will be included when Bluetooth is used (bluetooth specific config BT_RPMSG_NRF53 needs to be enabled):

• https://github.com/nrfconnect/sdk-nrf/blob/v1.8.0/subsys/partition_manager/CMakeLists.txt#L77
• https://github.com/nrfconnect/sdk-nrf/blob/v1.8.0/subsys/partition_manager/pm.yml.bt_rpmsg_nrf53#L4

As I'm not very familiar with Zigbee, I have asked the Zigbee experts internally and waiting for an answer. I will keep you updated.

Best regards,

Simon

[mnvcsUDT]

Thanks Simon, and also for the regular updates. 

[Simon]

A workaround for the issue is to set CONFIG_NRF53_UPGRADE_NETWORK_CORE=n in the prj.conf file (assuming you don't need to upgrade the Net Core application). I tested this with your sample and I no longer got the bus fault from modules/hal/libmetal/libmetal/lib/io.c. However, I had to comment out wdt_feed() and wdt_setup() in main.c, since it caused an Instruction bus error, but that's another issue.

We are working on a fix for the core of the issue and I will update this ticket when get more clarity.

You should be aware that we don't fully support Zigbee FOTA  on the nRF53 yet. If you have questions about this or other future/road map related questions, contact the Regional Sales Manager in your area, if you don't know who this is, send me a PM.

[Simon]

Have you tested the suggested workaround? Does it resolve the issue?

[mnvcsUDT]

Hi Simon,

Sorry just missed the notification of your previous answer.

I managed to get it to work with the workaround. Just needed a clean build. Thank you!!

We do not need Zigbee FOTA so that part is fine.

[mnvcsUDT]

Is there any other way, the watchdog to be enabled?