nrf9160 always recovers when updating applications

[llly]

Hello,

I used two samples, application_update and SLM, to test and try updating the app from application_update to SLM. I first downloaded the firmware of application_update v1 version to 9160, and the LED 1 on 9160dk lit up to indicate that the v1 version was being used. Then, I renamed the bin file of SLM to app_2RF9160dk_2.50_v2. bin and uploaded it to the server. Next, I pressed the button 1, 9160dk to download the v2 firmware normally (but in reality, it was SLM). After downloading, I pressed the button 1 again to update the app to SLM, 9160dk also successfully ran SLM,but after restarting, the bootloader restored the firmware to application_update, even though I had already called boot_writd_img_confirmed in the main function of SLM. I also try to call boot_is_img_confirmed in the main function of slm to check whether the firmware is confirmed. The return value is 0, indicating that it is normal.

When I tested with application_update, everything was fine. 9160 can switch between v1 and v2 versions without automatically restoring to the previous version, but replacing v2's bin with SLM is not possible. I don't know which step I overlooked or made wrong?

Thank you.

[JyriLehtinen]

The SLM-application is most likely built with a different partition table, as I assume you're not using same pm_static.yml (at least for the relevant parts).

Updating the application does not update the bootloader, so the bootloader in your DK is built with the partitions defined in your application_update build. This defines which address the bootloader will check when determining whether to revert the image or not.
Now, your SLM build probably has a different partition table, so when you call boot_writd_img_confirmed() it writes to a different address, defined by your partition table in that build.

[llly]

Are you referring to the differences here? If so, how should we modify the definition of this section？

[llly]

This is about the error message after using pm_static.yml

-- Found partition manager static configuration: F:/nrf9160_app/serial_lte_modem/pm_static.yml
Partition 'mcuboot' is not included in the dynamic resolving since it is statically defined.
Partition 'mcuboot_pad' is not included in the dynamic resolving since it is statically defined.
Partition 'mcuboot_primary' is not included in the dynamic resolving since it is statically defined.
Partition 'mcuboot_primary_app' is not included in the dynamic resolving since it is statically defined.
Partition 'mcuboot_secondary' is not included in the dynamic resolving since it is statically defined.
Partition 'mcuboot_sram' is not included in the dynamic resolving since it is statically defined.
Partition 'nrf_modem_lib_ctrl' is not included in the dynamic resolving since it is statically defined.
Partition 'nrf_modem_lib_rx' is not included in the dynamic resolving since it is statically defined.
Partition 'nrf_modem_lib_sram' is not included in the dynamic resolving since it is statically defined.
Partition 'nrf_modem_lib_tx' is not included in the dynamic resolving since it is statically defined.
Partition 'sram_nonsecure' is not included in the dynamic resolving since it is statically defined.
Partition 'sram_secure' is not included in the dynamic resolving since it is statically defined.
Partition 'tfm' is not included in the dynamic resolving since it is statically defined.
Partition 'tfm_nonsecure' is not included in the dynamic resolving since it is statically defined.
Partition 'tfm_secure' is not included in the dynamic resolving since it is statically defined.
Partition 'tfm_sram' is not included in the dynamic resolving since it is statically defined.
Partition manager failed: End of last partition is after last valid address
Failed to partition region sram_primary, size of region: 262144
Partition Configuration:
mcuboot_sram:
size: 32768
nrf_modem_lib_ctrl:
placement:
after:
- tfm_sram
- start
size: 1256
nrf_modem_lib_rx:
placement:
after:
- nrf_modem_lib_tx
size: 8192
nrf_modem_lib_sram:
size: 17768
nrf_modem_lib_trace:
placement: {}
size: 16384
nrf_modem_lib_tx:
placement:
after:
- nrf_modem_lib_ctrl
size: 8320
sram_nonsecure:
size: 229376
sram_primary:
size: 211608
sram_secure:
size: 32768
tfm_sram:
placement:
after:
- start
size: 32768

CMake Error at F:/nordic/ncs_v2.6.1/nrf/cmake/partition_manager.cmake:331 (message):
Partition Manager failed, aborting. Command:
C:/ncs/toolchains/cf2149caf2/opt/bin/python.exe;F:/nordic/ncs_v2.6.1/nrf/scripts/partition_manager.py;--input-files;F:/nrf9160_app/serial_lte_modem/build/mcuboot/zephyr/include/generated/pm.yml;F:/nrf9160_app/serial_lte_modem/build/zephyr/include/generated/pm.yml;F:/nrf9160_app/serial_lte_modem/build/modules/nrf/subsys/partition_manager/pm.yml.settings;F:/nrf9160_app/serial_lte_modem/build/modules/nrf/subsys/partition_manager/pm.yml.libmodem;F:/nrf9160_app/serial_lte_modem/build/modules/nrf/subsys/partition_manager/pm.yml.tfm;F:/nrf9160_app/serial_lte_modem/build/modules/nrf/subsys/partition_manager/pm.yml.trustzone;F:/nrf9160_app/serial_lte_modem/build/modules/nrf/subsys/partition_manager/pm.yml.mcuboot;--regions;sram_primary;otp;flash_primary;--output-partitions;F:/nrf9160_app/serial_lte_modem/build/partitions.yml;--output-regions;F:/nrf9160_app/serial_lte_modem/build/regions.yml;--static-config;F:/nrf9160_app/serial_lte_modem/pm_static.yml;--sram_primary-size;0x40000;--sram_primary-base-address;0x20000000;--sram_primary-placement-strategy;complex;--sram_primary-dynamic-partition;sram_primary;--otp-size;756;--otp-base-address;0xff8108;--otp-placement-strategy;start_to_end;--flash_primary-size;0x100000;--flash_primary-base-address;0x0;--flash_primary-placement-strategy;complex;--flash_primary-device;flash_controller;--flash_primary-default-driver-kconfig;CONFIG_SOC_FLASH_NRF
Call Stack (most recent call first):
F:/nordic/ncs_v2.6.1/zephyr/cmake/modules/kernel.cmake:248 (include)
F:/nordic/ncs_v2.6.1/zephyr/cmake/modules/zephyr_default.cmake:138 (include)
F:/nordic/ncs_v2.6.1/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
F:/nordic/ncs_v2.6.1/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:97 (include_boilerplate)
CMakeLists.txt:9 (find_package)

-- Configuring incomplete, errors occurred!
See also "F:/nrf9160_app/serial_lte_modem/build/CMakeFiles/CMakeOutput.log".
See also "F:/nrf9160_app/serial_lte_modem/build/CMakeFiles/CMakeError.log".
FAILED: build.ninja
C:\ncs\toolchains\cf2149caf2\opt\bin\cmake.exe --regenerate-during-build -SF:\nrf9160_app\serial_lte_modem -BF:\nrf9160_app\serial_lte_modem\build
ninja: error: rebuilding 'build.ninja': subcommand failed
FATAL ERROR: command exited with status 1: 'C:\ncs\toolchains\cf2149caf2\opt\bin\cmake.EXE' --build 'f:\nrf9160_app\serial_lte_modem\build'

This is the content in CMakeError. log

Compiling the C compiler identification source file "CMakeCCompilerId.c" failed.
Compiler: C:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc.exe
Build flags:
Id flags:

The output was:
1
c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/lib\libc.a(lib_a-exit.o): in function `exit':
exit.c:(.text.exit+0x34): undefined reference to `_exit'
collect2.exe: error: ld returned 1 exit status

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: C:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc.exe
Build flags:
Id flags:

The output was:
1
c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/lib\libc.a(lib_a-exit.o): in function `exit':
exit.c:(.text.exit+0x34): undefined reference to `_exit'
collect2.exe: error: ld returned 1 exit status

This is the content in CMakeOutput.log

The target system is: Generic - 3.5.99 - arm
The host system is: Windows - 10.0.19045 - AMD64
Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: C:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc.exe
Build flags:
Id flags: -c

The output was:
0

Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "CMakeCCompilerId.o"

The C compiler identification is GNU, found in "F:/nrf9160_app/serial_lte_modem/build/CMakeFiles/3.21.0/CompilerIdC/CMakeCCompilerId.o"

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: C:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc.exe
Build flags:
Id flags: -c

The output was:
0

Compilation of the CXX compiler identification source "CMakeCXXCompilerId.cpp" produced "CMakeCXXCompilerId.o"

The CXX compiler identification is GNU, found in "F:/nrf9160_app/serial_lte_modem/build/CMakeFiles/3.21.0/CompilerIdCXX/CMakeCXXCompilerId.o"

Checking whether the ASM compiler is GNU using "--version" matched "(GNU assembler)|(GCC)|(Free Software Foundation)":
arm-zephyr-eabi-gcc.exe (Zephyr SDK 0.16.5) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I don't know if these are helpful in solving this problem?

[JyriLehtinen]

(By the way, you can use the formatting tools in your posts to help the reader. For example, the build output could have been a code block)

Based on the output, the build fails at the partition_manager.py. The magic line is

Partition manager failed: End of last partition is after last valid address
Failed to partition region sram_primary, size of region: 262144

The error is most likely in your pm_static.yml.
Actually, you might as well share the file here, wouldn't hurt.

[llly]

Sorry, I will pay attention to these details in the future.

Fullscreen 1614.pm_static.yml.txt Download

EMPTY_0:
  address: 0xc000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x4000
app:
  address: 0x18000
  end_address: 0x88000
  region: flash_primary
  size: 0x70000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x88000
  orig_span: &id001
  - mcuboot_pad
  - app
  - tfm
  region: flash_primary
  sharers: 0x1
  size: 0x78000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x88000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0x77e00
  span: *id002
mcuboot_secondary:
  address: 0x88000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x78000
  
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id005
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x88000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

This is what I copied directly from the app_update sample without any changes. Can you help me check if there are any issues? Thank you.

[JyriLehtinen]

I just tested opening a fresh serial_lte_modem and application_update examples (SDK 2.6.1), and generated the build configurations for nRF9160-dk.

I copied the the partitions.yml from application_update as pm_static.yml for the slm and did a pristine build. No errors. Below is my pm_static.yml. Seems to be identical to yours, though at lines 36-38 the order is different (Windows and Linux output the items in different orders, but i don't think it matters, compiles with both).

EMPTY_0:
  address: 0xc000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x4000
app:
  address: 0x18000
  end_address: 0x88000
  region: flash_primary
  size: 0x70000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x88000
  orig_span: &id001
  - tfm
  - app
  - mcuboot_pad
  region: flash_primary
  sharers: 0x1
  size: 0x78000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x88000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0x77e00
  span: *id002
mcuboot_secondary:
  address: 0x88000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x78000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id005
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x88000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

So your error runs deeper than pm_static, there's something strange with your build process. I noticed some weird stuff while fiddling with the lwm2m_client example, Windows-built binaries had something messed up with the flash maps, the example, mcuboot complained that imagemagic was bad and application failed at "boot_write_img_confirmed()" with IO error (-5). The same source file, SDK version and build configuration worked fine on Linux.

At some point it started working again on windows though I don't know why, the generated .config and partition.yml files were identical.

I'm unable to replicate it anymore so it's hard to say at this point.

[llly]

Thank you for your testing. When I have time, I will download NCS again to ensure that there are no issues with the SLM itself, and then see if I can find any more clues

[JyriLehtinen]

By the way, are the samples from the same SDK version as you're currently using? For example, you initially generated them with v2.5.0 and have now switched to v2.6.1? Sometimes the libraries have changed and require updates in the source code.

[llly]

I was just starting to learn about 9160 and have been using version 2.6.1 since the beginning

[dejans]

Hi,

llly said:

When I have time, I will download NCS again to ensure that there are no issues with the SLM itself, and then see if I can find any more clues

Fresh installation of NCS might help. You can follow NCS installation guide. 

Best regards,
Dejan

[llly]

Hi dejan,

I reinstalled NCS for testing, and everything went smoothly this time. If you are upgrading from app_update to SLM, you need to copy the pm_static.yml file of app_update to SLM to ensure that the partitions are the same. If you are upgrading from the old SLM to the new SLM, there is no need to adjust the partitions. Thank you for your help