Thingy 91: FOTA reverting unless flashed merge.hex with JTAG

[Nikola Markovic]

Hi,

We have several boards on hand which fall into two categories:

Category A: Our application has been programmed with JTAG in the past

Category B: Out application has never been programmed with JTAG onto the boards.

Category A boards are all labeled version 1.2.0, Out of the Category B boards some are 1.2.0 and some are labeled 1.4.0

From the boards that have been flashed with JTAG (Category A) in the past, the following process works:

1) Flashing our application app_signed.hex via USB MCUBoot process and NRF programmer.

2) FOTA Update based on http_application_update sample.

3) After FOTA update, the newly downloaded application boots and runs permanently on the board.

From the boards that have never been flashed with JTAG, the step 3 fails. The board reverts the the original app flashed with MCUBoot. There is no indication that the newly flashed application even attempts to boot. It just rolls back to whatever was flashed with MCUBoot.

If we flash our application onto the Category B board, the board can go through steps 1, 2 and 3 without a problem.

We have also tried using the original http_application_update (with only modified URL) and observed the same behavior. The newly flashed application never boots.

We are aware that we need to confirm the boot_write_img_confirmed(), and that part works well on Category A boards.

The one thing that we concluded is that the JTAG flashing procedure of merged.hex flashes the MCU Boot portion of the board and now MCUBoot becomes "compatible". 

The questions that I have are:

1) Does the described process require MCUBoot update? 

2) If the process require MCUBoot update, can MCUBoot be flashed via the "MCUBoot" process or does it require JTAG?

3) If not, could you please point me to the document that explains the issue that we have ran into?

I apologize if this has been documented but I did not find any information that explains the issue that we am seeing.

I' pasting the prj.conf if it is of any use, but I am pretty confident that the same process fails without our application and just with http_application_update.

Thanks,

Nikola

# sys_rboot after OTA
CONFIG_REBOOT=y

# Sensors
CONFIG_ADXL362_TRIGGER_GLOBAL_THREAD=y
CONFIG_ADXL362_ABS_REF_MODE=1
CONFIG_ADXL362_ACTIVITY_THRESHOLD=200
CONFIG_ADXL362_INACTIVITY_THRESHOLD=200
CONFIG_ADXL362_INTERRUPT_MODE=1

# Library for buttons and LEDs
CONFIG_DK_LIBRARY=y
CONFIG_DK_LIBRARY_INVERT_LEDS=n

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=n
CONFIG_NET_SOCKETS_OFFLOAD=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y

# BSD library
CONFIG_BSD_LIBRARY=y
CONFIG_BSD_LIBRARY_SYS_INIT=y
CONFIG_BSD_LIBRARY_TRACE_ENABLED=n

# BSD library
CONFIG_BSD_LIBRARY=y
CONFIG_MODEM_KEY_MGMT=y

# LTE link control
CONFIG_LTE_LINK_CONTROL=y
CONFIG_LTE_AUTO_INIT_AND_CONNECT=n

# AT Host
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_AT_HOST_LIBRARY=y

# CONFIG_CJSON_LIB=n
CONFIG_NEWLIB_LIBC=y

# Added by W-ADS
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_LIB_TLS=y

# FOTA requirements BEGIN
# ======================

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_FLASH=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# FOTA library
CONFIG_FOTA_DOWNLOAD=y

# Download client
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_TLS=y
CONFIG_DOWNLOAD_CLIENT_SOCK_TIMEOUT_MS=10000
CONFIG_DOWNLOAD_CLIENT_MAX_TLS_FRAGMENT_SIZE=1024

# add debugging to fota log
CONFIG_LOG=y
CONFIG_LOG_IMMEDIATE=y
CONFIG_DOWNLOAD_CLIENT_LOG_LEVEL_DBG=y

# DFU Target
CONFIG_DFU_TARGET=y

# Application Upgrade support
CONFIG_BOOTLOADER_MCUBOOT=y

# ======================
# FOTA requirements END

# Main thread
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_HEAP_MEM_POOL_SIZE=4096

[Didrik Rokhaug]

Hi.

Based on your description, it sounds to me like your application uses a different flash partitioning scheme than the asset_tracker, which is what is running on the Thingy:91 by default.

If this hypothesis is correct, you are correct in identifying boot_write_img_confirmed() as the failing part.

What happens, is the bootloader already on your chip, and your application have two different opinions about where the image metadata is stored in the flash. Your application therefore writes that it is OK to the "wrong" address. E.g. if your application is writing to address 0x80000, but the bootloader is checking address 0x75000, the bootloader will not see that the application has set the OK flag. It will then assume that the image is not OK, and revert.

The reason for this happening, is that the flash partitioning is decided at build time, depending on your application's need, by the Partition Manager script.

The way you get around this problem, is to use a give the Partition Manager a static configuration file: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/1.3.0/nrf/scripts/partition_manager/partition_manager.html#static-configuration

While letting the Partition Manager create the partition map dynamically is fine during development, you will eventually want to use a static partion configuration of your own, to make sure that future versions of your application uses the same partition map.

You can find the partition map used by the asset_tracker on the Thingy:91 here: https://github.com/nrfconnect/sdk-nrf/blob/master/applications/asset_tracker/configuration/thingy91_nrf9160ns/pm_static.yml

If you add that file to your project (either copy the file so it is next to the CMakeLists.txt file in your project, or see the asset_tracker's CmakeLists.txt file to see how you can specify the path), you should be able to "upgrade" from the asset_tracker to your project.

Best regards,

Didrik

[Nikola Markovic]

"If this hypothesis is correct, you are correct in identifying boot_write_img_confirmed() as the failing part."

No, I never said that.

I think I used too words to describe the problem. Let's eliminate my application from the picture because we were able to reproduce the issue with involving http_application_update sample only.

To reproduce, get a brand new board from the factory that's never been flashed with jtag.

o Create 3 versions of the http_application_update so that oyu can differentiate which one booted up. Versions 0, 1 and 2. Put version 2 on the HTTP server.

o Use MCUBoot to upload version 1 app_signex.hex

o Version 1 boots.

o Reboot the board and verify that version 1 boots.

o Trigger the FOTA for app_update.bin version 2

o Once FOTA completes, reboot.

o Version 1 boots. NoK. Expected version 2 to boot. Version 2 does not even ATTEMPT to boot to get a chance to execute boot_write_img_confirmed

o Reboot the board again. Version 1 boots. NoK.

Now flash version 0 meged.hex onto the board with JTAG.

Verify that version 0 boots.

Repeat all of the above steps version 1 and 2. The NoK bold steps become OK. Version 2 boots and reboots with Version 2 and everything works as expected.

[Didrik Rokhaug]

OK, I'll try to reproduce it on my end.

Meanwhile, can you send me the application log from one of the failing devices?

[Nikola Markovic]

Hi Didrik,

There are really no useful logs because the application doesn't even boot. Normally after FOTA there's about a 10 second delay until the app is copied by the bootloader. This delay doesn't occur in case of failure and the original app boots.

We actually took some time to understand what you posted and concluded the following:

The boards distributed by Nordic come with Asset Tracker's static partitioning, so any application that doesn't have Asset Tracker's static partitioning built-in will not be able to update itself using the above explained process (no JTAG). That includes your http_applciation_update sample, so I would suggest that you also update it to use asst tracker's static partitioning or explain better the conditions under which it would not work so so other users don't run into the same issue.

A workaround is of course to use JTAG to flash the merged.hex with dynamic partitioning and overwrite the asset tracker's one, but that's what we were trying to avoid and resort to using only USB and never have to use JTAG for our boards - at least not until full production.

Ultimately, our solution was to copy asset tracker's pm_static.yaml and linclude it into CmakeLists, as you suggested. From then on every thing worked as expected.

Also, one my colleagues posted the same question - we didn't sync up, so this issue should be considered answered too:

devzone.nordicsemi.com/.../thingy-91-fota-reverting-unless-flashed-merge-hex-with-jtag