NRF5340 simultaneous multi-image FOTA DFU: works with initial firmware update, but can't upload any revised new version

[Sergey]

I'm trying to use simultaneous FOTA update of both cores with NRF5340DK, and there is a strange behavior with any update with modification - the uploading from mobile app just doesn't start. My setup:

1) NRF5340DK

2) mobile phone with Android NRF connect app for updating via Bluetooth

3) sample project https://github.com/hellesvik-nordic/samples_for_nrf_connect_sdk/tree/main/bootloader_samples/nrf5340/mcuboot_smp_ble_simultaneous  compiled with NRF Connect SDK 2.1.2

After firmware initial flashing with "west flash --force --erase" the firmware works as expected. I'm able to make DFU via Bluetooth with dfu_application.zip from build/zephyr folder - and I can do it multiple times without problems.

But if I change something in firmware (for example a log message), compile a new build and try to make DFU with this the new dfu_application.zip - uploading doesn't start. The NRF Connect Android app immediately disconnects after "Starting DFU". 

What I'm doing wrong? How to make DFU with a modified firmware, and why it works only with initial firmware

[Sergey]

I believe I found why this issue happens.

After many trials I took another NRF5340DK (fortunately I have two), flashed it with the same firmware - and updating works as it should be. Initially I decided that my first DK is broken, but it's not...

This sample uses external qspi flash for storing update images... I did full qspi erase with

nrfjprog --qspieraseall

And now updating of my first DK also works as expected.

I believe at one moment the external qspi flash was written with wrong data (I tested interruption of the updating process with power off). After that the upgrading functionality became broken

This leads me to a question: is it possible to handle such situation in any way? Maybe we should make full erase of an external flash before each DFU procedure? Or maybe it's possible to erase qspi with some MCUMGR command?

[AHaug]

Hi, 

Apologies for the very late response. I have been out of office due to circumstances I did not plan for when I picked up your case, and I will be until Friday this week.

A quick (and temporary) answer to your question: I believe there should be a config or something similar that will allow you to erase what is on the external flash before doing the DFU, and my best guess (with the resources I have at hand at this given point in time) is that you will have to erase what is on the flash before doing the DFU. Before I can verify this 100% I need to consult with some of my colleagues, and I will do my best to get a reply that answers and verifies your findings regarding your issue on Friday, no later than Monday. 

I hope that this prolonged response time has not caused too many issues for you,

Kind regards,
Andreas

[AHaug]

Hi, 

Apologies for the delay

Sergey said:

My main goal is to achieve DFU with encrypted app image (I described it here devzone.nordicsemi.com/.../how-to-encrypt-fw-for-mcuboot-image-update ) - now it's the second attempt :-) I wish I could do it early, but...

I'm sure you've been told this already, but AFAIK encrypted DFU is not supported by us, but it is not impossible to do. I will see if I can manage to find enough resources to achieve this, but before we do so I want you to verify/test the following so we're aligned in our processes:

Sergey said:

The main problem was - if I generate a new build with any changes (logs, or just change version number in config), I see "Starting DFU" on the mobile device screen, and then it immediately disconnects. I.e. firmware uploading even doesn't start. I would say that it's a bug in mobile app, but why it works after full erase of SPI FLASH?

I've still not been able to recreate not able to see recreate the DFU issue by adding logging, or other simple things to the application in between DFU. Below follows a sample of the procedure I've done. I assume this is what you've already done, but I'm adding it so we're following the same procedure. I've been using a nRF5340DK, v.0.11.0. 

In case there is a mobile difference, could you state which phone you've used to test the sample?

• Build and flash the sample without modifications
• Do modifications by adding logging to the bootloader:
• In proj.conf add:

 

CONFIG_LOG=y
CONFIG_MCUBOOT_BOOTUTIL_LIB=y
CONFIG_MCUBOOT_UTIL_LOG_LEVEL_INF=y

• In bootloader\mcuboot\boot\bootutil\src\bootutil_public.c add some info logs to various functions such as illustrated below to see if the changes has been done 
• Build the new build without flashing and add dfu_application.zip to your phone
• Connect the device to your phone through the nRF Connect application and press DFU. Select the newly added DFU app and press "Confirm Only"

The following output can then be observed:

1. DFU being done
2. Output after writing "nrfjrpog --reset" in a terminal
3. The new build is running

Then I added logging to the application and did DFU in sequence:

• Add the following to proj.conf

#Logging for app
CONFIG_LOG=y
CONFIG_USE_SEGGER_RTT=n
CONFIG_LOG_BACKEND_UART=y
CONFIG_LOG_DEFAULT_LEVEL=3

• Add the following to main.c:

#include <logging/log.h>

#define LOG_MODULE_NAME main
LOG_MODULE_REGISTER(LOG_MODULE_NAME);

void main(void)
{
    LOG_INF("Inside main(), added log_inf statement \n ");
    img_mgmt_register_group();
    start_smp_bluetooth();
	printk("Build with logs added to main.c and mcuboot, build_4  \n");
}

Then build without flashing once more, and upload now the new dfu_application.zip to your phone and repeat the dfu procedure

1. Starting from the boot of the application
2. Output from DFU
3. Output after nrfjprog --reset
4. New output from application

I observe the same results with and without erasing the device.

Do you manage to do simultaneous DFU (without erasing) this way?

Kind regards,
Andreas

[Robert de Brum]

Hello

I am running the same sample and I tried to follow the steps described above. When I do so, I get the following output. Note that the current firmware will include "Moo" in the printk in main.c, and my new firmware I am trying to update should be saying "Quack". As you can see the update does not take.

This is a screenshot of 2 attempts to apply dfu_application.zip. I can add that the qspi --erase will allow the DFU to work appropriately. I am anxious to get simultaneous DFU Updates working soon.

One question I have is why couldn't we include some code to erase the flash sectors on reset? If this seems to work from the command line, can we have the board erase the flash sector on reset so that after a DFU package gets applied, it then also gets erased so that it is prepared for the next update?

[AHaug]

Hi,

 Sergey  and Robert de Brum , which phones and what Android OS are you running? My colleague who wrote the sample are seeing the same issue that Sergey described with a  Google Pixel 4 where the DFU process does not start in the app. I am not seeing the issue, even when attempting DFU with the exact same build that fails for my colleague with a Samsung Galaxy S8+. For me the progress bar shows up as expected and I can run his version of the sample after updating.

I don't think this should be an issue that is mobile specific, but it might be, so if you could state which phones you're using then we can cross examine if there is a phone-related bug in the app.

Robert de Brum said:

One question I have is why couldn't we include some code to erase the flash sectors on reset? If this seems to work from the command line, can we have the board erase the flash sector on reset so that after a DFU package gets applied, it then also gets erased so that it is prepared for the next update?

As for this, the short answer is yes. The longer answer is that the entire point of having dual bank DFU is that the device should revert back to the previous firmware if anything faults during DFU or if the image can not be authenticated. If you do a full-erase before uploading the new firmware, there are no old firmware to revert back to in case either of the mentioned scenarios occur.

Kind regards,
Andreas

[Sergey]

Hi all,

I use Xiaomi Mi9T MIUI 12.1.4 (Android  11). I also have Iphone 6s, but as I wrote, there is a restiction of BLE throughput from iOS, which leads to another error. I guess I could figure out how to fix it, but I can't work again these days :-(

[Robert de Brum]

I am testing this on a Google Pixel 4 as well as a 2019 iPad - not the pro version.

Thanks for the input. If it could be device specific, have there been any known fixes for this?

[AHaug]

Robert de Brum said:

Thanks for the input. If it could be device specific, have there been any known fixes for this?

I am working on getting a larger sample size than us 3 to see if it is device specific. If it indicates that it is so, then I'll investigate it closer and examine if there are any possible fixes.

Kind regards,
Andreas

[AHaug]

Hi,

I've been out of office due to the christmas holidays here in Norway, but in the meanwhile I've had some colleagues of mine perform some tests and see if they could reproduce the errors on their end. Unfortunately it all seemed to succeed on their ends. 

Could you attempt to do the following:

1. Build and flash a clean version of https://github.com/hellesvik-nordic/samples_for_nrf_connect_sdk/tree/main/bootloader_samples/nrf5340/mcuboot_smp_ble_simultaneous
   1. If you have modified the SDK at any point, please try with a clean version of NCS.
2. Perform DFU with the pre-built zips found in the folder "dfu_zips" in https://github.com/aHaugl/NCS_Simult_DFU_test in sequence or try to follow the steps in the repository yourselves

If you still observe the same issues, please state so here and I'll follow up on them.

Kind regards,
Andreas

[Robert de Brum]

Hello,

Thanks so much for your time and efforts on this. I was able to successfully DFU both cores per the sample via Google Pixel 4.

I think that the first issue that I discovered was that I did not successfully build the sample with ncs version 2.1.2 - I was still using 2.0.0.

The other issue I am discovering is that while the pixel will successfully DFU every time, the iPad fails to do so - I was only able to DFU once with the iPad, from dfu_zips/3/dfu_application.zip to dfu_zips/1/dfu_application.zip.

[AHaug]

Robert de Brum said:

I think that the first issue that I discovered was that I did not successfully build the sample with ncs version 2.1.2 - I was still using 2.0.0.

A version conflict could explain the issue. The sample should be usable for NCS v2.0.0 up to the latest v2.1.x, and with v2.2.0 it yields some errors due to header changes (not checked what issues yet). So also note that this sample is something that is currently not a official sample we support in the SDK so in its current state it will not be maintained, but rather a sample used as suggestions/illustrations for developers to use in their own applications.

Robert de Brum said:

The other issue I am discovering is that while the pixel will successfully DFU every time, the iPad fails to do so - I was only able to DFU once with the iPad, from dfu_zips/3/dfu_application.zip to dfu_zips/1/dfu_application.zip.

This sounds strange. What do you observe when testing the pre-built dfu_zips on your iPad/where does it fail?

What we observerd when performing the test with an iOS phone was that the finalizing of the transfer did not show up in the app, but we saw the new firmware when observing the output in a terminal after the app timed out and restarting the device. Could it be the same case with the app version on the iPad?

Kind regards,
Andreas

[Sergey]

Hi,

I've just tested with my phone (the same phone which I used with experienced issue). It seems all works as expected with your pre-built zip files. 

I will work with my tests later, and if I will experience my issue, I will report here once again. Can't imagine why it happened...