Extremely slow and unstable FOTA on nrf5340 based device after code update in app

[nfbe]

Hi,

I am developing a nrf5340 based device (no external memory) using FOTA to update the firmware. This process used to work quite well for a while now, but I ran into trouble recently, because to my surprise the update over the air does not work anymore after a small update to the code (content wise not related to FOTA).

The Update gets stuck on the nrf Connect app on my iPhone at various points early in the process - never finishing successfully. Using my own iOS app I can manage to "force" the update process in an extremely slow speed restarting the process several times, because here, too, the FOTA-process breaks repeatedly at unpredictable times (it took more than an hour to do so). This "new behavior" is the same on several samples of my device - so it does not seem to be a hardware problem.

So: the device and the process still seem to work in principle, but became extremely slow and unreliable. Does anyone have an idea what can cause this behavior on the nrf5340? ... and how I could bring this back to the old quick and reliable update process over the air?

Does the speed and stability in any way depend on the size or other properties of the app_update.bin file, which now is (just a little) larger? What are the limits to the FOTA process with nrf5340 determining speed and stability? Could there be any cause effect relationship between the firmware upload via BLE while the firmware on the device communicates via BLE at the same time?

I am using nrf Connect SDK v2.4.0 (arm64) on the nrf5340 side and an up to date version of nrf Connect on my iPhone.

Thanks a lot for your advice.

Regards,

Jens

[nfbe]

Dear Abhijith,

I just tried it with the nrF Device Manager App - and the behavior is just the same. As long as my previous version of my firmware is on my nRF5340 is running, I can reproducibly do a FOTA update either with a just minutely changed previous version or with my new version... but if my new version is running, FOTA is not stable anymore - no matter, which version I try to upgrade to from there.

Yesterday I also updated the iOSMCUManager Library to version from 1.6.0 to 1.7.0 in my own iPhone App, but I do not see any different behavior - other than a slightly different way I need to implement upgrade process due to changes in the API, nothing in the behavior changed.

As I stated before, from the flash size and file size of the two versions there is only a very small difference. The newer version of my firmware, however, is using larger buffers and therefore needs significantly more RAM space (143kB instead of 106kB from what the build process tells me). Could this be the cause of the difference in the FOTA process - not enough RAM left for FOTA? How much free RAM space is required to do this FOTA update? Does the process try to store the entire image in RAM at any time?

Any further hints are very much appreciated.

Regards,

Jens

[nfbe]

Dear Abhijith,

I checked again my configs and found the only difference in the versions to be the use of the FPU (deliberately set: CONFIG_FPU=y... with CONFIG generated because of that), which I had not considered to be of any influence on the FOTA operation and therefore not mentioned. This I switched off for my new version and with this reached an exactly equal configuration to the older version in which FOTA still works fine.
=> Having the exact same config settings did not have any positive influence whatsoever

I increased my CONFIG_MAIN_STACK_SIZE, because I saw, that the mcuboot configuration has a larger stack setting, than my old application.
=> Unfortunately this did not change the behavior either - FOTA still does not work at all with the new version.

Deceasing the RAM usage of my application does not seem to have any influence on the FOTA process.
=> It still does not work.

I still do not understand, how the FOTA-process cannot work at exactly the same configuration parameters - is that not the only influence I have as the programmer of the application using the boot loader provided as well as the network MCU provided as is just via configuration? Which "behavior" of my application e.g. on memory usage or use of the bluetooth interface may influence the FOTA behavior on the nRF5340?

I am still puzzled by this - any enlightening thoughts on solving this mystery highly welcome.

Thanks,

Jens

[nfbe]

Dear Abhijith,

I finally found the reason for the difference in the FOTA behavior: It is a problem with thread handling. In my latest version I reduced/removed waiting times (usleep - calls) in my application. I found, that I can reproducibly "enable" stable FOTA by including sufficient calls to usleep and "disable" FOTA stability by removing them. It looks like my application has been "too dominant".

Including usleep-calls, however, (slightly) slows down the operation of my application not only during FOTA, but always. Is there any way for my application to detect that a FOTA process is active? Or the other way arround: is it possible to enable or disable FOTA-ability of my device at runtime of my application?

Thanks a lot for your advice.

Regards,

Jens

[Menon]

Hello,

It's good to hear that you found the root cause of the issue.

nfbe said:

Including usleep-calls, however, (slightly) slows down the operation of my application not only during FOTA, but always. Is there any way for my application to detect that a FOTA process is active? Or the other way arround: is it possible to enable or disable FOTA-ability of my device at runtime of my application?

I am not completely sure about this approach as I have not tested it before, but you might consider monitoring the characteristics of the SMP service. Implement handlers to monitor SMP commands related to DFU. You can check for specific image management commands to set and clear the DFU flag. Then use the DFU flag in your main application loop to adjust behavior based on DFU status.

You could also try using k_msleep and k_busy_wait instead of usleep. Let me know if that makes any difference in the performance.

Kind Regards,

Abhijith