Extremely slow and unstable FOTA on nrf5340 based device after code update in app

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

Menon — Tue, 30 Jul 2024 08:56:27 GMT

Hello,

It's good to hear that you found the root cause of the issue.

[quote user="nfbe"]Including usleep-calls, however, (slightly) slows down the operation of my application not only during FOTA, but always. Is there any way for my application to detect that a FOTA process is active? Or the other way arround: is it possible to enable or disable FOTA-ability of my device at runtime of my application?[/quote]

I am not completely sure about this approach as I have not tested it before, but you might consider monitoring the characteristics of the SMP service. Implement handlers to monitor SMP commands related to DFU. You can check for specific image management commands to set and clear the DFU flag. Then use the DFU flag in your main application loop to adjust behavior based on DFU status.

You could also try using k_msleep and k_busy_wait instead of usleep. Let me know if that makes any difference in the performance.

Kind Regards,

Abhijith

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

nfbe — Sat, 27 Jul 2024 07:04:17 GMT

Dear Abhijith,

I finally found the reason for the difference in the FOTA behavior: It is a problem with thread handling. In my latest version I reduced/removed waiting times (usleep - calls) in my application. I found, that I can reproducibly "enable" stable FOTA by including sufficient calls to usleep and "disable" FOTA stability by removing them. It looks like my application has been "too dominant".

Including usleep-calls, however, (slightly) slows down the operation of my application not only during FOTA, but always. Is there any way for my application to detect that a FOTA process is active? Or the other way arround: is it possible to enable or disable FOTA-ability of my device at runtime of my application?

Thanks a lot for your advice.

Regards,

Jens

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

nfbe — Thu, 25 Jul 2024 19:36:41 GMT

Dear Abhijith,

I checked again my configs and found the only difference in the versions to be the use of the FPU (deliberately set: CONFIG_FPU=y... with CONFIG generated because of that), which I had not considered to be of any influence on the FOTA operation and therefore not mentioned. This I switched off for my new version and with this reached an exactly equal configuration to the older version in which FOTA still works fine.
=> Having the exact same config settings did not have any positive influence whatsoever

I increased my CONFIG_MAIN_STACK_SIZE, because I saw, that the mcuboot configuration has a larger stack setting, than my old application.
=> Unfortunately this did not change the behavior either - FOTA still does not work at all with the new version.

Deceasing the RAM usage of my application does not seem to have any influence on the FOTA process.
=> It still does not work.

I still do not understand, how the FOTA-process cannot work at exactly the same configuration parameters - is that not the only influence I have as the programmer of the application using the boot loader provided as well as the network MCU provided as is just via configuration? Which "behavior" of my application e.g. on memory usage or use of the bluetooth interface may influence the FOTA behavior on the nRF5340?

I am still puzzled by this - any enlightening thoughts on solving this mystery highly welcome.

Thanks,

Jens

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

nfbe — Tue, 23 Jul 2024 19:40:18 GMT

Dear Abhijith,

I just tried it with the nrF Device Manager App - and the behavior is just the same. As long as my previous version of my firmware is on my nRF5340 is running, I can reproducibly do a FOTA update either with a just minutely changed previous version or with my new version... but if my new version is running, FOTA is not stable anymore - no matter, which version I try to upgrade to from there.

Yesterday I also updated the iOSMCUManager Library to version from 1.6.0 to 1.7.0 in my own iPhone App, but I do not see any different behavior - other than a slightly different way I need to implement upgrade process due to changes in the API, nothing in the behavior changed.

As I stated before, from the flash size and file size of the two versions there is only a very small difference. The newer version of my firmware, however, is using larger buffers and therefore needs significantly more RAM space (143kB instead of 106kB from what the build process tells me). Could this be the cause of the difference in the FOTA process - not enough RAM left for FOTA? How much free RAM space is required to do this FOTA update? Does the process try to store the entire image in RAM at any time?

Any further hints are very much appreciated.

Regards,

Jens

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

Menon — Tue, 23 Jul 2024 09:29:11 GMT

Hello,

[quote user="nfbe"]

The main problem remains though: Why is the speed and stability of FOTA changing very significantly (from 3.6kb/s stable transmission to something completely unstable, lasting an hour for 200kb or longer, breaking down every now and then...) for just slightly modified code of the app-core with no obvious relation to FOTA and using the exact same config parameters in both cases?

I need to get this stable again soon! And I need to understand, why this gets unstable showing a speed of close to 0b/s in order to avoid this situation for future code and config.

[/quote]

That should not be happening, and it is a bit strange. Is this issue occurring only with the nRF Connect iOS app? Have you tried the nRF Device Manager app? If so, does it exhibit the same behavior?

Kind regards,

Abhijith

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

nfbe — Thu, 18 Jul 2024 19:49:54 GMT

Hi Abhijith,

I will follow your advice and set the BT_MAX_CONN to the same value everywhere (including the sample code delivered from nordic, that is used for the net core...), even though I would think, that it should not make any difference and only depend on the decisive one used by the system. Otherwise having configuration overlays would not make a lot of sense, would it?

I need to get this stable again soon! And I need to understand, why this gets unstable showing a speed of close to 0b/s in order to avoid this situation for future code and config.

The bluetooth communication from my application to the same iPhone works perfectly fine, by the way - so generally a fast and efficient bluetooth communication is possible with this configuration.

Any further help and ideas very much appreciated!

Regards,

Jens

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

Menon — Thu, 18 Jul 2024 08:28:02 GMT

Hello,

[quote user="nfbe"]
Is it preferable to set a certain (higher) number of connections allowed? I do have contradicting settings for BT_MAX_CONN in the different config files on the network core side - one file sets it to 16, another one to 10 and the third and decisive one sets this parameter to 2. Other than that conflict "imported" from the nordic samples, I do not see any problems in my configuration... and more importantly - I have not changed any of the bluetooth relevant configuration parameters compared to the old version that worked. I did not work terribly fast, but reproducibly at about 3.6 kb/s for a 200kb file - and, rolling back to that version (flashing it on the device and then updating it with one byte for build number changed via FOTA), still works at that speed.[/quote]

I recommend standardizing the "BT_MAX_CONN" value across all configuration files. Setting different values in various configurations can lead to inconsistencies in resource allocation (bandwidth and throughput) and performance. It's advisable to assign the lowest possible value for "BT_MAX_CONN" to maintain consistency and optimize performance

[quote user="nfbe"]I would really like to understand, what determines this speed and stability of the nordic FOTA update[/quote]

I'm not certain, but I believe there isn't an exact number for the transfer rate because it depends significantly on various factors such as Bluetooth connection parameters, MTU size, the type of radio used, and the environment in which you're testing. Optimizing some of these parameters and conducting thorough testing under different conditions can increase the transfer rate to some extent.

Kind Regards,

Abhijith

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

nfbe — Mon, 15 Jul 2024 19:30:45 GMT

Dear Abhijith,

thank you very much for your reply.

I already have set the CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU_SPEEDUP to yes. in my project.conf for the app core side. It is also set to yes in the rpms_child_image_overlay.conf on the network core side, which is coming from the otp/nordic/ncs/v2.4.0/nrf/samples/common/mcumgr_bt_ota_dfu/ path.

So, unfortunately, this is not the solution to my problem.

Is it preferable to set a certain (higher) number of connections allowed? I do have contradicting settings for BT_MAX_CONN in the different config files on the network core side - one file sets it to 16, another one to 10 and the third and decisive one sets this parameter to 2. Other than that conflict "imported" from the nordic samples, I do not see any problems in my configuration... and more importantly - I have not changed any of the bluetooth relevant configuration parameters compared to the old version that worked. I did not work terribly fast, but reproducibly at about 3.6 kb/s for a 200kb file - and, rolling back to that version (flashing it on the device and then updating it with one byte for build number changed via FOTA), still works at that speed.

This older version by the way contains the same conflicting settings for BT_MAX_CONN on the network core side - and just works.

The app_update.bin file for the new version is 156 bytes larger than the old one... but does FOTA somehow depend on the content itself?

I would really like to understand, what determines this speed and stability of the nordic FOTA update.

Any hints welcome to reliably solve this problem.

Regards,

Jens

RE: Extremely slow and unstable FOTA on nrf5340 based device after code update in app

Menon — Mon, 15 Jul 2024 12:59:31 GMT

Hello,

Could you please try enabling the configuration CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU_SPEEDUP to expedite the OTA DFU transfer? There's a thread discussing a workaround to enhance OTA DFU speed, although it's Android-centric. Please let me know if this resolves the issue for you.

Kind regards,
Abhijith