Not enough free space to run swap upgrade

[JH7]

Hello,

I need to make the OTA DFU work and I followed the instructions here:
https://docs.nordicsemi.com/bundle/ncs-latest/page/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/README.html#smp-svr

In some situations I'm getting this error: "W: Not enough free space to run swap upgrade"

Environment:

• nRF52 development kit (PCA10040)
• nRF Connect SDK nRF Connect SDK v2.7.0-5cb85570ca43
• Zephyr OS v3.6.99-100befc70c74
• MCUboot v2.1.0-dev-daf2946a0f07

This is how I'm testing this:

• I have a simple application that blinks LED1, advertises a custom BLE service and implements the SMP server. Let's call that "app1".
• Another slightly different version of the application above that blinks LED2 instead. Let's call that "app2".

I enable McuBoot, flash the board with the merged firmware and I'm able to use nRF Connect App and Device Manager App to do DFU ota. I can easily update from app1 to app2 and vice versa.

When I add CONFIG_SHELL=y, I can get a nice shell command over uart port (let's call this "app3"). The problem is, this image gets bigger, and when I try to do DFU OTA with it, I get this message:

I: Starting swap using move algorithm.
W: Not enough free space to run swap upgrade
W: required 192512 bytes but only 188416 are available

which is understandable as there's no space to do the swap. 

The scenario now is the following. App3 failed to update and App1 is still running as it should. But when I try to upload app2 (same smaller image from before, no shell enabled),
I keep getting the same error message and cannot upload anything else because the App3 is still there in flash filling the memory.

Questions:

1 - If there's no space to swap, why the bootloader lets the upload to start? How to prevent that?
2 - If an image fails to upload completely for any reason, how to erase it from flash so it does not block future updates?
3 - If this is something related to a specific NCS/Zephyr/MCUboot version, could you point an older version that doesn't show this issue?

Thank you!

More information:

B0 compilation:
Memory region         Used Size  Region Size  %age Used
           FLASH:       22648 B        28 KB     78.99%
             RAM:        5576 B        64 KB      8.51%
        IDT_LIST:          0 GB        32 KB      0.00%

MCUBoot compilation:
Memory region         Used Size  Region Size  %age Used
           FLASH:       34720 B        48 KB     70.64%
             RAM:       25920 B        64 KB     39.55%
        IDT_LIST:          0 GB        32 KB      0.00%

Apllication compilation (CONFIG_SHELL=n):
Memory region         Used Size  Region Size  %age Used
           FLASH:      162084 B     192000 B     84.42%
             RAM:       49528 B        64 KB     75.57%
        IDT_LIST:          0 GB        32 KB      0.00%

Apllication compilation (CONFIG_SHELL=y):
Memory region         Used Size  Region Size  %age Used
           FLASH:      186416 B     192000 B     97.09%
             RAM:       54264 B        64 KB     82.80%
        IDT_LIST:          0 GB        32 KB      0.00%

west build -t partition_manager_report

  flash_primary (0x80000 - 512kB): 
+--------------------------------------------------+
+---0x0: b0_container (0x8000 - 32kB)--------------+
| 0x0: b0 (0x7000 - 28kB)                          |
| 0x7000: provision (0x1000 - 4kB)                 |
+---0x8000: s0 (0xc200 - 48kB)---------------------+
| 0x8000: s0_pad (0x200 - 512B)                    |
+---0x8200: s0_image (0xc000 - 48kB)---------------+
| 0x8200: mcuboot (0xc000 - 48kB)                  |
+--------------------------------------------------+
| 0x14200: EMPTY_0 (0xe00 - 3kB)                   |
+---0x15000: s1 (0xc200 - 48kB)--------------------+
| 0x15000: s1_pad (0x200 - 512B)                   |
| 0x15200: s1_image (0xc000 - 48kB)                |
+--------------------------------------------------+
| 0x21200: EMPTY_1 (0xe00 - 3kB)                   |
+---0x22000: mcuboot_primary (0x2f000 - 188kB)-----+
| 0x22000: mcuboot_pad (0x200 - 512B)              |
+---0x22200: app_image (0x2ee00 - 187kB)-----------+
+---0x22200: mcuboot_primary_app (0x2ee00 - 187kB)-+
| 0x22200: app (0x2ee00 - 187kB)                   |
+--------------------------------------------------+
| 0x51000: mcuboot_secondary (0x2f000 - 188kB)     |
+--------------------------------------------------+

  sram_primary (0x10000 - 64kB): 
+-------------------------------------------+
| 0x20000000: sram_primary (0x10000 - 64kB) |
+-------------------------------------------+

[Susheel Nuguru]

I am not expert in this, but searching for some info here are my few thoughts

Considering the constraints of flash memory and the issues with OTA updates when larger apps (like app3 with CONFIG_SHELL=y) push memory usage to its limit, here are some ways to address each question:

1 - If there's no space to swap, why the bootloader lets the upload to start? How to prevent that?

The bootloader, MCUboot in this case, doesn't automatically verify available swap space before beginning an upload. To mitigate this, you might look into two possible approaches:

1. Configure a Pre-Upload Check in the Application: Adding a check within the application using mcumgr commands or a custom routine to assess available swap space before initiating a DFU update could help. Keep in mind that this requires custom logic, as MCUboot doesn’t currently halt uploads based on space limitations.

2. Expand the Swap Partition (if feasible): Adjusting your partition setup to enlarge the swap area might make larger uploads possible, depending on the available flash space. You can modify pm_static.yml to reallocate memory for secondary images if needed.

2 - If an image fails to upload completely for any reason, how to erase it from flash so it does not block future updates?

To clear a failed image from flash:

• Erase the Secondary Slot Using mcumgr: The mcumgr tool has commands that can erase sectors or flash regions. You could run the following command to erase the secondary slot:

  mcumgr -t 5 -c <connection_type> flash erase <secondary_slot_offset> <size>

  Replace <secondary_slot_offset> and <size> with values from your partition layout; for instance, mcuboot_secondary begins at 0x51000 and spans 0x2F000 (188kB) in your case.

• Reboot and Retry the Upload: After erasing, reboot the device to let MCUboot clear any flags related to the prior upload, then try uploading a smaller image like app2.

3 - If this is something related to a specific NCS/Zephyr/MCUboot version, could you point an older version that doesn't show this issue?

This is likely due to partition size rather than a specific bug in recent versions. Still, some potential options include:

• Testing a Smaller Application Version: Temporarily reducing the app size by disabling some features could make DFU uploads more manageable.

• Trying nRF Connect SDK v2.6.x: This version may handle partitions and memory slightly differently. Running a test build with v2.6.x could highlight any improvements in memory management.

Implementing these strategies should help manage space limitations during DFU updates. Also, fine-tuning your partition layout and reducing the application’s footprint may prevent similar issues in future updates.

[JH7]

Hi Susheel, thanks for your reply.

I'm thinking in the scenario where my product is released and the only way to update it is over BLE.

mcumgr -t 5 -c <connection_type> flash erase <secondary_slot_offset> <size>

Can this be executed over BLE?

If I understand your suggestions, there will always be the possibility of having a bigger image that would lead to this problem, no matter how I adjust the application and partitions, right?
I could make sure to test a new image everytime before releasing it, but I'm thinking this can also be a vector of attack as a malicious user could prevent updates only by uploading an invalid image that is too big. Does it make sense?

It looks like the options are: change the MCUBoot code to check the size, make it erase the secondary partition, and/or try to control the access to the BLE OTA (let only authorised apps to update firmware).

Do you have any other idea?

Thank you!

[Hieu]

Hi,

I will continue to support you on this case.

JH7 said:

I'm thinking in the scenario where my product is released and the only way to update it is over BLE.

[...]

Can this be executed over BLE?

Yes, it can. Please try to do so with our Device Manager app.

JH7 said:

If I understand your suggestions, there will always be the possibility of having a bigger image that would lead to this problem, no matter how I adjust the application and partitions, right?

It can happen during development, but it should never happen with your end users, in the sense that the new image should be tested extensively before deployment.

JH7 said:

It looks like the options are: change the MCUBoot code to check the size

I believe what Susheel recommended was to change the application code to check the size, rather than MCUboot.

JH7 said:

make it erase the secondary partition

This is a reasonable approach.

JH7 said:

try to control the access to the BLE OTA (let only authorised apps to update firmware).

For most applications, this is recommended regardless. The only exception I can think of is a hacking/development platform.

JH7 said:

Do you have any other idea?

Your mobile app can also pre-emptively erase the flash area after a failed attempt, depends on the failure reason.

Obviously, this doesn't help the end-user having to sit through the entire download time. However, this issue should be circumvented altogether by testing the new application before deployment to end-users, like I mentioned above.

Hieu