Bluetooth OTA fails: iOS DFU Library complains that Image 0 (slot 1) is already pending

[responsible_duck]

We are suddenly getting lots of cases where devices aren't able to perform Bluetooth OTA via our iOS app. 

Primarily this is what we are seeing is this: `error=There is no free slot to place the image`.

First we start the update:

This is what the image list response right before this looks like:

dfu: Image List response:
  Header:
    version   : "1"
    op        : 1
    flags     : 0
    length    : 248
    group     : 1
    seqNum    : 60
    commandId : 0

  Payload:
    images:
      - bootable  : true
        hash      : 0xAA00383B0829FA8BA63CC09BD126D7CBEF866D9A8DD8C9BFD176EE13A8FDAAA6
        active    : true
        permanent : false
        slot      : 0
        version   : "0.0.101"
        confirmed : true
        pending   : false

      - version   : "0.0.103"
        hash      : 0xA576F7F8B294DA3A26979D8846E0D2CCE6454370F8A57B0E9C7053EDD5526884
        active    : false
        slot      : 1
        pending   : true
        permanent : false
        bootable  : true
        confirmed : false

    splitStatus : 0

The library throws this error:

We also see `dfu: Image 0 (slot 1) is already pending.`

but proceeds to upload image. Device then restarts upon upload completion but seems to restart on the old firmware, showing the same image list. A further update attempt throws this error:

dfu: There is no free slot to place the image

and stops the update.

We use a `testOnly` strategy (not testAndConfirm) from the library so that we confirm the image in the app. This was working quite fun for a few months until now.  

We see this commit which attempted to fix a similar issue but it seems like it did not resolve this specific case. Also saw this devzone ticket but we didn't make any changes to pm_static.yml and how the secondary slot is allocated. 

We also tried force testing the image stuck in secondary slot using `mcumgr` with no luck. 

[responsible_duck]

Ah we are on NCS 2.6.1

[responsible_duck]

Some additional logs from an OTA attempt that might be helpful:

🟡 dfu: Secondary slot of image 0 is already confirmed
🔵 dfu: Scheduling upload (hash: 0x5E5E75DFCF8A4E6FE6F069C533CA05875DE75A7452B942357F88F6BD5F6F69BF) for image 0 (slot: 1)
🔵 Upgrade state moved from validate to upload
🔵 dfu: Image Confirm response: Header: {"version": "1", "op": "3", "flags": 0, "length": 248, "group": 1, "seqNum": 167, "commandId": 0}, Payload: {"images" : {{"bootable" : true, "active" : true, "confirmed" : false, "permanent" : false, "slot" : 0, "pending" : false, "hash" : 0x024A54FAE4A4831EEFEAF04D6051BEE67332AEB05DA19B59E1116911BAD8F043, "version" : "0.0.111"}, {"pending" : false, "confirmed" : true, "version" : "0.0.108", "hash" : 0x056578B127AA68892BE5092D00DEC162E132490C49000BE5E62026F8EC26A51C, "permanent" : false, "bootable" : true, "slot" : 1, "active" : false}}, "splitStatus" : 0}
🔵 dfu: Image List response: Header: {"version": "1", "op": "3", "flags": 0, "length": 248, "group": 1, "seqNum": 167, "commandId": 0}, Payload: {"images" : {{"bootable" : true, "active" : true, "confirmed" : false, "permanent" : false, "slot" : 0, "pending" : false, "hash" : 0x024A54FAE4A4831EEFEAF04D6051BEE67332AEB05DA19B59E1116911BAD8F043, "version" : "0.0.111"}, {"pending" : false, "confirmed" : true, "version" : "0.0.108", "hash" : 0x056578B127AA68892BE5092D00DEC162E132490C49000BE5E62026F8EC26A51C, "permanent" : false, "bootable" : true, "slot" : 1, "active" : false}}, "splitStatus" : 0}
🟡 dfu: Secondary slot of image 0 is already confirmed
🔵 dfu: Scheduling upload (hash: 0x5E5E75DFCF8A4E6FE6F069C533CA05875DE75A7452B942357F88F6BD5F6F69BF) for image 0 (slot: 1)
🟡 image: An image upload is already in progress
🔴 dfu: There is no free slot to place the image
🔴 image: Upload cancelled due to error: groupCode(iOSMcuManagerLibrary.McuMgrGroupReturnCode)
🟡 Firmware update failed

[responsible_duck]

What's crazy is `mcumgr` only shows one image despite the iOS library showing two:

mcumgr -c acm0 -t 120 image list
Images:
 image=0 slot=1
    version: 0.0.108
    bootable: true
    flags:
    hash: 056578b127aa68892be5092d00dec162e132490c49000be5e62026f8ec26a51c
Split status: N/A (0)

[Menon]

Hello,

Thanks for sharing the complete log. I assume you are working with the nRF5340. Is this a custom board or a development kit? If it is a custom board, were you able to see the same issue on a development kit? Have you tried using the nRF Connect mobile application, and if so, are you experiencing a similar issue? I may need to report this internally, but first, please let me know if this is reproducible on a development kit.

Kind Regards,

Abhijith