FOTA SMP very slow

[LukeC]

I am trying to move our existing code based on the recommended NCS SDK.  I was able to get MCU boot and SMP working, however, the update is extremely slow.

I now seem to be getting like 1Kb/s.  On our existing NRF5 based products, I am able to get ~9-10 Kb/s on the old DFU.

This seems like a big regression compared to what we currently have.

I have tried to use the highest throughput settings (see below), but my concern is that Nordic has much less control over this new architecture and we are limited to what Zephyr/MCUboot has to offer.  The speed of the NRF5 SDK on the DFU was super useful and provided a great user experience for in-field updates.

Also, does anyone have any examples using MCUboot to store a FOTA image to a an external SPI flash chip? 

CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_ATT_PREPARE_COUNT=2
CONFIG_BT_CONN_TX_MAX=10
CONFIG_BT_L2CAP_TX_BUF_COUNT=10
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_DYNAMIC_CHANNEL=y
CONFIG_BT_CTLR_PHY_2M=y
CONFIG_BT_CTLR_RX_BUFFERS=2
CONFIG_BT_BUF_ACL_TX_COUNT=10
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

nrfnrf

[Håkon Alseth]

Hi,

 

I now seem to be getting like 1Kb/s.  On our existing NRF5 based products, I am able to get ~9-10 Kb/s on the old DFU.

You can get a bit of a better speed by adjusting the BT params (data length max etc). Here's a small sequence of config's that should help (note: these are not included in the below .zip file!)

# Speed up BT transfer
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_BUF_ACL_TX_COUNT=7
CONFIG_BT_BUF_ACL_RX_SIZE=255
CONFIG_BT_BUF_EVT_RX_SIZE=255

 

Also, does anyone have any examples using MCUboot to store a FOTA image to a an external SPI flash chip? 

I have used one of our tests as the basis for this alteration of the smp_svr example:

https://github.com/nrfconnect/sdk-nrf/tree/main/tests/modules/mcuboot/external_flash

 

What I did to add external flash over qspi support:

* add external flash to the board (nrf52840dk_nrf52840) by placing an overlay file in boards/ folder of the application

* add mcuboot configurations under child_image/ folder

* add "overlay-bt.conf" to prj.conf

* add CONFIG_NORDIC_QSPI_NOR_STACK_WRITE_BUFFER_SIZE=32 (required for writing mcuboot magic header) and CONFIG_PM_PARTITION_SIZE_MCUBOOT_SECONDARY=0xee000 to prj.conf

* add static partition layout (pm_static.yml).

 

Here's the altered example:

ncs_1_7_smp_svr_nrf52840dk_nrf52840_qspi_ext_flash.zip

Upload image:

sudo /path/to/mcumgr --conntype ble --connstring ctlr_name=hci0,peer_name='Zephyr' image upload zephyr/app_update.bin

 

List image (to get hash):

sudo /path/to/mcumgr --conntype ble --connstring ctlr_name=hci0,peer_name='Zephyr' image list

 

Mark image for test:

sudo /path/to/mcumgr --conntype ble --connstring ctlr_name=hci0,peer_name='Zephyr' image test <hash of image 0 slot 1>

 

Reset to test-boot your new image:

sudo /path/to/mcumgr --conntype ble --connstring ctlr_name=hci0,peer_name='Zephyr' image reset

 

mcuboot should now show this output to indicate that it's test booting the new image:

....

*** Booting Zephyr OS build v2.6.99-ncs1-1  ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: test
I: Bootloader chainload address offset: 0xc000

....

 

once the new image is booted, you can confirm it from the uart serial console by writing "mcuboot confirm".

 

Kind regards,

Håkon

[LukeC]

Thanks for the SPI Flash example Håkon, I will definitely try it out.

One additional thought on the speed of the FOTA vs the DFU.

I thought when I looked at the mobile apps for the NRF5 DFU one of the tricks being used to get better speeds was using WRITE_NORESPONSE to send out the data.  Basically, send packets using no response writes and have the Nordic peripheral ack the large file buffer writes.

Maybe it would be possible to write a custom FOTA BLE protocol that sends the app *.bin file quickly with CRC checking.  If we came up with a custom BLE  comms protocol we might be able to have much better throughput.  Once the file data is verified on the receiving side, it would just be hooking this into MCU boot.

It is just going to be very hard to go to our product guys and tell them we updated to a newer Bluetooth stack and reduced performance (from their limited perspective).  Obviously, NCS has lots of benefits in leveraging Zephyr, OpenThread, etc...  

[LukeC]

One other thought on this.  Isn't a big reason the FOTA is slower than the DFU due to the NCS no longer separates out the Softdevice from the application.

Before, we could just update the app without updating the softdevice, but now the softdevice is merged into the app correct?  So we are increasing the update size significantly for BLE applications.

[Håkon Alseth]

Hi,

 

LukeC said:

One other thought on this.  Isn't a big reason the FOTA is slower than the DFU due to the NCS no longer separates out the Softdevice from the application.

Before, we could just update the app without updating the softdevice, but now the softdevice is merged into the app correct?  So we are increasing the update size significantly for BLE applications.

This is correct. the application + stack is now one application. The time it takes to upgrade will heavily depend on how large your image is, and historically it has been smaller for a nRF5 SDK based application due to this stack + application split.

LukeC said:

I thought when I looked at the mobile apps for the NRF5 DFU one of the tricks being used to get better speeds was using WRITE_NORESPONSE to send out the data.  Basically, send packets using no response writes and have the Nordic peripheral ack the large file buffer writes.

smp_bt already uses this:

https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.0-ncs1/subsys/mgmt/mcumgr/smp_bt.c#L189-L190

So, there's still room-for-improvement from our side here, by adjusting the link parameters and so forth and providing a clear method on how to configure and set this up for the purpose of higher throughput. 

 

LukeC said:

It is just going to be very hard to go to our product guys and tell them we updated to a newer Bluetooth stack and reduced performance (from their limited perspective).  Obviously, NCS has lots of benefits in leveraging Zephyr, OpenThread, etc...  

One thing is that the binary itself is larger, but the throughput (per second) should still be matched in both SDKs. As mentioned, I've reported this back to the developers, so that we can look at improving the throughput.

 

Kind regards,

Håkon

[LukeC]

Håkon Alseth said:

smp_bt already uses this:

https://github.com/nrfconnect/sdk-zephyr/blob/v2.7.0-ncs1/subsys/mgmt/mcumgr/smp_bt.c#L189-L190

So, there's still room-for-improvement from our side here, by adjusting the link parameters and so forth and providing a clear method on how to configure and set this up for the purpose of higher throughput. 

From my experience, as long as the peripheral connection settings are optimized, the biggest factor is the mobile app.

Looking at the Device Manager Library, it does look like it is using a writeNoResponse architecture.  When I wrote a custom MCU BLE bootloader, this was the most critical piece to get fast bulk write speeds.

 // Check that data length does not exceed the mtu.
        let mtu = targetPeripheral.maximumWriteValueLength(for: .withoutResponse)
        if data.count > mtu {
            log(msg: "Length of data to send exceeds MTU", atLevel: .error)
            // Fail with an insufficient MTU error.
            fail(error: McuMgrTransportError.insufficientMtu(mtu: mtu) as Error, callback: callback)
            return false
        }
        

https://github.com/NordicSemiconductor/IOS-nRF-Connect-Device-Manager/ 

Håkon Alseth said:

One thing is that the binary itself is larger, but the throughput (per second) should still be matched in both SDKs. As mentioned, I've reported this back to the developers, so that we can look at improving the throughput.

At least for my setup, I am seeing around 1.5 kb/s with FOTA with the DLE settings above versus the 9+ Kb/s I get with the NRF5 DFU, so I certainly think there must be some tweaks we can try to get closer to the same performance.  With the old DFU, we have some updates that are very large (including some graphics), and it maintains the speed very well over the whole update.