nrfutil dfu random failures

[simone.bonetti]

We're using nrfutil v.6.1.0 in order to perform DFU updates to our target board using nrf52840-Dongle, and we're experiencing some issues. Everytime we try a download, even if sometimes we're able to successfully perform it, we often witness a high failure rate, mostly caused by random errors happening while trying to connect to the target board, like the following:

pc_ble_driver_py.exceptions.NordicSemiException: Failed to ble_gattc_exchange_mtu_req. Error code: NRF_ERROR_INVALID_PARAM

pc_ble_driver_py.exceptions.NordicSemiException: MTU exchange request failed. Common causes are: missing att_mtu setting in ble_cfg_set, different config tags used in ble_cfg_set and connect.

We tried with a lower MTU, or disabling the MTU negotiation inside the nrfutil python script, but it did not solve the issue.

We tried to perform the download using nRF Connect v3.5.0 (Bluetooth Low Energy App v.2.4.0) using the same dongle, and we experienced no issues at all, 100% download success rates;
so we assumed that the issue was not inside the nrf connectivity Fw of the dongle, nor in our target board bootloader.

We did some research and found out that at the time of writing, nRF Connect v3.5.0 relies on pc-ble-driver-js v2.7.2 library to perform DFU updates, while nrfutil uses pc-ble-driver-py v0.14.2.
Both libraries relies on pc-ble-driver library but the phyton lib is outdated and currently points to pc-ble-driver v4.1.1, while the Node.JS lib points to pc-ble-driver v4.1.2.

In the release notes of pc-ble-driver v4.1.2, the following it's stated:

Bug fixes

• Fixed an issue where events after SD_BLE_GAP_CONNECT were occasionally not received
• Fixed an issue where USB connectivity firmware hangs after receiving many notifications

We're suspecting that those fixes are related to the issues we're experiencing and so we're wondering if you could update pc-ble-driver-py to the latest pc-ble-driver version and release an updated nrfutil tool.
Since we're planning to use nrfutil to perform updates on the company assembly line by december, we cannot rely on the current unstable behavior, so the issue is quite urgent and we'd like to have a stable version asap.

Can you please comment on our considerations? could it be the root cause? are you already planning a new release of nrfutil? if yes, when?

Thank you very much for your support.

Regards.

[Edvin]

Hello,

How did you install nrfutil v6.1.0? Can you try to install it using pip install?

I can't say that I have seen this issue before. What dongle version do you have? What does it say on the sticker on the dongle?

Best regards,

Edvin

[simone.bonetti]

Hi,

yes, we did both pip install and installation from sources, but both methods produced failures.

Dongle Information:

nRF52840-Dongle

PCA10059

1.2.0

2019.37

F9:4F:15:99:31:88

we tried flashing the Dongle Connectivity Fw using both Nrf Connect Ble and nrfutils (using -f option) but the results were the same.

We are experiencing the same problems on 2 different PCs with 2 different dongles ( the second has 2020.07 reported on the sticker, the other values are the same except for the MAC address obviously).

[Edvin]

When you are using nrfutil, you should flash the device using nrfutil as well. It is not the same FW running when you use the dongle as the connectivity dongle on nRF Connect for Desktop and when you use it as the connectivity dongle for nrfutil.

Let me give it a go tomorrow (I am sorry. I am not in the office today, and I don't have a dongle with me, so I will have to try tomorrow).

Just in case I have some difficulties reproducing. What SDK is your bootloader from, and what Hardware is the target device running? Does it reproduce with an unmodified bootloader as well?

Best regards,

Edvin

[simone.bonetti]

i narrowed down the issue in order to help you reproduce it. The best configuration i found was the following:

Target board: PCA10100

IDE: Segger Embedded Studio

i compiled and flashed the following fw: examples\dfu\secure_bootloader\pca10100_s140_ble from SDK 17.0.0 using a public key generated from a private key generated with nrfutils 6.1.0

i compiled the following fw: \examples\ble_peripheral\ble_app_uart\pca10100\s140 from from SDK 17.0.0 and created a DFU package using the previously created private key.

i started DFU update with nrfutils 6.1.0 after flashing the dongle Fw with the -f option (the dongle shows a green LED on) and i tried 10 downloads. 2 of them showed the following error just after the connection phase:

pc_ble_driver_py.exceptions.NordicSemiException: MTU exchange request failed. Common causes are: missing att_mtu setting in ble_cfg_set, different config tags used in ble_cfg_set and connect.

I also tried with PCA10056 eval board using precompiled FW from examples\dfu\secure_dfu_test_images\ble\nrf52840 and i was not able to reproduce the issue, so COULD be a something influenced by the HW, but  in my opinion still related to the tools performing the update, not to the target itself.

I still believe that an update of pc-ble-driver-py library with the pc-ble-driver v4.1.2 could be one of the first things to do, due to the release notes issues that i posted in the main post.

[Edvin]

Hello,

I tested this now, and tried to run 10 DFU updates in a row two times, and all of them succeeded as expected. 

Did you do any changes to the bootloader, other than changing the private key and public keys?

Can you please try to copy the attached .bat script to SDK17\examples\dfu\secure_bootloader\pca10056_s140_ble\armgcc, and run it from a command line?

This should work out of the box * if you use an nRF52840 DK as your target board. Just replace the serial number in:

set TARGET_SNR=123456789

with the serial number of your target DK (the one containing the bootloader), and replace the 

set CONNECTIVITY_COM_PORT=COM5

with the COM port of your pre-programmed connectivity dongle. Please note the line saying: "TODO: ..."

NB: this will delete the dfu_public_key.c and the private.key, so please make a copy of those for backup.

Fullscreen test.bat Download

set TARGET_SNR=683963588
set CONNECTIVITY_COM_PORT=COM5

del ..\..\..\private.key
del ..\..\..\dfu_public_key.c

nrfutil keys generate ..\..\..\private.key
nrfutil keys display --key pk ..\..\..\private.key --format code --out_file ..\..\..\dfu_public_key.c

make -j9
make -j9 -C ..\..\..\..\ble_peripheral\ble_app_buttonless_dfu\pca10056\s140\armgcc\

mkdir files

del files\buttonless.hex
copy ..\..\..\..\ble_peripheral\ble_app_buttonless_dfu\pca10056\s140\armgcc\_build\nrf52840_xxaa.hex files\buttonless.hex

del files\bl_settings.hex
nrfutil settings generate --family NRF52840 --application files\buttonless.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 --key-file ..\..\..\private.key files\bl_settings.hex

nrfjprog -e --snr %TARGET_SNR%
nrfjprog --program ..\..\..\..\..\components\softdevice\s140\hex\s140_nrf52_7.0.1_softdevice.hex --verify --snr %TARGET_SNR%
nrfjprog --program _build\nrf52840_xxaa_s140.hex --verify --snr %TARGET_SNR%
nrfjprog --program files\buttonless.hex --verify --snr %TARGET_SNR%
nrfjprog --program files\bl_settings.hex --verify --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

del files\dfu_image_2.zip
del files\dfu_image_3.zip
del files\dfu_image_4.zip
del files\dfu_image_5.zip
del files\dfu_image_6.zip
del files\dfu_image_7.zip
del files\dfu_image_8.zip
del files\dfu_image_9.zip
del files\dfu_image_10.zip
del files\dfu_image_11.zip

nrfutil pkg generate --application files\buttonless.hex --application-version 2 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_2.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 3 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_3.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 4 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_4.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 5 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_5.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 6 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_6.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 7 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_7.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 8 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_8.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 9 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_9.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 10 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_10.zip
nrfutil pkg generate --application files\buttonless.hex --application-version 11 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_11.zip

:: TODO: flash the dongle with firmware connectivity. Use e.g. nrfutil dfu ble -pkg files\dfu_image_2.zip -ic NRF52 -p COMX -n "Nordic_Buttonless" -f once before runnign this script. Be aware that flashing this may change the COM port for the dongle.

nrfutil dfu ble -pkg files\dfu_image_2.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_3.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_4.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_5.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_6.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_7.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_8.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_9.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_10.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"
TIMEOUT 5
nrfutil dfu ble -pkg files\dfu_image_11.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "Nordic_Buttonless"

BR,

Edvin

[simone.bonetti]

We're having hard time try to focus where the problem actually is, what i can tell you right now is the following:

• your script runs fine on nRF52840 DK eval board, no failures
• using nRF52833 DK we experienced some failures with untouched secure_bootloader\pca10100_s140_ble
• we never experienced failures using nRFConnect Ble 

After many tests, we're pretty confident that neither the target Hw, nor the target bootloader is responsible for the failures, it's something related to the PC side (libs or dongle fw).

We would really like to try to use nrfutils with an updated version of  pc-ble-driver-py library which links to the latest pc-ble-driver v4.1.2, rather than the current pc-ble-driver v4.1.1, is this possible?

[Edvin]

simone.bonetti said:

We would really like to try to use nrfutils with an updated version of  pc-ble-driver-py library which links to the latest pc-ble-driver v4.1.2, rather than the current pc-ble-driver v4.1.1, is this possible?

 I understand, but this is not something I possess. 

I tried to run a similar script using the nRF52833 as the target device:

Fullscreen test833.bat Download

set TARGET_SNR=685133991
set CONNECTIVITY_COM_PORT=COM5

del ..\..\..\private.key
del ..\..\..\dfu_public_key.c

nrfutil keys generate ..\..\..\private.key
nrfutil keys display --key pk ..\..\..\private.key --format code --out_file ..\..\..\dfu_public_key.c

make -j9
make -j9 -C ..\..\..\..\ble_peripheral\ble_app_uart\pca10100\s140\armgcc

mkdir files

del files\ble_uart.hex
copy ..\..\..\..\ble_peripheral\ble_app_uart\pca10100\s140\armgcc\_build\nrf52833_xxaa.hex files\ble_uart.hex

del files\bl_settings.hex
nrfutil settings generate --family NRF52 --application files\ble_uart.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 --key-file ..\..\..\private.key files\bl_settings.hex
del files\fake_settings.hex
nrfutil settings generate --family NRF52 --application _build\nrf52833_xxaa_s140.hex --application-version 1 --bootloader-version 1 --bl-settings-version 1 --key-file ..\..\..\private.key files\fake_settings.hex

nrfjprog -e --snr %TARGET_SNR%
nrfjprog --program ..\..\..\..\..\components\softdevice\s140\hex\s140_nrf52_7.0.1_softdevice.hex --verify --snr %TARGET_SNR%
nrfjprog --program _build\nrf52833_xxaa_s140.hex --verify --snr %TARGET_SNR%
::nrfjprog --program files\ble_uart.hex --verify --snr %TARGET_SNR%				COMMENTED OUT ON PURPOSE
::nrfjprog --program files\bl_settings.hex --verify --snr %TARGET_SNR%			COMMENTED OUT ON PURPOSE
nrfjprog --reset --snr %TARGET_SNR%

del files\dfu_image_2.zip
del files\dfu_image_3.zip
del files\dfu_image_4.zip
del files\dfu_image_5.zip
del files\dfu_image_6.zip
del files\dfu_image_7.zip
del files\dfu_image_8.zip
del files\dfu_image_9.zip
del files\dfu_image_10.zip
del files\dfu_image_11.zip

nrfutil pkg generate --application files\ble_uart.hex --application-version 2 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_2.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 3 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_3.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 4 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_4.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 5 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_5.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 6 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_6.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 7 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_7.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 8 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_8.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 9 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_9.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 10 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_10.zip
nrfutil pkg generate --application files\ble_uart.hex --application-version 11 --hw-version 52 --sd-req 0xCA --key-file ..\..\..\private.key files\dfu_image_11.zip

:: TODO: flash the dongle with firmware connectivity. Use e.g. nrfutil dfu ble -pkg files\dfu_image_2.zip -ic NRF52 -p COMX -n "Nordic_Buttonless" -f once before runnign this script. Be aware that flashing this may change the COM port for the dongle.

nrfutil dfu ble -pkg files\dfu_image_2.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_3.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_4.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_5.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_6.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_7.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_8.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_9.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_10.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

nrfutil dfu ble -pkg files\dfu_image_11.zip -ic NRF52 -p %CONNECTIVITY_COM_PORT% -n "DfuTarg"
TIMEOUT 5
nrfjprog --program files\fake_settings.hex --verify --sectorerase --snr %TARGET_SNR%
nrfjprog --reset --snr %TARGET_SNR%

Can you please try to copy this to the location:

SDK17\examples\dfu\secure_bootloader\pca10100_s140_ble\armgcc.

You need to call the script SDK17\external\micro-ecc\build_all.bat before you compile the bootloader using the script (but I guess you already know this if you compiled the bootloader earlier. 

Can you please try this in an unmodified SDK?

The reason I ask you to test these is not just to have you spend time on unnecessary tests, but I am trying to narrow down what the issue may be. Did you do any changes to the bootloader? When I ran this script, it didn't fail at any of the DFU updates. 

If you don't fail on any of the 10 tests in the script, can you try to zip a project + bootloader combination that reproduces the issue, and zip the projects and send them to me here? Please make sure that the projects compile when you unzip them in an unmodified SDK before sending them. 

BR,

Edvin