NRFutil DFU over BLE works the first time and then fails with error code 13

[Bart Nathan]

Hi Everyone, 

I'm using nrfutil in an arm cortex a8 running Debian 9 with kernel 4.14.49. The processor is connected to a NRF52 dongle through USB. I'm trying to flash (using DFU) a new application to a remote NRF52 chip through BLE. I can successfully send the application firmware one time, but if I try to repeat the operation I get an error. To get rid of the error I need to disconnect and re-connect the USB (NRF52 Dongle), then I can send one new application successfully, and after that the error repeats itself.

The software versions:
nrfutils was installed from pip and its version 3.5.1
pc-ble-driver-py was compiled from source and its version 0.11.3
The firmware running in the NRF52 usb dongle is the connectivity firmware example from the SDK 15.0 for softdevice 132, (named ser_s132v3_usb_hci).

The error output that I get (for the second attempt to send th DFU over BLE) is:

user@system:~$ nrfutil -v -v dfu ble -ic NRF52 -pkg ~/nrfutils/bt_data_logger_mr_pickle.zip -p /dev/ttyACM0 -n "Pickle 15791"
2018-09-20 11:43:57,826 Shared library: /usr/local/lib/python2.7/dist-packages/pc_ble_driver_py-0.11.3-py2.7.egg/pc_ble_driver_py/lib/linux/x86_32/libpc_ble_driver_shared_sd_api_v3.so
2018-09-20 11:43:57,830 Swig module name: pc_ble_driver_sd_api_v3
2018-09-20 11:43:58,279 Using connectivity board at serial port: /dev/ttyACM0
2018-09-20 11:43:58,319 Sending Application image.
Traceback (most recent call last):
File "/usr/local/bin/nrfutil", line 11, in <module>
sys.exit(cli())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/nordicsemi/__main__.py", line 840, in ble
dfu.dfu_send_images()
File "/usr/local/lib/python2.7/dist-packages/nordicsemi/dfu/dfu.py", line 141, in dfu_send_images
self._dfu_send_image(self.manifest.application)
File "/usr/local/lib/python2.7/dist-packages/nordicsemi/dfu/dfu.py", line 97, in _dfu_send_image
self.dfu_transport.open()
File "/usr/local/lib/python2.7/dist-packages/nordicsemi/dfu/dfu_transport_ble.py", line 450, in open
self.dfu_adapter.open()
File "/usr/local/lib/python2.7/dist-packages/nordicsemi/dfu/dfu_transport_ble.py", line 106, in open
self.adapter.driver.open()
File "/usr/local/lib/python2.7/dist-packages/pc_ble_driver_py-0.11.3-py2.7.egg/pc_ble_driver_py/ble_driver.py", line 127, in wrapper
raise NordicSemiException('Failed to {}. Error code: {}'.format(wrapped.__name__, err_code))
pc_ble_driver_py.exceptions.NordicSemiException: Failed to open. Error code: 13

Any idea of where this error might come from? I tried to reset the NRF52 dongle, but it does not solve the problem. Any help would be very appreciated. 

[Vidar Berg]

Hi, 

It sounds like it may be related to issue posted in this thread: https://devzone.nordicsemi.com/f/nordic-q-a/29059/pc-ble-driver-py-failed-to-open. As a test, could you try to pull the CTS line low and see if it helps?   

[Bart Nathan]

Hi Vidar, 

First of all, thanks a lot for your help. The thread you are pointing to is using a PCA10000 dongle, with an NRF51 chip and a Segger programmer on the same board. I am using the NRF52 PCA10059 dongle, which does not have a Segger programmer chip on it. Therefore, the programming is done through USB DFU. Meaning that I do not have a UART interface, where I could pull the CTS line low. 

What else could we try? Thanks again for the help.

[Bart Nathan]

You asume correctly, I am building the connectivity FW myself. I tried the .zip file (using nrfutil for USB DFU), as you recommended, but the results are the same. I can successfully update the remote NRF52 the first time, but after that the following attempts fail with the same error as before. 

What else could we check?

Thanks for the help. 

[Vidar Berg]

Unfortunately, I have not be able to reproduce the problem yet making troubleshooting a bit hard. I used the 52840 dongle with Ubuntu running in VirtualBox. I will try the same on a raspberry pi since I don't have anything with A8 that I can use for test.

Some follow up questions:

1. Is the serial port listed in /dev/ after the first attempt. I think it should be based on the error code, so this is just to confirm this assumption. 

2. Is the failing consistent. I.e., does it always fail on the 2nd attempt, or have you seen cases where it fails after 3 attempt,etc?

3. Is it possible to check if the port is kept open after the first attempt?  

4. What version of boost did you use for building the driver? 

[Bart Nathan]

I'm using a beaglebone black (the version with wireless and the 'Octavo Systems' chip). I could prepare an sd card image for you to replicate my setup. Would that work for you?

1 - The serial port is always listed, it does not disappear. 

2 - The failing is quite consistent,  maybe once every 20 attempts it will manage to work two times in a row, but most of the times it fails in the second attempt.

3 -  I am not sure how to check this. If I open another python instance, I can correctly open the port, not sure this is a reliable experiment to deduce if the port was closed correctly or not. Any ideas how I could check for this?

4 - The boost version used to build the driver is 1.68.0

[Vidar Berg]

I did not manage to reproduce this on the raspberry pi 3 with pc-nrfutil (release/v.4.0.0) and pc-ble-driver-py (release/v.0.11.3). Maybe you could you try to build these instead and see if you get the same result? 

I don't have a beaglebone black wireless board, but will try to get hold of one.     

[Danius]

Hello Vidar,

Bart is on a holiday, so I will be tracking this issue now.

We will try to build the pc-nrfutil version that you specified and let you know if it works.

Thank you for your support.

[Vidar Berg]

Hello,

Sounds good. I've ordered a beaglebone and will try the same here when I get it. 

[Danius]

I tried to run the DFU with pc-nrfutil (release/v.4.0.0). Unfortunately I'm still getting the same error.

A few things I tried as well and found some workarounds:

• Resetting (using the reset switch) the dongle between DFUs - works if I reset it every time.
• Running a small python script which just opens and closes the serial port between DFUs - works if I run the script every time.
• Remove the USB power (using a MOSFET) and connect it between DFUs - works well.

The first two options are not as reliable as the last one, because if the workaround is not applied between DFUs, the error starts occurring again and then the workarounds don't work anymore. The last one is reliable and allows to recover from the error state as well.

As I said, these are workarounds and it would be great to find the real problem. Hopefully this information will help you.

Sounds great about the beaglebone, will be interesting to see what kind of results you get.

Let us know if there is anything else we can provide you with.

[Vidar Berg]

Thanks for confirming, and for the additional details, it's definitely useful information. The developers suspect it might be timing related, either in the connectivity FW or in the host implementation, but it is difficult to track down unfortunately. There was a similar issue reported here where the problem actually got resolved by updating the kernel itself, but I doubt it is a kernel issue. 

I've confirmed this exact behavior on the beaglebone black wireless board which is not the outcome I had hoped for. I hoped it would work if I tried to build everything from scratch on a fresh install. 

[Danius]

Thanks for letting me know. I saw that similar issue and was hoping that we wouldn't need to start playing with our kernel. Hopefully the developers will be able to identify where the issue is coming from.

Please keep us posted with your findings.

[Vidar Berg]

I found this interesting comment in uart_boost.cpp (on master branch):

serialPort->open(portName);

// Wait a bit before making the device available since there are problems 
// if data is sent right after open.
//
// Not sure if this is an OS issue or if it is an issue with the device USB stack.
// The 200ms wait time is based on testing with PCA10028, PCA10031 and PCA10040.
// All of these devices use the Segger OB which at the time of testing has firmware version
// "J-Link OB-SAM3U128-V2-NordicSemi compiled Jan 12 2018 16:05:20"
//std::this_thread::sleep_for(std::chrono::milliseconds(200));

So I added a delay after port open like the comment suggested and it seems to have fixed the issue. I performed a loop test (100x) and got zero failures. Can you try the same on your side to confirm?  

My pc-ble-driver-py build:

outdir.zip

Note: I wasn't sure how to use the sleep_for() function, so just used a for loop instead. 

Update: std::this_thread::sleep_for(std::chrono::milliseconds(200)); depends on chrono.h and thread.h. Should work if you add those includes to uart_boost.cpp.