Custom DFU transport (sdk 12)

[Ray]

Hello. I am at the point in my project (using sdk v12) where I need to get DFU implemented.

In my case, DFU must occur through an alternative transport that is none of the 'usual suspects' of BLE, serial, etc.

Having struggled to get DFU implemented at all in the past, I am extremely impressed by how you've generalized the DFU transport mechanism. I don't understand it all yet, but the modularity is really slick.

I'm also incredibly impressed with the new secure code signing mechanisms. Such an improvement, and really the right thing to do for the industry.

That said, the actual DFU mechanisms are still pretty much a black box to me at this point, and given the complexity I think it's best that I treat it like a black box as much as I can - focusing on transport and not on the logic performed to actually unpack the image and do the DFU.

As a starting point, I've got DFU in my project implemented to the point where it (now, finally) works flawlessly with BLE dfu from my iphone.

For my custom DFU, the state is that:

1. I've got an external storage device attached to the UART that is addressable with AT commands

2. The only AT command relevant here is that I can ask the device to stream the file contents into the nrf52 over the uart. The file contents is streamed inward in variable sized chunks, from beginning-of-file to end-of-file. It is sequential fetch only; no random seek allowed.

3. Before I ever get into the bootloader to begin the DFU, the ZIP file for the image has been downloaded into that device. The bootloader will go into buttonless DFU mode if and only if the AT commands indicate that the file is present on that device.

And so, ideally, my task will be to "simply" (!) enhance the bootloader to have a custom transport that does as little logically as possible other than to read and feed the file into someone else's already-debugged logic to process it and to "do what the phone does", ultimately generating and calling nrf_dfu_req_handler_on_req's to process what's in the zip file and do the dfu.

Given that I am about to embark upon the process of reverse engineering the IOS DFU library to see what the heck it's doing and how it works, it occurs to me that I should just stop and ask you:

Do you have a different alternate transport written yet (such as UART), even if pre-release, that I can use as a starting point rather than to do this extremely complicated reverse engineering?

Your assistance would be greatly appreciated because this task is looking to be a bit daunting at this point.

Thanks much.

[Hung Bui]

Hi Ray,

UART Bootloader is available since SDK v6. However with the introduce of Secure Bootloader in SDK v12, we removed UART as it's not been updated to secure bootloader yet.

So if you need UART bootloader, I would suggest to use the Legacy bootloader from SDK v11.

However, in your case, the DFU master side is simply a storage device with AT command ? Then you may need to modify the bootloader a little bit as the storage device may not be able to handle the DFU starting sequence. I don't think you really need to "reversed engineering" the DFU master, but simply follow what we described in our documentation to make your DFU master or to modify the bootloader to cope with your setup. The documentation for UART bootloader is here.

[Ray]

Hi. Thanks for your time.

My situation, in greater detail, is:

• the developer does a 'git push' of updated builds to a special repo on our 'base service. From this moment on, everything happens in an unattended manner.

• Each of a widely distributed set of devices poll the service once nightly to check if an update is pending for the firmware of its specific configuration. If such an update is available, the download is performed by our nRF app.

• the download is done over 3G cellular. This is all run by a tiny nonprofit, and so it is important that the firmware is small. And so we are happy that nrfutil generates a ZIP file, and this is why we distribute the application only. The firmware was delivered 100% in the clear, and we don't trust link level encryption anyway, and so we love the new digitally signed images.

• The download is done over FTP and this ZIP file is buffered entirely within the 3G cellular module's own 2GB flash drive (a SIMCOM 5320, FYI).

• after download, each of these solar-powered devices will wait until it has sufficient battery life to ensure a complete DFU. It may be immediate, or it may be days. When it is ready, the nRF app reboots the device.

• Now back in the bootloader, we initialize serial and use it to talk with the 3G module to look for the new firmware ZIP file. If it isn't present, we just start the app. If the ZIP file is present, we will begin the DFU process. In essence this implements a buttonless DFU. (By the way, I really like your use of _WEAK methods to allow me to override things like the "should start" method.)

• Unfortunately, the only transfer method from the 3G module's Flash drive and the nRF is to initiate a stream of bytes from the zip file, from beginning to end, with no random access or flow control. As such, I have configured it to transfer very slowly (9600bps), and it uses app_sched of course to allow the bootloader to process it in 512-byte chunks.

All the code above is completed and tested. As you can see, there is no "controller" or "master" of the DFU process; the device must update itself given nothing more than a ZIP file.

And so given that I need to decode this zip file on the fly, and verify its integrity and digital signature, and to do the DFU, all inside the bootloader, I'm trying to come up with a plan as to what code I can borrow/transliterate and what I need to write.

Thanks again for taking the time to read and assist.

[Hung Bui]

I understand Ray. Since you have some constrains as you described, you will need to implement almost your own bootloader. You can re-use what we got to store data to swap page, check CRC, then replace old image with new one. That's it. Other than that you may need to implement your own protocol for receiving the init packet, the image, etc. I would suggest you to have a look at the UART bootloader we have as the base for your development.

Regarding the .zip, we usually don't do un-zip on the nRF5, but on the phone (DFU Master) instead. You should choose to do un-zip and hex to bin converting on the device with less constrain on power consumption. Maybe on your 3G module ? What you store in the flash driver is simply the binary image (and the init data if needed) that the nRF5 should flash into its ROM.

[Hung Bui]

Hi again Ray,

I had a discussion with one of our coworkers, Vidar. It seems that it's actually possible to reuse most of the secure bootloader. As you already noticed we tried to create the DFU with transport agnostic in mind. We only have one type of transport medium for now (BLE). But we think it's possible to replace the transport layer with your own custom transport. In your transport layer, you can instead of waiting for command from "DFU master", the NRF5 actually read the flash and check if there is a new image or not, if there is, it continue to read flash to get the init packet, ( the signature, CRC, size) and continue to call NRF_DFU_OBJECT_OP_EXECUTE so the upper layer can check the packet as it is done now with the nrf_ble_dfu.c and dfu_reg_handling.c (upper layer).

We are working on the UART transport layer for the secure bootloader now, most likely it will come with SDK v13 production, but I don't know when.

[Ray]

Interesting thought, and thanks for continuing to brainstorm about this.

Yesterday I did some analysis and realized that because the .BIN is basically uncompressable, there isn't much on-air savings for sticking with ZIP. And so I deconstruct the ZIP on the service, discard the manifest, and send the .DAT and .BIN down to the device uncompressed. This will make it easier for the bootloader.

I am now traveling but I will indeed try to find the time to look at and understand the existing code. Your thoughts sound fascinating. Thanks again.