Deterministic Builds in nRF5 SDK using Segger Embedded Studio

[EAn]

Hello everybody!


I developed a way to perform differential updates (Delta FUOTA) and it depends heavily on the outputted binary resulting from the build. I compare the diffs between the two different build versions binaries and generate a patch file.

i noticed that when I have a small change in the code, say, a string that is written somewhere gets changed by N bytes. Ideally there should be N bytes of difference between one and the other version binaries. The reality is, the code optimizations and linker totally mess it up to the point that adding few dozen bytes in a string results in about 7kB of changes between one binary and the next.

My question is: Is there a way to implement a deterministic build system so that we can overcome this difficulty and produce smaller deltas? I am using Segger Embedded Studio with the recommended settings for the nRF5 SDK, in Linux, Mac OS and Windows. It is easier for me to experiment in Linux, though. My compiler is GCC as it comes in the stock SES+nRF5.


Thank you for precious insight!

Regards,
//Emanuel A.

[ovrebekk]

Hi Emanuel

I don't know of any easy way to change this unfortunately. If you disable optimizations you might get a more predictable build output, but at the cost of higher memory and CPU usage. 

It is possible to split your project into multiple separate linker sections. Then a smaller change in the code should at least not cause changes outside its own section. The problem of this method is that you complicate the project configuration significantly, and you would also need to add some overhead to all your sections which again will increase flash usage. 

In the nRF52 series there is a patch mechanism that allows you to fix bugs in the code by issuing small, incremental patch fixes. There is a sample available here that shows how this can be done. The problem again is that you introduce more complexity to your project, and you have to develop the application with this system in mind. Also, issuing a patch every time you want to increase the length of a string is not a very efficient way to do it.
Another word of caution is that this mechanism is not supported in the M33 core used in later Nordic devices, so if you want to migrate to a newer Nordic series in the future (like the nRF53 series) then the patch mechanism will no longer be available. 

Best regards
Torbjørn 

[EAn]

Hello Torbjørn,

Thank you for your swift reply. Yes, I see what you mean. The way we are producing patch files is using a binary diff tool. We then broadcast that binary over our network of IoT devices through a DFUOTA service that we implemented in our proprietary mesh network. Meaning, we already have a bit of complexity going on here...
My hopes with the string/array size example was just to use it as a quick test of how constant and consistent the binaries would be and the resulting diff when we apply a small change to the code. Obviously it isn't the use case for most firmware updates :) .
Do you have any pointer as to where I could learn how to tweak the linker to adapt to try to minimize binary code changes?

Thank you again!
Best regards,

Emanuel

[ovrebekk]

Hi Emanuel

EAn said:

Do you have any pointer as to where I could learn how to tweak the linker to adapt to try to minimize binary code changes?

No, I don't really have any more info on this unfortunately. 

The majority of our customers do DFU over standard BLE GATT connections, where sending the image in its entirety doesn't really make a big impact (the bottle neck in this case is the time it takes to erase the entire flash). 

I'm not involved with BT mesh much anymore, but last time I checked the solution here was to slowly trickle an update through the mesh over a long period while still keeping the mesh application running on the side, in order to make the slow update time less of an issue. 

Best regards
Torbjørn

[EAn]

Hello Torbjørn,

Thank you again for replying. Yes, we are aware of the other means of performing a DFUOTA , however we are optimizing the system to be able to reduce update times in a mesh scenario.

It's ok, I'll close this issue, for now.
If anyone has any ideas to contribute as to how we can have more consistent binary builds, please, let us know! :D

Thank you very much.

Best regards,
//E

[ovrebekk]

Hi Emanuel

You welcome, I wish I had some more concrete advice for you. With some luck the community can help out

In the mean time, the best of luck with your project. 

Best regards
Torbjørn