Reproducible build

RE: Reproducible build

Einar Thorsrud — Wed, 26 Oct 2022 14:14:51 GMT

Good to hear! Thanks for letting us know.

RE: Reproducible build

vcbz — Wed, 26 Oct 2022 13:46:51 GMT

Ok, it seems the binaries are now identical :) It was just that my OS was using another CMAKE version. So basically to reproduce the builds one has to check that the same version bundled by the NRF Connect Desktop app in YOUR_NCS_INSTALL_DIR/YOUR_VERSION/zephyr/opt/bin/cmake (currently cmake 3.20.5) is used.

RE: Reproducible build

vcbz — Wed, 26 Oct 2022 12:23:54 GMT

> A last point is that you need to build with the exact same tools in order to get the same binaries

Yes, this is what I would think happens in theory in my ci/cd. I have checked the variables passed to cmake. Snippet as ref:

get_cmake_property(_variableNames VARIABLES)
list (SORT _variableNames)
file(WRITE vars.txt)
foreach (_variableName ${_variableNames})
    file(APPEND vars.txt "${_variableName}=${${_variableName}}\n")
endforeach()

I found out in my west native build without vs code, it uses my CMake installation in C:\Prog Files\Cmake

After uninstalling cmake, and setting the binary path to C:\MY_NCS_SDK_PATH\v2.0.0/opt/bin, I get identical builds between
west and VS code. I will attempt to fix this in the CI/CD and check if it works.

And about the requirement, is that we would like to have reproducible builds between CI/CD and local builds. It shouldnt matter if a developer uploads the same compiled source code of a specific commit from local machine or server.
This also allows to debug the same software release if uploaded from local machine or if it was uploaded via OTA from the gitlab server artifacts.

RE: Reproducible build

Einar Thorsrud — Wed, 26 Oct 2022 12:11:39 GMT

Hi,

I see you have resolved most of this. A last point is that you need to build with the exact same tools in order to get the same binaries, so the only thing that seems sensible is to also use the docker image in your local builds.

This seems like a slightly odd requirement though, particularly for development. Out of curiosity, can you share why you want this? I would suggest to keep a local docker image and use that for builds whenever you need something that would be binary identical, but use VS Code with the toolchain from the toolchain manager for every day development and debugging (for simplicity).

RE: Reproducible build

vcbz — Tue, 25 Oct 2022 21:48:04 GMT

Ok I have found something interesting. I built the project without the gnu build id to try to dissect the difference between the server and local build.

the changes to the project are in the branch simple

https://git.fh-aachen.de/vc9917e/build_id_test/-/tree/simple

What I found out is that if I build the project directly with west from the SDK install dir made by NRF-Connect Desktop I get an almost 100% same binary.

This is the command

west build --build-dir MY_BUILD_DIR THE_PROJECT_DIR --pristine --board nrf52833dk_nrf52833

The only difference with the zephyr.bin generated by the ci/cd server is that the zephyr banner at boot is different. Compiling with the server I only obtain the abbreviated hash commit and with the NRF Connect SDK (obtained via NRF Connect desktop app) on my local machine it retrieves the tag version.

https://github.com/zephyrproject-rtos/zephyr/blob/3702f7220affc5bcf377c4e6c41fc320423d5045/kernel/banner.c#L38

I guess this is due to how zephyr gets the BUILD version which is obtained via "git describe" and this soley depends on the depth the repo was cloned.

https://github.com/zephyrproject-rtos/zephyr/blob/604f705fda149dea73e5fc830975e2ccefa994c8/cmake/gen_version_h.cmake#L9

However, if I build via the NRF VS Code plugin the binary changes a lot more. The only difference I see is that the VS Code plugin adds to the west build command the following: "-DNCS_TOOLCHAIN_VERSION:STRING="NONE". I have to clarify that the binary size is the same, which after analyzing briefly the assembly seems that some sections of the .text file are moved. This in turn makes references to functions/data change in dissassembly (objdump -d zephyr.elf)

RE: Reproducible build

vcbz — Tue, 25 Oct 2022 19:36:28 GMT

Second update. Using -fdebug-prefix-map solved the difference in debug path if compiling the project in diff. directories on the same machine (i.e., gitlab ci/cd). However, the gnu build id is still different between local build and server build

zephyr_compile_options(-fdebug-prefix-map=${CMAKE_BINARY_DIR}=BUILD_DIR
-fdebug-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=SRC_DIR
)

RE: Reproducible build

vcbz — Tue, 25 Oct 2022 14:47:03 GMT

I have an update. I madea new branch and changed the build directory and the build id has changed

https://git.fh-aachen.de/vc9917e/build_id_test/-/compare/main...build_dir_change?from_project_id=26912

This means that even just changing the build dir affects the binary even though its the same project, toolchain and sdk.