merged.hex file built from the same code base is different.

[Learner]

Hello,

I noticed that if I rebuild the merged.hex file from exactly the same code base then all subsequent merged.hex files are different. This is causing us a problem at production.

I am guessing/suspecting this because the file signature/CRC has changed and the reason for its change is something like the timestamp.

Attached is a snapshot of the differences reported by 'Beyond Compare' utility. All the differences are at the end of the merged.hex file.

Q1/ Can you please explain why this is happening?

Q2/ Could you please point me to a document detailing the format of the merged.hex file.

Q3/ Is there a utility that checks only the code content of the merged.hex without including timestamp, CRC, Signature etc...?

Thank you.

Kind regards

Mohamed

[AHaug]

Hi

Q1/ Can you please explain why this is happening?

It is as you said due to your firmware beeing modified since you have a timestamp that changes from build to build

Q2/ Could you please point me to a document detailing the format of the merged.hex file.

Q3/ Is there a utility that checks only the code content of the merged.hex without including timestamp, CRC, Signature etc...?

I will have to ask around some more to see if we have something for this

I will get back to you before the weekend.

Kind regards,
Andreas

[Learner]

Good Morning Andreas,

Any update on my two questions above?

Kind regards

Mohamed

[Learner]

Hi Andreas,

I am sorry to keep chasing you about the answers I am looking for.

I am stuck on this and I need help from someone from Nordic.

Thank you.

Kind regards

Mohamed

[AHaug]

Hi,

First of apologies for the far too long response time. It took longer than expected to find an answer for the two questions

In the general case, the signature schemes are non-deterministic so the difference in the binary is to be expected.

In addition, if you're using development keys (for NSIB, MCUBoot uses the same key every time), the keys are also re-generated when building from scratch 
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/config_and_build/bootloaders_and_dfu/fw_update.html#using-development-keys.

Given the signature is non-deterministic, one way to ensure two builds have created the same firmware is to do these two steps:

1. Strip the signature and hash before comparing the firmware itself.
2. Use the imgtool verify command to check that the signature of both images can be verified with the same key.

This option is also suggested in https://reproducible-builds.org/docs/embedded-signatures/ 

This case discusses the topic above and you can use that as a reference  MCUBoot - Firmware signature looks different at each build  

The Merged.hex is also a composition of different files and we recommend that you compare it to zephyr.hex to look closer into where the difference occurs

Once again, apologies for the long response time and I hope this answers your questions sufficiently

Kind regards,
Andreas

[Learner]

Hi Andreas,

Thank you for all your efforts.

AHaug said:

Strip the signature and hash before comparing the firmware itself.

Do you mean I manually strip the part of the output build files that show differences built from the same code base i.e merged.hex, app_update.bin and app_signed.hex?

AHaug said:

Use the imgtool verify command to check that the signature of both images can be verified with the same key.

I am not familiar with imgtool, so I had a look at the case you shared with me where Benjamin V used the command line below,

imgtool verify --key <key.pem> app_update.bin

Where do I get the key.pem file from ?

AHaug said:

The Merged.hex is also a composition of different files and we recommend that you compare it to zephyr.hex to look closer into where the difference occurs

I compared the files merged.hex and zypher.hex from the same build and I am no wiser.

Thank you.

Kind regards

Mohamed

[AHaug]

Hi,

Learner said:

Do you mean I manually strip the part of the output build files that show differences built from the same code base i.e merged.hex, app_update.bin and app_signed.hex?

That is my understanding of it, yes

Learner said:

I am not familiar with imgtool, so I had a look at the case you shared with me where Benjamin V used the command line below,

imgtool verify --key <key.pem> app_update.bin

I recommend that you have a look at https://github.com/hellesvik-nordic/samples_for_nrf_connect_sdk/tree/main/bootloader_samples/keys_and_signatures, https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/mcuboot/imgtool.html and  https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/config_and_build/bootloaders_and_dfu/fw_update.html#ug-fw-update-keys-imgtool to familiarize yourself regarding imgtool

Learner said:

Where do I get the key.pem file from ?

Some default keys can be found in <sdk version>/bootloader/mcuboot but we recommend that you generate your own keys following the documentation to do so https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/config_and_build/bootloaders_and_dfu/fw_update.html#generating-private-keys

If you have your own custom keys they are in CONFIG_BOOT_SIGNATURE_KEY_FILE

Kind regards,
Andreas