nrfutil generates wrong CRC

[XavierYin]

nrfutil settings generate" gives different CRC value from what bootloader  calculates. Besides, I used JLinkExe savebin to dump application from DUT and the calculated CRC matches bootloader's value. It looks like nrfutil is problematic.

macOS 10.14
SDK15
nrfutil version 4.0.0
Segger Embedded Studio 3.5.2

Attached files
https://drive.google.com/open?id=1g2Eo4dLErHefG9S3-j3Nha7JTjOGtNq7

Note:
1. app.elf & app.hex are generated by SES
2. setting_page.hex is generated by:
    nrfutil settings generate --family NRF52 --application app.hex --application-version 3 --bootloader-version 2 --bl-settings-version 1 --no-backup setting_page.hex
3. app_dump is by JLinkExe savebin app_dump 0x26000 0x00010698
4. nrfutil settings display setting_page.hex shows:
Bootloader DFU Settings:
* File: setting_page.hex
* Family: nRF52
* Start Address: 0x0007F000
* CRC: 0x1AEE538E
* Settings Version: 0x00000001 (1)
* App Version: 0x00000003 (3)
* Bootloader Version: 0x00000002 (2)
* Bank Layout: 0x00000000
* Current Bank: 0x00000000
* Application Size: 0x00010698 (67224 bytes)
* Application CRC: 0x243A6A75
* Bank0 Bank Code: 0x00000001
5. crc32 app_dump gives 758d7b9e, which is the same as what I see when debugging bootloader
6. Verify in SES, both by elf or intel hex will pass

[Vidar Berg]

Hi,

In Segger embedded studio, can you try to upload the .hex file instead of the .elf file? I suspect this is may be related to the observation I made here: https://devzone.nordicsemi.com/f/nordic-q-a/37507/buttonless-dfu-debugging-problem/144353#144353 

[XavierYin]

Hi Vidar Berg 
Do you use arm_linker_additional_output_file_gap_fill="0xff" in your .emProject so it fills the gap with 0xff?

By default, without filling gap as all SDK examples, it looks the same either by programming with elf or hex. But with gap filling 0xff I can observe the inserted bytes in hex and the dump will have the same CRC as setting page.

I think the problem is about nrfutil since it generates the same CRC for both version of hex, with or without gap filling. I suspect there nrfutil assumes gaps as 0xff, kind of following the fact that erased regions on flash contains 0xff. Thus whether filling gap with 0xFF makes no difference to it. And the cause for problem is SES programs elf by default and assumes 0x00.

[Vidar Berg]

Hi,

Thanks, I was not aware of this option. However, I've been using the default configuration. Also compared the elf and hex file you gave, they both result in the same data being loaded to flash. 

I was not able to reproduce the CRC error with the hex files you provided. I uploaded the bootloader settings pag and app.hex file in addition to stock bootloader and softdevice. CRC: 0x243A6A75 did match in my case. Do you have any code that could potentially modify data in the application region at runtime? Can you try the default bootloader and see if you get the same result?

Commands I used:

nrfjprog --program softdevice.hex --chiperase

nrfjprog --program bootloader.hex

nrfjprog --program app.hex

nrjprog --program settings_page.hex --sectorerase -r

 

[XavierYin]

I "thought" I was all default as you did. But perhaps I did missed somewhere.
(BTW, I program by SES instead of using nrfjprog. I guess they should have the same effect...

By loading & debugging w/ your setup, together with using both arm_linker_additional_output_file_gap_fill="0xff" and
arm_linker_script_generator_default_fill_pattern="0xff"
I'm now happy working with SES.
(I also used user build steps for setting page)

Sorry that at I'm now chasing the schedule and may not afford to do more experiments at least for a week. Thanks for Vidar Berg. I'll try to get back to this ticket when possible.

[Vidar Berg]

Ok good, so it works with SES now. Just let me know if you want to revisit this when you have more time and try finding the root cause. 

[mtsunstrum]

I think I am experiencing the exact same problem, but pretty close to concluding it is not a nrfutil problem.

Rather it seems to be a SES / J-Link flash programming issue.

---

I have been using SES 4.12 for over a year now. Our project has a DFU bootloader, and our application .emproject has been set to do a post-build generation of a setting.hex (CRC) file.

Via SES IDE, we load both application and settings.hex file via Debug->Go

So we can do a nice debug session, in the presence of our DFU Bootloader.

settings.hex is generated our batch file, which uses nrfutil to generate the updated file.

As mentioned, this has worked for a long time now. Flawless.

---

Feeling that I should upgrade SES, I upgraded to SES 4.18.

With NO changes in our project setup, we can no longer properly debug via SES. Code remains in DFU Bootloader due to CRC error.

Only differences between a 4.12 and 4.12 build is the SES provided thumb_crt0.s file. Minor differences. Tried replacing our 4.18's with the 4.12 thumb_crt0.s file ... but no difference.

Tried above discussion on fill patterns. Tried 0x00 and 0xFF. No difference.

Tried newer SES versions ... 4.50, 4.52b. No difference.

Interestingly, I merged my 4 hex files (Bootloader, Softdevice, settings.hex, and our Application), and flashed nRF52 device using nrfjprog, and everything works perfectly. 

---

The ONLY difference hence is how the nRF52 device is programmed. SES IDE will use J-Link utilities, not nrfjprog.

That is why Vidar Berg mentions his nrfjprog method always works.

But that is not convenient when using SES IDE ...

Possible that there is a weakness in the nrfutil calculations that are not tolerant to differences in how J-Link programs flash ... so solution may need to be a co-operative effort with Segger / Nordic.

---

Stepped back to SES 4.12 and works perfectly.

Appears to use J-Link 6.40 version ... whereas newer SES versions use newer J-Link (6.54c, 6.70, ...)

Hence, I have my strong suspicions that there is some subtlety in J-Link's programming that causes CRC error ...

[Vidar Berg]

Hello,

Another difference is that SES loads the *.elf output file instead of the *.hex output. So I still suspect it may be related to padding. Have you tried to read back the flash to see what the difference is?

Comparing flash with nrfjprog:

1. Program the app with SES then run "nrfjprog --memrd 0x0 --n 0x80000 > flash_dump_after_programming_with_ses.txt"

2. Program the same FW with nrfjprog and run "nrfjprog --memrd 0x0 --n 0x80000 > flash_dump_after_programming_with_nrfjprog.txt"

3. Run a 'Diff' on the two text files

Note that there is an option to easily disable CRC boot validation with the newer bootloaders. You just select no boot validation when you generate the settings page.

[mtsunstrum]

Thanks Vidar ... good to know about SES using .elf instead of .hex

Yes, I had on my notes to do a flash readback comparison.

I will revisit this early next week and give that a try.

Thanks, Martin

[mtsunstrum]

OK, I have time to get back at this odd issue.

As mentioned previously SES 4.12 and less works fine when flash programming via SES.

For any SES version greater than 4.12 I see this issue.  But SES 4.12 is kind of unstable, crashes a lot, so looking to move to a newer SES version.

My application memory settings are below:

FLASH_PH_START=0x0
FLASH_PH_SIZE=0x80000
RAM_PH_START=0x20000000
RAM_PH_SIZE=0x10000
FLASH_START=0x26000
FLASH_SIZE=0x52000
RAM_START=0x20002a98
RAM_SIZE=0xd568

In SES, I leave Linker "Default Fill Pattern" and "Additional Output File Gap Fill Value" to 'None'.

Proceeded to use nrfjprog to dump whole contents of Flash memory to see differences.

( I can supply complete dump files if needed)

In Application/Settings.hex file range, only differences are:

---

SES Flash programming of Application/Settings.hex file ... does not pass Bootloader CRC

0x00032310: 47704770 00004770 0002D37D 00000000 

0x00075000: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 

---

nrfjprog programming of Application/Settings.hex file ... works perfectly

0x00032310: 47704770 FFFF4770 0002D37D 00000000 

0x00075000: DEADC0DE F11E01FF FFFFFFFF FFFFFFFF 

---

Looking at address 0x00032310, map file indicates some code at end of SES libc library functions for heap_lock and heap_unlock.

.text.libc.strlen
0x00000000000322b2 0x60 C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 4.52b/lib/libc_v7em_fpv4_sp_d16_hard_t_le_eabi.a(libc2_asm.o)
0x00000000000322b2 strlen
.text.libc.__heap_lock
0x0000000000032312 0x2 C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 4.52b/lib/libc_v7em_fpv4_sp_d16_hard_t_le_eabi.a(libc.o)
0x0000000000032312 __heap_lock
.text.libc.__heap_unlock
0x0000000000032314 0x2 C:/Program Files/SEGGER/SEGGER Embedded Studio for ARM 4.52b/lib/libc_v7em_fpv4_sp_d16_hard_t_le_eabi.a(libc.o)
0x0000000000032314 __heap_unlock
0x0000000000032316 __text_end__ = (__text_start__ + SIZEOF (.text))

---

As for 0x00075000, I am not sure who is manipulating that. I assume that is partly from settings.hex, but I don't see DEADC0DE marking ... whatever that means.

---

If I change Linker "Default Fill Pattern" and "Additional Output File Gap Fill Value" to '0xFF', I can see it fill in some areas with FF.

But the behavior is the same. SES Flash programming fails, nrfjprog is happy.

A diff on their hex files shows the exact same differences as my dump above.

Note that fill 0xFF did not fill FF into address 0x00032314

My guess is this is the root of the issue ...

---

Any help from the flash programming / Bootloader CRC experts is much appreciated.

I don't think it is prudent for us to disable CRC checking. 

Thanks, Martin

[mtsunstrum]

Note that above information is when using SES 4.52b to program the flash memory (Application/Settings.hex)

If I go back to SES 4.12b and run the same tests, the nrfjprog dump shows they have perfectly matched dump contents byte for byte across the whole 512KB range.

[Vidar Berg]

Hello, I haven't had any luck replicating this with ses 4.52b. I made sure the __heap_lock and __heap_link part of the lib didn't get discarded by the linker. What I remember from earlier experiments with this is that the problem occurred only for some builds and that unrelated code changes could make it work again. So my guess was that it had to do with code alignment. Have you tried with and without code optimization enabled to see if it leads to the same result?

A possible solution is to make SES program the hex output instead and only load the debug information from the .elf output. Could you try that? The screenshots below show the two project settings you would need to change for this.

Program the generated *.hex instead of the *.elf file:

Load debug symbols from *.elf file

mtsunstrum said:

As for 0x00075000, I am not sure who is manipulating that. I assume that is partly from settings.hex, but I don't see DEADC0DE marking ... whatever that means.

It's the magic words used by FDS to tag its allocated flash pages, see Page tag. It's written at runtime by the fds_init(), so that's why you don't see it when the checksum validation fails.

mtsunstrum said:

I don't think it is prudent for us to disable CRC checking. 

 I understand. I wouldn't recommend disabling it for FW going into production. It could make it easier to work with the bootloader during development.

[mtsunstrum]

Thanks Vidar, your magic solved my problem. Setting up SES to program with the hex file, rather than it's default .elf file seemed to solve the problem for me.

I think we have finally gotten to the root of it.

To answer some of your questions:

• It would happen on both Debug (no optimizations) and Release builds
• It would happen only on SES versions greater than 4.12
• It would happen across different nRF52 applications, so did not seem to be related to the nature of the application being built.

Thanks again !