Hard fault when using NVS with Zephyr on nRF52

[NathanML]

I'm attempting to write to flash memory on our custom board using the nRF52840 using the NVS library in Zephyr 1.5.0. I've ported over the example code from samples/subsys/nvs and nvs successfully initializes:

[00:00:01.011,199] <inf> fs_nvs: 3 Sectors of 4096 bytes
[00:00:01.011,535] <inf> fs_nvs: alloc wra: 0, f40
[00:00:01.011,901] <inf> fs_nvs: data wra: 0, 133

The storage file system structure is initialized identically to the example code.

But, when I go to actually save data, I get a hard fault, specifically MPU Fault:

[00:00:04.200,897] <err> os: ***** HARD FAULT *****
[00:00:04.201,416] <err> os:   Fault escalation (see below)
[00:00:04.201,934] <err> os: ***** MPU FAULT *****
[00:00:04.202,423] <err> os:   Data Access Violation
[00:00:04.202,911] <err> os:   MMFAR Address: 0xf8130
[00:00:04.203,460] <err> os: r0/a1:  0x4001e400  r1/a2:  0x00000000  r2/a3:  0x4001e504
[00:00:04.204,223] <err> os: r3/a4:  0x01ffffff r12/ip:  0x20000aa4 r14/lr:  0x000053f3
[00:00:04.205,017] <err> os:  xpsr:  0x81000018
[00:00:04.205,535] <err> os: Faulting instruction address (r15/pc): 0x000054d4
[00:00:04.206,176] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:04.206,848] <err> os: Fault during interrupt handling

[00:00:04.207,397] <err> os: Current thread: 0x20001b60 (unknown)
[00:00:04.223,052] <err> fatal_error: Resetting system

My prj.conf file includes the same lines from the example, specifically CONFIG_MPU_ALLOW_FLASH_WRITE=y which I saw was mentioned as a fix on some git tickets.

This is all on a custom board, with the devicetree based on the nRF52840DK and successfully working with our board specific peripherals. Nothing in the flash memory section has been changed.

Also strange is that when running the default nvs code on the dev kit, I get this error: <err> flash_nrf: not word-aligned, followed by the memory address, for every attempted write. 

Thanks in advance for your help!

[Simon]

NCS Tutorial-->Error handling-->Other tools and tips-->Hard faults may help getting to the bottom of this.

Best regards,

Simon

[NathanML]

Hi Simon,

Thanks for the response!

As I'm already seeing the full log output, similar to the hard fault example, I ran the addr2line command in my <application>/<build folder>/zephyr but did not receive a useful response: 

$ addr2line -e zephyr.elf 0x000054d4
:?

The fault occurs when I call the nvs_write function. Using breakpoints in nvs.c I see that the underlying function causing the hard fault is flash_write (as called by nvs_flash_al_wrt, which was called by nvs_flash_data_wrt, which was called by nvs_flash_wrt_entry).

I'm not sure where to go from here. Based on your tutorial link to hard faults, and note that mentions they are often do to a stack overflow, I increased CONFIG_MAIN_STACK_SIZE to give that a try, but the results were the same.

Also, could you address the word-alignment error I'm getting with the default nvs sample code on the nRF52840 development kit? I'd like to avoid this error on our custom hardware as well, once we can get flash writes working.

I appreciate your help!

[Simon]

Also strange is that when running the default nvs code on the dev kit, I get this error: <err> flash_nrf: not word-aligned, followed by the memory address, for every attempted write. 

I tested the NVS example (NCS v1.5.1) on an nRF52840 DK and was not able to reproduce this. Could you follow the exact same steps as me and see if it still happens:

• Install NCS v1.5.1 through the Toolchain Manager
  •  Installing the Toolchain manager
  • Installing the nRF Connect SDK
• Open bash as expained in 1.3.2 Build and flash
• Redirect to ncs/v1.5.1/zephyr/samples/subsys/nvs
• Connect the nRF52840 DK to the PC and open Termite
• Run these commands
  • nrfjprog --eraseall
  • west build -b nrf52840dk_nrf52840 -d build_52840
  • west flash -d build_52840
• Then you should see the follwoing output:

*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 98
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,239] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: alloc wra: 0, cb0[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: data wra: 0, 228[1B][0m
Reboot counter history: ...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1
*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 99
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,270] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,300] [1B][0m<inf> fs_nvs: alloc wra: 0, ca8[1B][0m
[00:00:00.258,300] [1B][0m<inf> fs_nvs: data wra: 0, 22c[1B][0m
Reboot counter history: ...99...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1
*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 100
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,239] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,239] [1B][0m<inf> fs_nvs: alloc wra: 0, ca0[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: data wra: 0, 230[1B][0m
Reboot counter history: ...100...99...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1

Please test the above and report back what result you got

Best regards,

Simon

[NathanML]

Hi Simon,

Thanks again for the response. I was able to solve the word alignment issue in the example code by running nvs_clear. I'm guessing that I didn't erase the flash prior to migrating from my previous FDS usage with the old sdk and for some reason it was trying to initialize writes at an unaligned address.

I fixed my issue regarding the hard fault with nvs on our custom board with our custom application. When I disabled BT (commented out CONFIG_BT=y) nvs worked. I did some digging and saw that enabling BT changes some flash configuration options. I worked under the assumption that BT was probably using enough flash space to cause issues with the 3 pages I was trying to initialize for NVS.

I fixed the issue by increasing the amount of space for the storage partition; I added the following line to my prj.conf: CONFIG_PM_PARTITION_SIZE_NVS_STORAGE=0x8000 (the default is 0x6000).

So, for anyone stumbling across this thread: if you get a word alignment issue, make sure your flash is erased first, or run nvs_clear. If you are experiencing a hard fault/MPU fault and are using the BT module along with NVS, set the config for nvs storage to a larger value (as shown above).

Something that I think could be helpful for this sort of scenario is documentation outlining dependencies/interferences between modules. I had to do a lot of KConfig digging as well as a diff between my autogenerated configs.c files between our custom application and the sample nvs application to get an idea of what could be going wrong. It's also possible this sort of issue is documented somewhere and I just couldn't find out where. Just my two cents. Anyways, I appreciate your help, Simon! 

[Simon]

Hi Nathan,

I'm happy you were able to resolve the issue, and thanks for sharing the solution, wich may benefit other people in the future. I will share your feedback with some developers.

[Simon]

Regarding "error: <err> flash_nrf: not word-aligned", check out this PR: https://github.com/zephyrproject-rtos/zephyr/pull/34592. You should also set CONFIG_MPU_ALLOW_FLASH_WRITE=n.

NathanML said:

. I'm guessing that I didn't erase the flash prior to migrating from my previous FDS

FDS and NVS ar completely different, so there won't be any compatibility between these.

Best regards,

Simon

[NathanML]

Hi Simon,

That's good to know, thanks!

What is the purpose of setting CONFIG_MPU_ALLOW_FLASH_WRITE to n instead of y?