Hard fault when using NVS with Zephyr on nRF52

RE: Hard fault when using NVS with Zephyr on nRF52

Simon — Fri, 07 Jan 2022 07:43:26 GMT

Could you open a new case and explain your problem there? Then it will get a assigned to an engineer that will take care of it

Best regards,

Simon

RE: Hard fault when using NVS with Zephyr on nRF52

DaWaN — Thu, 06 Jan 2022 16:45:09 GMT

I further checked my MPU fault and the MMFAR address was pointing at the stack and remarkable close to the edge of it.
Changed my main stack size to 2048 (CONFIG_MAIN_STACK_SIZE=2048) and this fixed the issue for my board config.
Not sure what inside my board config lead to the higher stack usage, but this solved my problem.

RE: Hard fault when using NVS with Zephyr on nRF52

DaWaN — Wed, 05 Jan 2022 20:38:32 GMT

I am suffering from the same issue, getting Hard Faults on the MPU when using NVS.
I am using an NRF52DK board with nRF52832. I am using the nRF Connect SDK V1.8.0
When I build the NVS samples they work when using the nRF52DK board config, but they do not work with my own board config.
The issue occurs within the function nvs_write when it starts requesting the MPSL timeslot. The program counter also matches the MPSL library when checking the map file. So the issue seems MPSL related.
I have been digging through the config files, comparing the different board configs and almost everything seems identical. I noticed only one warning and I think it is related to the issue. The warning I am seeing is this: # CONFIG_TEMP_NRF5_MPSL is not set

This is the logging I get when the issue gets triggered:

<err> os: ***** HARD FAULT *****
[00:00:01.050,384] <err> os:   Fault escalation (see below)
[00:00:01.050,415] <err> os: ***** MPU FAULT *****
[00:00:01.050,415] <err> os:   Data Access Violation
[00:00:01.050,415] <err> os:   MMFAR Address: 0x20005bd4
[00:00:01.050,415] <err> os: r0/a1:  0x00008672  r1/a2:  0x0000003b  r2/a3:  0x00000000
[00:00:01.050,415] <err> os: r3/a4:  0x00000036 r12/ip:  0x000086ad r14/lr:  0x00000000
[00:00:01.050,445] <err> os:  xpsr:  0x61000000
[00:00:01.050,445] <err> os: Faulting instruction address (r15/pc): 0x00000b6a
[00:00:01.050,445] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0

Would be nice if someone with more knowledge about the MPSL library can shed some light on this, as I would like to have this problem solved.

RE: Hard fault when using NVS with Zephyr on nRF52

NathanML — Tue, 11 May 2021 22:09:54 GMT

Hi Simon,

That's good to know, thanks!

What is the purpose of setting CONFIG_MPU_ALLOW_FLASH_WRITE to n instead of y?

RE: Hard fault when using NVS with Zephyr on nRF52

Simon — Tue, 11 May 2021 10:58:47 GMT

Regarding "error: <err> flash_nrf: not word-aligned", check out this PR: https://github.com/zephyrproject-rtos/zephyr/pull/34592. You should also set CONFIG_MPU_ALLOW_FLASH_WRITE=n.

[quote userid="101674" url="~/f/nordic-q-a/74789/hard-fault-when-using-nvs-with-zephyr-on-nrf52/308815#308815"]. I'm guessing that I didn't erase the flash prior to migrating from my previous FDS[/quote]

FDS and NVS ar completely different, so there won't be any compatibility between these.

Best regards,

Simon

RE: Hard fault when using NVS with Zephyr on nRF52

Simon — Tue, 11 May 2021 10:00:10 GMT

Hi Nathan,

I'm happy you were able to resolve the issue, and thanks for sharing the solution, wich may benefit other people in the future. I will share your feedback with some developers.

RE: Hard fault when using NVS with Zephyr on nRF52

NathanML — Fri, 07 May 2021 22:27:26 GMT

Hi Simon,

Thanks again for the response. I was able to solve the word alignment issue in the example code by running nvs_clear. I'm guessing that I didn't erase the flash prior to migrating from my previous FDS usage with the old sdk and for some reason it was trying to initialize writes at an unaligned address.

I fixed my issue regarding the hard fault with nvs on our custom board with our custom application. When I disabled BT (commented out CONFIG_BT=y) nvs worked. I did some digging and saw that enabling BT changes some flash configuration options. I worked under the assumption that BT was probably using enough flash space to cause issues with the 3 pages I was trying to initialize for NVS.

I fixed the issue by increasing the amount of space for the storage partition; I added the following line to my prj.conf: CONFIG_PM_PARTITION_SIZE_NVS_STORAGE=0x8000 (the default is 0x6000).

So, for anyone stumbling across this thread: if you get a word alignment issue, make sure your flash is erased first, or run nvs_clear. If you are experiencing a hard fault/MPU fault and are using the BT module along with NVS, set the config for nvs storage to a larger value (as shown above).

Something that I think could be helpful for this sort of scenario is documentation outlining dependencies/interferences between modules. I had to do a lot of KConfig digging as well as a diff between my autogenerated configs.c files between our custom application and the sample nvs application to get an idea of what could be going wrong. It's also possible this sort of issue is documented somewhere and I just couldn't find out where. Just my two cents. Anyways, I appreciate your help, Simon!

RE: Hard fault when using NVS with Zephyr on nRF52

Simon — Fri, 07 May 2021 10:09:28 GMT

[quote user=""]Also strange is that when running the default nvs code on the dev kit, I get this error: <err> flash_nrf: not word-aligned, followed by the memory address, for every attempted write. [/quote]

I tested the NVS example (NCS v1.5.1) on an nRF52840 DK and was not able to reproduce this. Could you follow the exact same steps as me and see if it still happens:

Install NCS v1.5.1 through the Toolchain Manager
- Installing the Toolchain manager
- Installing the nRF Connect SDK
Open bash as expained in 1.3.2 Build and flash
Redirect to ncs/v1.5.1/zephyr/samples/subsys/nvs
Connect the nRF52840 DK to the PC and open Termite
Run these commands
- nrfjprog --eraseall
- west build -b nrf52840dk_nrf52840 -d build_52840
- west flash -d build_52840
Then you should see the follwoing output:

*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 98
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,239] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: alloc wra: 0, cb0[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: data wra: 0, 228[1B][0m
Reboot counter history: ...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1
*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 99
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,270] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,300] [1B][0m<inf> fs_nvs: alloc wra: 0, ca8[1B][0m
[00:00:00.258,300] [1B][0m<inf> fs_nvs: data wra: 0, 22c[1B][0m
Reboot counter history: ...99...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1
*** Booting Zephyr OS build v2.4.99-ncs2  ***
Id: 1, Address: 192.168.1.1
Id: 2, Key: ff fe fd fc fb fa f9 f8 
Id: 3, Reboot_counter: 100
Id: 5, Longarray: 0 1 2 3 4 5 6 7 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 
[00:00:00.258,239] [1B][0m<inf> fs_nvs: 3 Sectors of 4096 bytes[1B][0m
[00:00:00.258,239] [1B][0m<inf> fs_nvs: alloc wra: 0, ca0[1B][0m
[00:00:00.258,270] [1B][0m<inf> fs_nvs: data wra: 0, 230[1B][0m
Reboot counter history: ...100...99...98...97...96...95...94...93...92...91...90...89...88...87...86...85...84...83...82...81...80...79...78...77...76...75...74...73...72...71...70...69...68...67...66...65...64...63...62...61...60...59...58...57...56...55...54...53...52...51...50...49...48...47...46...45...44...43...42...41...40...39...38...37...36...35...34...33...32...31...30...29...28...27...26...25...24...23...22...21...20...19...18...17...16...15...14...13...12...11...10...9...8...7...6...5...4...3...2...1...0
Oldest reboot counter: 0
Rebooting in ...5...4...3...2...1

Please test the above and report back what result you got

Best regards,

Simon

RE: Hard fault when using NVS with Zephyr on nRF52

NathanML — Thu, 06 May 2021 15:06:21 GMT

Hi Simon,

Thanks for the response!

As I'm already seeing the full log output, similar to the hard fault example, I ran the addr2line command in my <application>/<build folder>/zephyr but did not receive a useful response:

$ addr2line -e zephyr.elf 0x000054d4
:?

The fault occurs when I call the nvs_write function. Using breakpoints in nvs.c I see that the underlying function causing the hard fault is flash_write (as called by nvs_flash_al_wrt, which was called by nvs_flash_data_wrt, which was called by nvs_flash_wrt_entry).

I'm not sure where to go from here. Based on your tutorial link to hard faults, and note that mentions they are often do to a stack overflow, I increased CONFIG_MAIN_STACK_SIZE to give that a try, but the results were the same.

Also, could you address the word-alignment error I'm getting with the default nvs sample code on the nRF52840 development kit? I'd like to avoid this error on our custom hardware as well, once we can get flash writes working.

I appreciate your help!

RE: Hard fault when using NVS with Zephyr on nRF52

Simon — Thu, 06 May 2021 10:57:02 GMT

NCS Tutorial-->Error handling-->Other tools and tips-->Hard faults may help getting to the bottom of this.

Best regards,

Simon