Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

Håkon Alseth — Thu, 01 Jul 2021 11:59:19 GMT

Hi Michael,

Those stack traces and faults seems to match better.

The fault is now a MPU-fault (ie: a alloc is asking for more RAM than it has), and it seems that you have already found a suspect in the net-bufs. I agree with your findings, both call stacks point to the NET stack slab memory on both scenarios:

https://github.com/nrfconnect/sdk-zephyr/blob/master/subsys/net/ip/net_pkt.c#L112-L113 (expands to _k_mem_slab_buf_rx_pkts)

Adjusting the NET_BUF_*X_COUNT should fix these two faults, but I think you're on the right track to also keep an eye on the stack size as well.

Kind regards,

Håkon

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

mglettig — Thu, 01 Jul 2021 11:02:47 GMT

Hi Håkon

[quote userid="2115" url="~/f/nordic-q-a/76771/nordic-nrf-sdk-v1-6-issue-with-civetweb-on-nrf9160/317966#317966"]

There's a helper function you can enable via CONFIG_SPM_SERVICE_NS_HANDLER_FROM_SPM_FAULT introduced with this PR: https://github.com/nrfconnect/sdk-nrf/pull/3933

[/quote]

FYI: I need to apply the following patch to compile with this flag:

diff --git a/lib/fatal_error/fatal_error.c b/lib/fatal_error/fatal_error.c
index 8d35a27c6..4d234199a 100644
--- a/lib/fatal_error/fatal_error.c
+++ b/lib/fatal_error/fatal_error.c
@@ -14,7 +14,7 @@
 #include <secure_services.h>
 #endif
 
-LOG_MODULE_REGISTER(fatal_error, CONFIG_FATAL_ERROR_LOG_LEVEL);
+LOG_MODULE_REGISTER(fatal_error, 1);
 
 extern void sys_arch_reboot(int type);

For me it looked like the console output was still the same with this option enabled:

[00:00:18.230,072] <err> os: ***** MPU FAULT *****
[00:00:18.250,762] <err> os:   Instruction Access Violation
[00:00:18.272,277] <err> os: r0/a1:  0x00000000  r1/a2:  0xe000ed00  r2/a3:  0x2002a730
[00:00:18.296,356] <err> os: r3/a4:  0x00000001 r12/ip:  0x00000001 r14/lr:  0x0005a0b1
[00:00:18.320,434] <err> os:  xpsr:  0x60000000
[00:00:18.340,881] <err> os: Faulting instruction address (r15/pc): 0x20035f18
[00:00:18.364,135] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:18.387,298] <err> os: Current thread: 0x20019c78 (unknown)
[00:00:18.429,382] <err> fatal_error: Halting system

[quote userid="2115" url="~/f/nordic-q-a/76771/nordic-nrf-sdk-v1-6-issue-with-civetweb-on-nrf9160/317966#317966"]If you debug the secure image (spm/zephyr/zephyr.elf) and break on the hardfault handler (or other fault handlers, like z_arm_fatal_error())[/quote]

I set a breakpoint in z_arm_fatal_error(). Then when the error occurred I got the following call stack:

The other time I tried it I got the following callstack:

--> here I noticed that the buffer that is generated in pkt_alloc has an address of 0x2b083 which seems to be invalid.

Next I will try to tweak the following values:

CONFIG_NET_BUF_RX_COUNT
CONFIG_NET_BUF_TX_COUNT
CONFIG_NET_RX_STACK_SIZE
CONFIG_NET_TX_STACK_SIZE

Kind regards,

Michael

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

Håkon Alseth — Wed, 30 Jun 2021 12:23:05 GMT

Hi Michael,

[quote user="mglettig"][00:00:32.181,610] <err> os: Exception occurred in Secure State[/quote]

If you debug the secure image (spm/zephyr/zephyr.elf) and break on the hardfault handler (or other fault handlers, like z_arm_fatal_error())

in that image, you should be able to figure out what the failure is (reading the "esf" or "esf_copy" variable).

There's a helper function you can enable via CONFIG_SPM_SERVICE_NS_HANDLER_FROM_SPM_FAULT introduced with this PR: https://github.com/nrfconnect/sdk-nrf/pull/3933

It could also be beneficial to disable the reset on fault via. "CONFIG_RESET_ON_FATAL_ERROR=n" for both spm and your applications prj.conf.

Kind regards,

Håkon

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

mglettig — Wed, 30 Jun 2021 11:13:52 GMT

Hi Håkon

[quote userid="2115" url="~/f/nordic-q-a/76771/nordic-nrf-sdk-v1-6-issue-with-civetweb-on-nrf9160/317890#317890"]The line numbers in civetweb.c indicates that you're on a newer version than what was tagged out for ncs v1.6.0. Have you manually updated this?[/quote]

You are right I was using a newer version of civetweb. But I reverted now to what is officially fetched with ncs v1.6.0.

Since I'm now calling mg_start from a POSIX thread the initial problem is resolved. That means that the mg_start is successful. However the first time I access the webserver I get the following crash:

[00:00:32.181,610] <err> os: Exception occurred in Secure State
[00:00:32.203,460] <err> os: ***** HARD FAULT *****
[00:00:32.224,243] <err> os: Fault escalation (see below)
[00:00:32.245,727] <err> os: ***** BUS FAULT *****
[00:00:32.266,418] <err> os: Precise data bus error
[00:00:32.287,384] <err> os: BFAR Address: 0x50008158
[00:00:32.308,563] <err> os: r0/a1: 0x000011a9 r1/a2: 0x00000000 r2/a3: 0x00000000
[00:00:32.332,641] <err> os: r3/a4: 0x00000000 r12/ip: 0x4ad7ff9d r14/lr: 0x00018fa3
[00:00:32.356,689] <err> os: xpsr: 0x06165200
[00:00:32.377,166] <err> os: Faulting instruction address (r15/pc): 0x00059992
[00:00:32.400,390] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:32.423,522] <err> os: Current thread: 0x2001b578 (idle 00)
[00:00:32.465,606] <err> os: Halting system

I did not yet manage to find the location of where this crash occurs. Do you have a suggestion on how to make use of the fault information to find the source for the BUS FAULT? Or is it possible to configure the J-Link debugger in a way that it stops immediately when the HARD Fault occurs?

By the way my idle stack size is pretty big:

CONFIG_IDLE_STACK_SIZE=2048

Please note that I needed to apply the following patch in civetweb because it doesn't allocate enough stack for the worker threads:

diff --git a/src/civetweb.c b/src/civetweb.c
index 909a93a9..580ac4a6 100644
--- a/src/civetweb.c
+++ b/src/civetweb.c
@@ -186,7 +186,7 @@ mg_static_assert(sizeof(void *) >= sizeof(int), "data type size check");
 #if defined(USE_STACK_SIZE) && (USE_STACK_SIZE > 1)
 #define ZEPHYR_STACK_SIZE USE_STACK_SIZE
 #else
-#define ZEPHYR_STACK_SIZE 8096
+#define ZEPHYR_STACK_SIZE 16*1024
 #endif
 
 K_THREAD_STACK_DEFINE(civetweb_main_stack, ZEPHYR_STACK_SIZE);

Best regards,

Michael

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

Håkon Alseth — Wed, 30 Jun 2021 08:35:22 GMT

Hi Michael,

The line numbers in civetweb.c indicates that you're on a newer version than what was tagged out for ncs v1.6.0. Have you manually updated this?

If you revert to the git hash that was used in ncs v1.6.0 (which is the same as the git hash in ncs v1.5.1), do you experience the same issues?

Kind regards,

Håkon

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

mglettig — Tue, 29 Jun 2021 12:14:54 GMT

Hi Håkon

I'm one step further. I guess that the calling thread of mg_start (the one that starts civetweb webserver) needs to be a POSIX thread. It looks like civetweb is expecting this because it uses

struct posix_thread *thread = (struct posix_thread *)pthread_self();

I guess than it's kind of a coincidence that it works on QEMU and it worked on SDK v1.5.1. I'm still working on getting it working but I will keep you posted.

Best regards,

Michael

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

mglettig — Tue, 29 Jun 2021 09:10:35 GMT

Hi Håkon

For me it looks like one entry is invalid in the thread->key_list. This is when the debugger hits pthread_setspecific the first time:

Now this are the individual iterations in the for loop:

I guess 0x6f727245 is an invalid address. Is that correct? Could this issue have anything to do with THREAD_LOCAL_STORAGE support? I noticed that this is disabled on my target due to lack of support for it from the toolchain.

Best regards,

Michael

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

Håkon Alseth — Tue, 29 Jun 2021 07:02:09 GMT

Hi Michael,

My apologies, this was also shown in your initial picture, which just flew by me, unfortunately. This line seems to be causing issues for you:

https://github.com/nrfconnect/sdk-zephyr/blob/master/lib/posix/pthread_key.c#L96

In the stack frames, you should be able to right-click and select this specific frame, allowing you to peek at the variables at the time of the fault occurring. The value of thread_spec_data (and .key member) is very interesting here.

Kind regards,

Håkon

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

mglettig — Mon, 28 Jun 2021 16:41:21 GMT

Hi Håkon

Thanks for the help. Increasing the main stack size didn't help. I increased the stack up to 16 kB. Resolving the address pointed me to the module pthread_key:96

/opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-addr2line -e build/zephyr/zephyr.elf 0x0001a684
/opt/zephyrproject/zephyr/lib/posix/pthread_key.c:96

This is inline with what the debugger was showing me.

Do you have any idea why the error happens on the target and doesn't happen in QEMU. It would also help if we could figure out due to which change from v1.5.1 to v1.6.0 the error was introduced.

Best regards,

Michael

RE: Nordic NRF SDK v1.6: Issue with CIVETWEB on NRF9160

Håkon Alseth — Mon, 28 Jun 2021 07:42:41 GMT

Hi,

The thread that is failing looks to be your main thread. Did you try adjusting the CONFIG_MAIN_STACK_SIZE?

[quote user=""]

Could you help me to narrow down the issue? How can I use the exception information to figure out more about the issue?

[/quote]

You can use arm-none-eabi-addr2line -e build/zephyr/zephyr.elf 0xFAULTING_ADDR to see if it gives a hint on where in your src the fault occurred.

The content of PC and LR is in the flash, so you can try to do a lookup on those addresses

arm-none-eabi-addr2line -e build/zephyr/zephyr.elf 0x1a60b

Kind regards,

Håkon