Trying to figure out a bus fault error (Zephyr + NRF52840)

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Vidar Berg — Thu, 04 Jul 2024 13:31:38 GMT

Hi Alberto,

I'm happy to hear you found the problem! Thank you for the update.

Cheers,

Vidar

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Albert_0 — Thu, 04 Jul 2024 13:28:19 GMT

Hi Vidar,

The overflow was on the buffer because the index was getting bigger than it should have.

I ran some tests for the last four and a half hours and it all seems to work well. Some issues were also due to some checksum errors, and I added additional checks to prevent accessing buffers out of boundaries.

Thanks you very much for the help, I'm happy how this is working now and I think all is solved.

Cheers,

Alberto

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Vidar Berg — Thu, 04 Jul 2024 13:22:14 GMT

Hi Alberto,

Maybe the overflow could be in the 'controller_receive_buf' or the 'controller_data_buf' buffer. They are pretty close to the 'adc_filt_idx' in RAM.

Cheers,

Vidar

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Albert_0 — Thu, 04 Jul 2024 05:50:25 GMT

Hey Vidar,

Thanks, sure, I can upload the map file.

devzone.nordicsemi.com/.../5141.zephyr.map

I had some progress. I noticed one pipe where the error was happening had a different buffer size than defined originally, it was larger, so that was overwriting the other pipe I believe. That issue has been solved.

The test ran now for over 3 hours, unfortunately, there was another bus crash somewhere else which I need to look at now... baby steps.

I'll try to find out where the error happened now.

Cheers,

Alberto

Edit:

Looks like another overflow somewhere else...

That value does not look right...

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Vidar Berg — Thu, 04 Jul 2024 05:36:59 GMT

Hi Alberto,

Thanks for the update. It does look like there is a buffer overrun that is overwriting more than just the thread object. Maybe you can post the zephyr.map file here. It might provide some clues even though we haven't seen the source code.

Cheers,

Vidar

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Albert_0 — Thu, 04 Jul 2024 01:10:01 GMT

I was able to attach the debugger and it stopped at the fault. I'm also able to look at the variables and the call stack, and the fault is close to where I expected it to be.

I also noticed that one of the k_pipes I'm using for data buffering seems to be corrupt. This is a normal pipe with OK values (not sure what "buffer" refers to):

The one causing the issue looks like:

I'm sure I must be corrupting it somewhere.

I'll see what else I can find. Thanks for your input so far.

Cheers,

Alberto

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Albert_0 — Wed, 03 Jul 2024 10:58:15 GMT

I ran a test increasing the stack for that thread to double the size. In the end that didn't help. It may have lasted a few minutes longer, but I'm not sure, in the end it felt it failed in the same position.

I will try with the debugger connected tomorrow and disable the reset on hard fault. I will also look at the variables in the map file. Unfortunately the problem seems to happen on a Zephyr file, and I don't have so much experience with Zephyr OS other than as user.

Thanks for confirming the 0x80bf68b4 is rubbish, and I have a few places I can look at.

I'll keep you posted.

Cheers,

Alberto

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Vidar Berg — Wed, 03 Jul 2024 10:17:53 GMT

Thank you Turbo J .

Alberto, to avoid the hardfault reset during, you may temporalily disable the CONFIG_RESET_ON_FATAL_ERROR setting in your release build. This will cause the program enter an infinte loop in the fault handler. You can then use Segger Ozone or nrfjprog to inspect the memory. Remember to select "attach to running target" if using Ozone so the chip does not reset.

[quote userid="18935" url="~/f/nordic-q-a/112689/trying-to-figure-out-a-bus-fault-error-zephyr-nrf52840/492006"]Its usually simpler to check the map file what sturctures and varables are "close to" the k_thread structure thats getting trashed and check code that touches these for BO.[/quote]

This is a good suggestion. You can check the name of the 'struct k_thread' variable(s) you have declared in your application and then look up their addresses in the build/zephyr/zephyr.map file.

Reading memory with nrfjprog

$ nrfjprog --memrd <insert RAM address here>

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Turbo J — Wed, 03 Jul 2024 08:46:41 GMT

You would have to catch the crash with a connected debugger - a hard fault reset will likely just overwrite that structure in RAM.

Won't help you much in buffer overflow cases - the cow has already turned into hamburger at this point in time.

Its usually simpler to check the map file what sturctures and varables are "close to" the k_thread structure thats getting trashed and check code that touches these for BO.

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Albert_0 — Wed, 03 Jul 2024 08:38:59 GMT

Hey Vidar, Turbo J,

I will try by increasing the stack size, that is my first suspicion, everything else seems to work well.

Vidar Berg I'm not sure how to check the 'struct k_thread' variables after the crash... I don't have the debugger connected or running. The firmware is compiled in release mode, I have access to J-Link, but it would take me some time to figure out how to check the variables using Ozone. And of course, after the crash, the unit resets... but I'll look into it.

Any other way I can check the k_thread variables after the crash?

Cheers,

Alberto

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Vidar Berg — Wed, 03 Jul 2024 07:08:21 GMT

Hello Alberto,

It looks like the bus fault is caused by 'thread' pointer passed as the first argument to pipe_walk_op() being corrupted. '0x80bf68b4' and '0x80bf68c8' does not map to any valid memory on the device, and the fault is triggered as soon as the function tries to assign the 'pipe descriptor' from the corrupt thread struct pointer.

I suspect the thread struct is being overwritten elsewhere in your appliction, maybe by a buffer overrun. A stack overflow should normally have been caught by the stack guard. Though, increasing the stack sizes are worth a try.

Are you able to check the 'struct k_thread' variables in your application after the crash to see if they have been overwritten?

Best regards,

Vidar

RE: Trying to figure out a bus fault error (Zephyr + NRF52840)

Turbo J — Wed, 03 Jul 2024 06:12:31 GMT

You are looking at a stack overflow that has overwritten important thread information that the scheduler uses when switching between threads.

Can happen when you increase (or tun on) logging levels. Try a bigger stack for that thread.