Crash issue on sdk v2.9.0

[Leo Mo]

SOC: nrf52840

SDK: v2.9.0

Device is a central, it connected to 16 peripherals, these peripherals will power on 90 seconds and then power off 10 seconds(one cycle).

After ten days of operation, there will be a crash.

Has anyone run into this problem? How do you solve it? Thank you!

Crash 1, reason is K_ERR_KERNEL_OOPS=3.

Crash 2, reason is K_ERR_ARM_BUS_IMPRECISE_DATA_BUS=26.

CONFIG_LOG_PROCESS_THREAD_STACK_SIZE=768

logging thread max stack usage: 744 B(97%)

 

[Hieu]

Hi Leo,

Since you started monitoring stack size in the second attempt, I take it you have suspected stack overflow.

That would be my guess too. I am not sure if the stack monitoring can monitor up to the crash. What I mean is: If the stack overflow and crash, then can the monitor catch that, or is the event lost?

Generally, I think if a max usage goes over 90%, then it is a good idea to budget more.

Could you please give the log thread more stack? Please also check other threads to see if any is having similar issues.

Another thing to watch out for is to see if the stack use increases over time, which would indicate a memory leak somewhere.

Hieu

[Leo Mo]

Hi Hieu,

Thank you! 

I will increase the stack for these thread.

Leo

[Hieu]

No problem. I hope that resolves the issue.

[Leo Mo]

I increase CONFIG_LOG_PROCESS_THREAD_STACK_SIZE to 1024, this error still appears

bt_sdc_hci_driver: SoftDevice controller ASSERT: 33,748

And new bugs have appeared：

[20:26:54.767,669] <err> bt_sdc_hci_driver: SoftDevice Controller ASSERT: 53, 295
[20:26:54.767,700] <err> os: ***** HARD FAULT *****
[20:26:54.767,700] <err> os:   Fault escalation (see below)
[20:26:54.767,700] <err> os: ARCH_EXCEPT with reason 3

 

[20:26:54.767,730] <err> os: r0/a1:  0x00000003  r1/a2:  0x00000004  r2/a3:  0x00000003
[20:26:54.767,730] <err> os: r3/a4:  0x00000003 r12/ip:  0x20025860 r14/lr:  0x00058087
[20:26:54.767,761] <err> os:  xpsr:  0x01000011
[20:26:54.767,761] <err> os: Faulting instruction address (r15/pc): 0x0004cdb0
[20:26:54.767,822] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[20:26:54.767,822] <err> os: Fault during interrupt handling

 

[20:26:54.767,852] <err> os: Current thread: 0x2001aff8 (idle)
[20:26:55.171,417] <err> os: Halting system

[Leo Mo]

Hello Hieu,

@Hieu

Have you encountered these issues?

[Hieu]

Hi Leo,

Unfortunately, I haven't encountered them before, but I have started an internal inquiry about them.

ASSERT: 33,748 is still under investigation. We are not certain about this yet.

As for ASSERT: 53, 295, could you please let me know if you are using LFRC or LFXO? Do you encounter this on one particular unit, or multiple units?

[Leo Mo]

Hi Hieu,

I use the LFRC(CLOCK_CONTROL_NRF_K32SRC_RC=y).

Only two units encounter this issue(ASSERT: 53, 295) so far.

[Hieu]

Hi Leo,

Sorry I should have asked earlier, but how many units total are you testing? 

Of the two units that have that problem, does it consistently occur again?

[Leo Mo]

Hi Hieu,

There are 4 units under testing, two of which are connected to J-Link in debug mode.

Only the devices connected to J-Link have encountered this issue (ASSERT: 53, 295).

I want to know what these error codes mean. If they are caused by the application, I can make changes on my side.

Partial configuration：

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_EXT_LOW_SWING is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_EXT_FULL_SWING is not set

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y

CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_LF_ALWAYS_ON=y

CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_PERIOD=4000

CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_MAX_SKIP=1

CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_TEMP_DIFF=2

CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_250PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_150PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_100PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_75PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_50PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_30PPM is not set

# CONFIG_CLOCK_CONTROL_NRF_K32SRC_20PPM is not set

CONFIG_CLOCK_CONTROL_NRF_ACCURACY=500

[Leo Mo]

 Hi, Hieu

 Hieu 

I would like to know the meaning of these error codes(ASSERT: 53, 295, ASSERT: 33,748) ; can you ask the software engineer directly? 

[Hieu]

Hi Leo,

Unfortunately, the asserts are part of the SoftDevice Controller, so we are very limited in what we can disclosed. Right now, they are still being investigated, and I am not clear to disclose anything yet.

I will update you once I have anything I can share. Our apologies for the inconvenience.

[Hieu]

Hi Leo,

ASSERT: 33,748 is due to a race-condition bug. The condition is very difficult to trigger, but with 16 concurrent connections, it happened faster. We have registered it and is working on a fix. However, I cannot comment about the timeline on DevZone, and you will have to talk to sales for that.

ASSERT: 53, 295 means that the central role was ran longer than expected. Does your application have any highest priority interrupts, or does it disable interrupts for extended period of time?