nrf9160 secure services causing a crash

RE: nrf9160 secure services causing a crash

fastfox — Tue, 01 Dec 2020 09:28:22 GMT

I got around that crash by increasing the SPM stack size. See the whole story from here: devzone.nordicsemi.com/.../nrf9160-sdk-1-4-crash-in-random-number-generation

RE: nrf9160 secure services causing a crash

Jameson — Mon, 30 Nov 2020 20:37:31 GMT

I am upgrading to the v1.4 SDK and am seeing similar behavior. I see it occur the net driver is getting initialized (specifically sys_init_net_init0), however it ultimately calls "spm_request_random_number" and crashes/restarts. It then just gets stuck in a boot loop.

Are there any updates on this?

RE: nrf9160 secure services causing a crash

fastfox — Mon, 16 Nov 2020 09:50:39 GMT

Interestingly, my app started to crash on spm_request_random_number(stack is generating COAP tokens) after upgrading to 1.4 SDK. Dunno yet if it is related to this or not

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Mon, 09 Nov 2020 12:49:21 GMT

This is the workaround I posted earlier.

It should remove the problem, yes, but it doesn't really solve the fundamental issue of synchronization accross the secure/non-secure boundry. It just works around it by not letting the non-secure domain do anything.

There are more changes and improvements coming in the future that should make this workaround no longer necessary.

RE: nrf9160 secure services causing a crash

miked531 — Sat, 07 Nov 2020 01:12:29 GMT

I noticed this recent commit: https://github.com/nrfconnect/sdk-nrf/commit/6efb6fd633a4ea47279943c5660239083c5bdd65

Does it resolve the issue?

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Mon, 28 Sep 2020 13:35:11 GMT

A workaround is scheduled to be part of the next NCS release.

The workaroud is dependent on some changes made in Zephyr, so the developers are currently waiting for the next upmerge to complete.

RE: nrf9160 secure services causing a crash

miked531 — Mon, 28 Sep 2020 00:18:00 GMT

Is there any update on this?

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Mon, 03 Aug 2020 07:04:01 GMT

[quote user="miked531"]I looked at the PR and it doesn't seem to be progressing at the moment.[/quote]

We are just coming out of the summer vacation period here in Norway, so the lack of progress might be due to low staffing during the vacation.

As for when the solution will be available in an NCS release, I can not really give you an answer.

Hopefully, it will be a part of the next release. Although it could also take longer, I will let you know if the developers have an estimate.

Anyway, I have asked the developers for a status update.

RE: nrf9160 secure services causing a crash

miked531 — Fri, 31 Jul 2020 16:09:37 GMT

I looked at the PR and it doesn't seem to be progressing at the moment. Is this still the intended way to resolve this? Actually my real question is when will a solution be available in a NCS SDK release?

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Wed, 24 Jun 2020 14:47:27 GMT

There is also a more complete proposal for a solution: https://github.com/zephyrproject-rtos/zephyr/pull/26414

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Tue, 23 Jun 2020 11:46:53 GMT

Hi.

A PR with a workaround has been opened: https://github.com/nrfconnect/sdk-nrf/pull/2519/files

Best regards,

Didrik

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Wed, 03 Jun 2020 12:58:29 GMT

Hi again.

There has been some further progress in finding the root cause, but it does not look like there is an easy fix, so it will probably take some time until it is properly fixed.

Here is an explanation of what probably happens, given by one of our developers:

"Short story: if a Zephyr thread context-switch occurs while doing a Secure call, the Zephyr kernel may crash.

Elaboration:

This might, actually, be happening in the scenario reported in this issue: I believe, so, because it seems the secure service takes some time to finish, so an NSPE Zephyr system time may fire, in the mean time, and lead to a thread re-schedule point.

When in a regular function call, the Zephyr Cortex-M swap logic (running in PendSV IRQ, possibly tail-chained from a higher priority IRQ context) will simply stack the current execution context into the thread's stack, tweak PSP, and eventually return into the new thread's execution context, using that thread's stack(ed) information.

When in a Secure function call, however, things are not as simple as the above scenario. The swap logic will still change the NSPE context, but it will return to the SPE to finish the secure service. When the secure service is complete, the TrustZone logic will attempt to branch to the NSPE (most likely using BXNS LR instruction), however, since the NSPE context has changed, the service call is not returning to the context it was called from.

This bug, may be causing system crashes (possibly such as the one we are currently observing here), but is, also a direct Security violation, since secure service result is exposed to a NSPE context other than the caller."

We are looking into what is the best way to ensure that a secure call always returns to the caller.

Best regards,

Didrik

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Fri, 15 May 2020 08:30:48 GMT

Hi.

A quick update:

The error does not seem to be linked to the buttons library in particular, but to the workqueue.

It seems the workqueue thread is not allowed to run when the timeout happens while in secure mode, and consequently an error happens once the main thread is unloaded.

We are investigating ways to solve this.

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Thu, 07 May 2020 14:41:50 GMT

Hi again.

I have found another way of avoiding the crash: changing the system workqueue's priority to a positive priority (e.g. 1).

The changes the system workqueue thread from a cooperative one to a preemptible one. That allows other threads (with higher priorities) to interrupt the system workqueue.

However, I have not been able to find the root cause of the crash. I will continue to investigate together with the SDK team.

Best regards,

Didrik

RE: nrf9160 secure services causing a crash

miked531 — Thu, 30 Apr 2020 17:44:31 GMT

I'm glad you see it too. I looked into it a little bit and it seemed to be related to the button_scan_fn or maybe zephyr's handling of the workq, but that's as far as I got.

RE: nrf9160 secure services causing a crash

Didrik Rokhaug — Thu, 30 Apr 2020 15:12:28 GMT

Hi.

What an interesting problem you have found.

I have done some testing myself, and the fault only happens if the buttons are initialized. Logging does not matter, other than for printing the fault message.

It also only happens when we try to get random numbers and not any of the other secure services.

Based on your comment regarding the scan interval, I expect it is a race condition related to the transition between the secure and non-secure domains.

However, I will have do look deeper into it next week to be able to pinpoint the cause of the problem.

Best regards,

Didrik

RE: nrf9160 secure services causing a crash

miked531 — Wed, 29 Apr 2020 20:12:32 GMT

The behavior changes to "working" by increasing CONFIG_DK_LIBRARY_BUTTON_SCAN_INTERVAL in prj.conf from the default of 10 to 100. I can't even come up with an explanation for this