Random SecureFault on nRF9161

[pefech]

Hi!

We are currently building a product using nRF9161 on (rather old) nrf-sdk version v2.9.2.

After production we are testing the hardware functions of this product using a firmware that has the Zephyr shell configured for our test cases.
For example reading / writing GPIO, ADC or sensors, so nothing special, just Zephyr shell on UART0

We observed that some of our boards fail because the test firmware is not able to start up correctly when used with this firmware.
If this error happens, this is the only output of the board:

Booting TF-M v2.1.1-ncs2-2
[Sec Thread] Secure image initializing!
FATAL ERROR: SecureFault
All pins have been configured as non-secure

After this output the board hangs.

I don't exactly know what the problem for this error could be as just some boards show this behavior, others do not, even its exactly the same firmware.
If I try another firmware (like the final firmware) the boards start up normally.

On the other hand, some boards show the same behavior using the AT monitor firmware. We also use it in after production testing for programming the IMEI.

The boards that shows the errors do not correlate. It's rather random which boards throw the error.
But if a board has the error with one of the firmwares, it has it permanently. Boards that do not have SecureFault problems with any firmware will not have problems permanently.

I understand that if a SecureFault is triggered, that there is some kind of security violation. But I can't imagine why it happens on some boards permanently and on other boards not at all.
In my understanding the software should behave deterministic and if I have a problem shouldn't I have it on all my boards?

What I tried to do on not working ones:

- Mass erase everything and reflash

- try to set 

CONFIG_TFM_LOG_LEVEL_SILENCE=y

CONFIG_TFM_SECURE_UART=n

I read this in another post on here (But I can't find it now) but it didn't worked. I don't know how to continue debugging at the moment. I'd be really happy if someone has an idea of whats going on. There are to many boards not working by now because of this problem and that is not acceptable for mass production in the future.

Thanks a lot for help!

[Elfving]

Hi again Peter,

I agree that this sounds odd. Are you seeing the same things with DKs? Have you had us doing a HW review on these boards before?

Regards,

Elfving

[pefech]

Hi Elfving,

no, we didn't had Nordic do a HW review on these boards. To be honest I didn't used the DKs much, as I took over this  project as there was already prototype hardware. But I will test the firmware on the DK.

Is there any chance that there is something like a manual that describes in which situations the nRF9161 is going into this SecureFault state? I did not find much in the reference manual about that. Of course this happens if the firmware accesses some memory location that it is not allowed to. But that would be the same on all of my boards. Maybe the trusted firmware also can trigger on some external events?

Maybe something like this could help me figure out whats happening.

Can you tell me how I could get a HW review by Nordic, if I wanted to?

Regards

Peter

[Elfving]

pefech said:

Is there any chance that there is something like a manual that describes in which situations the nRF9161 is going into this SecureFault state?

Well you can find some about issues a few places, here and there, here and a bit here. Access of something that is not allowed is the common thing here, though what I find myself focusing on is the fact that it for some reason happens to just some of the boards. To me that is a sign that HW could play a role.

pefech said:

Can you tell me how I could get a HW review by Nordic, if I wanted to?

You would upload the schematics and potentially layout to a (preferably private) DevZone case (this one happens to be open). I can also make this one private if you want me to.

Regards,

Elfving

[carsondm]

Has there been any resolution on this? I’m experiencing a very similar issue.

[Elfving]

No I have yet to hear anything else from the customer on this one. Are you able to reproduce it on a  DK?

[pefech]

We had no real solution for this by now.

As we had some problems with cold SMD solder joints on the nrf9161 in our manufacturing process for the prototypes, we thought that maybe a floating pin or a fluctuating VCC or similar could be the trigger for the chip to detect a secure fault.

I'll wait for our next badge to be produced and will take a look on it. I'll keep informed if I know something new.

I actually do not had time (and a nrf9161 DK) to test it on a DK, but I'm planning to test this on a DK again.

[Elfving]

Understood. Let us know when you need anything else.