nRF52840 logging crashes in FreeRTOS

RE: nRF52840 logging crashes in FreeRTOS

Susheel Nuguru — Mon, 08 Oct 2018 20:20:25 GMT

Hi Nopik,

The flash corruption is clearly a bi-product of the high frequency reset. Please let me know of any other way of reproducing this issue other than this reset.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Mon, 08 Oct 2018 16:27:08 GMT

The high frequency reset was just a by-product of trying to revel the logging crash (which started this thread). I though I was able to reproduce easily it earlier, but apparently I wasn't. But since I've realized, that in many cases logging is crashing mostly in first few milliseconds, I added quick resets to pinpoint the problem. It, however, uncovered the flash corruption problem, instead.

RE: nRF52840 logging crashes in FreeRTOS

Susheel Nuguru — Mon, 08 Oct 2018 13:24:28 GMT

Nopik,

What is your usecase exactly? why do you want to reset the hardware with such a high frequency?

There is some flash activity(Read/Write) happening in the initialization phase and I am almost certain that such frequent resets would interrupt an ongoing flash activity. As the result of this the behavior of the thread stack would become unpredictable and eventually cause a hardfault. Looking at the call stack of when the hardfault occurs, it matches my theory that it happens when there is a flash read from a previous written (interrupted write, most probably, corrupting the flash).

RE: nRF52840 logging crashes in FreeRTOS

Susheel Nuguru — Mon, 08 Oct 2018 12:30:03 GMT

Hi Nopik, This thread somehow skipped my queue, otherwise I would not miss this for so many days. Sorry about that. I just tried to test your code, I see what you see now. Very strange that the flash is getting corrupted. I am debugging this now. Will be back to you when i find something.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Mon, 08 Oct 2018 09:37:14 GMT

Did you tried to reproduce the flash corruption issue, as I described here?

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Thu, 27 Sep 2018 11:56:19 GMT

Aryan - the setup is very simple:

1) Regular PCA10056 board

2) Thread SDK 1.0

3) For examples/thread/freertos_coap_server/main.c use the version I provided earlier (https://gist.github.com/Nopik/d59d269032fa5fe4e3b2af5001dd4307)

4) Either replace lines 316-317 by `NVIC_SystemReset();` call or add jumper wire between GPIO P1.11 and P0.18

5) Flash PCA10056 and run the software

6) Leds will be blinking for 13-14 seconds, then they will stop

7) Flash becomes corrupted, PCA10056 wont boot anymore without full erase

RE: nRF52840 logging crashes in FreeRTOS

Susheel Nuguru — Thu, 27 Sep 2018 10:54:12 GMT

Hi Nopik,

FreeRTOS does not do any flash operations by itself. I would not correlate the flash corruption to FreeRTOS. Flash operations are application specific and could go wrong if you gave an uninitialized pointer to a flash operation.

Adding a lot of logs seems to worsen the behavior, which indicates that there must be a timings issue too. It is really hard for us to narrow down the problem without getting your whole setup to replicate the issue.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Thu, 27 Sep 2018 08:37:28 GMT

Also, using `NVIC_SystemReset();` instead of pinreset corrupts flash as well.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Wed, 26 Sep 2018 13:09:37 GMT

For the record, Thread SDK v2.0.0 gives the same behaviour.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Wed, 26 Sep 2018 12:31:43 GMT

Also, btw. the replacing FreeRTOS.h with your version makes no difference.

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Wed, 26 Sep 2018 12:29:49 GMT

So, the problem is still there, though it seems to be much more buried. When I got your reply, I started with replicating the bug on my side, and indeed - it now happens to me much less often (more like 1 in 100, as you mentioned). I'm using the same bin file and same hardware as before, which is quite puzzling. Yet, the problem originates from our proprietary software (which is using FreeRTOS and Thread), on custom board.

Earlier I've pinpointed the problem to the log output - when I've been putting tons of logs (just for troubleshooting purposes, stuff like `for( int i=0;i<100;i++ ) NRF_LOG("foo");`, the code was crashing almost instantly. Then, I reproduced it on PCA10056 and almost clean example, so that is what I reported. I will try to investigate it again on our code version.

In meantime, I tried to speed up the crashes in the example you've been working on. Since often code was crashing quickly after start, I did something simple: if task looped for some time, then reset. I installed wire between some random gpio (P1.11 in my case) and P0.18 on my PCA10056, and added to the main task1 something like this:

```

if( cnt > 20 ) {
nrf_gpio_cfg_output(RESET_PIN);
nrf_gpio_pin_clear(RESET_PIN);
}

```

i.e. reset after roughly 100 logs.

The results were just stunning, so different, that I'm afraid that I've uncovered some other problem.

When I run this code, it runs for 13-14 seconds (100% constant), then crashes - by corrupting flash. After it crashes, it wont boot up anymore, until I do `nrfjprog -e` and flash the code again. UICR is intact, `nrfjprog --eraseuicr` is not enough to recover. When I dumped flash before and after crash, indeed, some parts of it (but not the code itself) are overwritten by some repeating pattern.

I'm not sure if the reset method is valid (though with external watchdog it is very similar to what would happen), but whatever happens, flash content shouldn't be corrupted.

If I extend the timer and reset only after `cnt` reaches 100 or 1000, it just happens slower, but also corrupts the flash.

Can you confirm, that you can reproduce this problem? My current `main.c` is at: https://gist.github.com/Nopik/d59d269032fa5fe4e3b2af5001dd4307 Is this reset method valid? I suspect that electrically the reset impulse is simply too short and causes some misbehavior, or something.

PS. I've seen that Thread SDK v2.0 was released today, I will try it as well, and let you know.

RE: nRF52840 logging crashes in FreeRTOS

Edvin — Tue, 25 Sep 2018 12:13:30 GMT

Hello,

I had a colleague look at this (who has worked with our FreeRTOS stack).

I did manage to reproduce your issue, as I mentioned, but not nearly as often as you report. Maybe one in 100 times or so, so it took quite a while. My go-to was to just restart it unless LED3 didn't blink at all.

However, my colleague suggested some changes in the FreeRTOSConfig.h, and I haven't been able to reproduce it after this.

Could you please try to replace your FreeRTOSConfig.h file with the attached one, and see whether you are able to reproduce it with this version?

devzone.nordicsemi.com/.../FreeRTOSConfig.h

Best regards,

Edvin

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Tue, 25 Sep 2018 11:31:43 GMT

Any progress?

RE: nRF52840 logging crashes in FreeRTOS

Nopik — Mon, 17 Sep 2018 14:15:08 GMT

Ok, thanks. Hopefully you'll be able to pinpoint the reason soon. Good to know, that you've reproduced the issue.

RE: nRF52840 logging crashes in FreeRTOS

Edvin — Mon, 17 Sep 2018 14:11:43 GMT

Hello,

Sorry for the late reply. I have been trying to reproduce this. That is, I have managed to reproduce it, but not while debugging, so I am having some problems to find the reason. I just wanted to let you know that your ticket is being handled. I will get back to you as soon as I know something more.

Best regards,

Edvin