NRF LOG crashlog backend

RE: NRF LOG crashlog backend

BretH — Fri, 31 Jan 2020 18:31:24 GMT

It's possible that the issue here relates to https://devzone.nordicsemi.com/f/nordic-q-a/57093/nrf_log_backend_flash-start-address-overlap-application

The suggested solution by abhipray could be related.

RE: NRF LOG crashlog backend

Susheel Nuguru — Fri, 03 May 2019 06:47:52 GMT

unfortunately no, i will ping the team to request prioritize this. For your reference, the ticket ID is NRFFOETT-2282

RE: NRF LOG crashlog backend

Eliot Stock — Thu, 25 Apr 2019 21:00:38 GMT

Nordic folks, was this fixed in SDK 15.3.0?

RE: NRF LOG crashlog backend

Kevin Choi — Mon, 18 Feb 2019 01:34:51 GMT

Anyone solve this issue? I also have same problems.

RE: NRF LOG crashlog backend

Susheel Nuguru — Tue, 18 Dec 2018 09:10:44 GMT

I see, I am not able to replicate that problem, but I will report this to the team. Thanks for letting us know.

RE: NRF LOG crashlog backend

abhipray — Thu, 13 Dec 2018 19:10:50 GMT

Yes, I understand. What I meant was if I set NRF_LOG_BACKEND_FLASH_START_PAGE to 0, the default RUNTIME START_ADDR is set to the flash page right after CODE_PAGE.
i.e #define RUNTIME_START_ADDR \
((NRF_LOG_BACKEND_FLASH_START_PAGE == 0) \
? (CODE_PAGE_SIZE * (CEIL_DIV((uint32_t)CODE_END, CODE_PAGE_SIZE))) \
: FLASH_LOG_START_ADDR)

However, this causes the problem where flashlog read does not present any logs even though flashlog status shows it's accumulating logs. Things only start working after flashlog clear.

Everything starts working normally if I change that line of code above to
#define RUNTIME_START_ADDR \
((NRF_LOG_BACKEND_FLASH_START_PAGE == 0) \
? (CODE_PAGE_SIZE * (CEIL_DIV((uint32_t)CODE_END, CODE_PAGE_SIZE) + 1)) \
: FLASH_LOG_START_ADDR)

RE: NRF LOG crashlog backend

Susheel Nuguru — Thu, 13 Dec 2018 11:49:22 GMT

You can use CODE_END as that does not guarantee that the page is empty to be used by log.

You always need assign FLASH_START_PAGE to CEIL the CODE_END using below

 (CODE_PAGE_SIZE*CEIL_DIV((uint32_t)CODE_END, CODE_PAGE_SIZE))

RE: NRF LOG crashlog backend

abhipray — Wed, 12 Dec 2018 17:30:29 GMT

By CODE_PAGE I meant the flash page corresponding to CODE_END which is the macro for the address of the end of the code.

RE: NRF LOG crashlog backend

Susheel Nuguru — Tue, 11 Dec 2018 14:14:56 GMT

what is CODE_PAGE here?

RE: NRF LOG crashlog backend

abhipray — Mon, 10 Dec 2018 23:09:39 GMT

I'm using arm-gcc. Turned out that there was an error in the linker script for the RAM section. I was not aligning one of the sections and variables were not ending up where they were supposed to, somehow corrupting the RAM. Patching the linker script fixed this particular issue where m_log_flash_fstorage was getting corrupted.

Currently, everything looks good except for when I set the FLASH_START_PAGE to right after CODE_PAGE, then I can't read logs until I clear them. As a temporary patch, I set FLASH_START_PAGE to CODE_PAGE+1 which seems to work.

RE: NRF LOG crashlog backend

Susheel Nuguru — Mon, 10 Dec 2018 08:25:13 GMT

That is very strange. Which compiler are you using? looking at the code it does seem to do a memset on any structure which is related to this.

RE: NRF LOG crashlog backend

abhipray — Tue, 04 Dec 2018 18:22:38 GMT

Yes, that will work by extending the time before this situation arises again. If I increase NRF_LOG_BACKEND_PAGES, the situation is only delayed until the flash is filled up again. How can I prevent it from happening permanently? As a temporary patch, I simply don't allow crashlog to enter panic mode if there is not enough room in the flash.

RE: NRF LOG crashlog backend

Susheel Nuguru — Tue, 04 Dec 2018 14:27:46 GMT

seems like you need to increase the buffer if you want to enable crashlog and used up the backend pages before that

#define NRF_LOG_BACKEND_PAGES

RE: NRF LOG crashlog backend

abhipray — Tue, 04 Dec 2018 07:38:48 GMT

Another bug I found was with nrf_log_backend_flash_init(). It appears that m_log_flash_fstorage evt_handler is not being set. Adding m_log_flash_fstorage.evt_handler = fstorage_evt_handler; fixes the issue where logs weren't being logged properly.

Any reason why m_log_flash_fstorage might be reset to 0x00 on entry into the nrf_log_backend_flash_init() function even though it is being initialized in the NRF_FSTORAGE_DEF() declaration?

RE: NRF LOG crashlog backend

abhipray — Mon, 03 Dec 2018 22:13:31 GMT

OK, found the source of the issue. When NRF_LOG_FINAL_FLUSH() is called when there's no room in the flashlog, the crashlog_marker_inject() function inside nrf_log_backend_crashlog_panic_set() makes a call to nrf_fstorage_write(). nrf_fstorage_write() has a PARAM_CHECK call for addr_is_within_bounds() which fails. On failure, it prints to NRF_LOG_ERROR. At this point, crashlog is in panic state so it accepts that entry and tries another flash write which fails and another NRF_LOG_ERROR function is called ad infinitum/until hardfault.

How do you suggest I patch this issue? If I disable the PARAM CHECK, then nrf_fstorage_write() succeeds and the adjacent flash page is written to which is undesirable.

RE: NRF LOG crashlog backend

abhipray — Mon, 03 Dec 2018 19:52:10 GMT

Some additional info-- When flashlog is filled and I comment out NRF_LOG_FINAL_FLUSH, the rest of the firmware functions fine but it encounters this error right after booting:

[00:00:12.239,000] <error> nrf_fstorage: addr_is_within_bounds(p_fs, dest, len) check failed in nrf_fstorage_write() with value 0x10.

RE: NRF LOG crashlog backend

abhipray — Mon, 03 Dec 2018 19:33:52 GMT

Any connection to this? https://devzone.nordicsemi.com/f/nordic-q-a/38514/nrf52840-logging-crashes-in-freertos

RE: NRF LOG crashlog backend

abhipray — Mon, 03 Dec 2018 19:32:01 GMT

I doubled both defines:

#define configMINIMAL_STACK_SIZE (1000)
#define configTOTAL_HEAP_SIZE (48000)

Again, everything is good until the flashlog gets filled. Then I see the HardFault when NRF_LOG_FINAL_FLUSH is called.

How do you think the FreeRTOS heap config is related?

RE: NRF LOG crashlog backend

Susheel Nuguru — Mon, 03 Dec 2018 16:14:00 GMT

Just a hunch but can you try to increase (maybe double) the below values in your FreeRTOSConfig.h file to see if that helps?

#define configMINIMAL_STACK_SIZE
#define configTOTAL_HEAP_SIZE