fds_gc() garbage collect always causes a reset

[DaraiusH]

Using SDK 17.1.0, SD s140_nrf52_7.2.0

I have FDS working - I can read/write records.  I am using it to store ongoing logs.

When I get to a FDS_ERR_NO_SPACE_IN_FLASH, I delete a certain number of records to make room.  Then call fds_gc().

fds_gc() always causes a reset.  After reset however, it seems to have done the needful, I have the expected amount of flash available again, and I can continue writing records until I hit a NO_SPACE error again.

Any ideas on what to do here ?

Thanks - Daraius

[Jørgen Holmefjord]

Hi,

Calling fds_gc() in itself should not cause a reset. A reset is a typical mechanism used to recover from an error, for instance if a function call returns an unexpected error code, if an assert fails, or if a HardFault occurs in the chip, Do you have logging enabled in your application? The log normally tells you if there was an error, and if you build the application in Debug configuration, the log should tell you exactly where the error occured.

You could also try to debug the application, by setting breakpoints at any calls to NVIC_SystemReset() and read out the call stack to see where the problem originated.

Best regards,
Jørgen

[DaraiusH]

Hi Jorgen
Thanks for your response.  Getting back to this.

I added in a lot of serial debug messages and diagnostic commands to pursue this issue.
I have a command that can generate a large number for records in FDS.  at some point an fds_record_write() results in a FDS_ERR_NO_SPACE_IN_FLASH error.  And then ...
- Pick the N oldest records and proceed to delete them one by one
- Run fds_gc()
- Retry the write that failed and continue with the remaining writes.

In this sequence of actions I see crashes during a Write and/or during fds_gc.
But I have some general questions.

1) Can you write multiple records (say 1000) in a hard for() loop back to back ?  I have been adding an nrf_delay_ms(2) in between each so that other tasks can run
Same with Delete's can you run them back to back ?

2) I am unclear how the fds_event handler works here.
In your FDS example code with the SDK only 3 events are handled -
FDS_EVT_INIT, _WRITE, _DEL_RECORD  and on the FDS_EVT_WRITE there are just NRF_LOG_INFO messages and no flag being set to indicate the operation is complete.
 
There is no handling for FDS_EVT_DEL_FILE or FDS_EVT_GC
Is it necessary to wait on the event handler for each operation ?

Right now my crash is happening like this... in xPortSVHandler

Earlier I have seen the Call stack show a series of Add Queue, Queue_Process calls before crashing.

I have tried greatly increasing the FreeRTOS heap size and the task stack size during xTaskCreate()
configMINIMAL_STACK_SIZE=360,  configTOTAL_HEAP_SIZE=32768

None of that has helped.

[DaraiusH]

Here is capture of a crash when running fds_gc().  In this case I am not using the softdevice.  I have set FDS_BACKEND=1.

This all started with a single call to fds_gc()

Sorry a large screen capture and thats only half the list.  SES does not seem to have a way to copy the contents.

I have seen this exact hard fault many times - dies at vListInsertEnd().

gc_execute seems to run recursively until it runs out of stack ?

[Jørgen Holmefjord]

Yes, it looks like it runs recursively until it runs out of stack.

How/where do you call fds_gc() in the application?

I see that you are using FreeRTOS. Have you tried to reproduce this behavior in a project without FreeRTOS?

Can you provide a minimal project to reproduce the behavior?

[DaraiusH]

It runs from my main App thread.  I have been avoiding trying to build a barebones project to duplicate :)  But since nothing else is obviously wrong I will have to do that.   Was hoping FDS would just work.

i have also considered using FStorage instead and managing flash myself.  Will follow up if I can get a minimal project going.

Thanks for taking a look.

[DaraiusH]

I have now verified that if I use FDS in the exact same sequence from main() without FreeRTOS it works fine.  

I have it using only 4 pages.  I am writing about 300 records.  Then when a fds_record_write() returns FDS_NO_SPACE_IN_FLASH, I delete the oldest 100 records, and then run fds_gc()

Doing the exact same thing from a single FreeRTOS thread crashes every time. Same crash as shown above.

Its the only thread, its the only thing the thread is doing.

	PF2_fds_Init();
	while(true)
	{
		PF2_fds_FillLogPackets();
		nrf_delay_ms(500);
		
		vTaskDelayUntil(&xLastExecutionTime, Convert_ms_to_Ticks(APP_TASK_RATE_MS));  
	}

I have seen a few other posts about FDS problems with FreeRTOS.  No solutions really.

Any help appreciated.  Thanks