FDS_ERR_NOT_FOUND issue of yps_storage_find in sdk 14.0.1

[edmond.yun]

Hello,

I implemented overall flow related with FDS function including garbage collection, when I check garbage collection, it is ok, it confirm that the data will be saved with writing or updating function in event handler, please refer the following codes, (refer to FDS_EVT_GC in the log)

but I have one problem, FDS_ERR_NOT_FOUND error is sometimes happened with yps_storage_find, when I check the flow of codes, it is happened in yps_storage_write & yps_storage_read function, so I can't read the data at that time,

I can't know the reason, I repeatedly tried to delete record before writing data, so it should be done as to fds_record_write function, is it right ?

please help to let me know the reason and what it is wrong, thankful for your support in advance,

thanks.

Error log :

Event: FDS_EVT_DEL_RECORD received (FDS_SUCCESS) 
Deleted record ID: 242 0x0
yps_storage_find not found 0x1111 0x3333
Event: FDS_EVT_WRITE received (FDS_SUCCESS) 
fds_record_write : 0 243 
yps_password = 159 
Uncalibrated time:	01/01/18 - 15:26:40
Calibrated time:	01/01/18 - 15:26:40
DelEvent: FDS_EVT_DEL_RECORD received (FDS_SUCCESS) 
eted record ID: 243 0x0
yps_storage_find not found 0x1111 0x3333
Event: FDS_EVT_WRITE received (FDS_SUCCESS) 
fds_record_write : 0 244 
yps_password = 160 
Uncalibrated time:	01/01/18 - 15:26:41
Calibrated time:	01/01/18 - 15:26:41
Event: FDS_EVT_DEL_RECORD received (FDS_SUCCESS) 
Deleted record ID: 244 0x0
yps_storage_find not found 0x1111 0x3333
fds_record_write : 0 245 
yps_storage_read not found 0xEvent: FDS_EVT_WRITE received (FDS_SUCCESS) 
1111 0x3333
yps_password = 160 
Uncalibrated time:	01/01/18 - 15:26:42
Calibrated time:	01/01/18 - 15:26:42
Event: FDS_EVT_DEL_RECORD received (FDS_SUCCESS) 
Deleted record ID: 245 0x0
yps_storage_find not found 0x1111 0x3333
Event: FDS_EVT_WRITE received (FDS_SUCCESS) 
fds_record_write : 0 246 
yps_password = 162 
Uncalibrated time:	01/01/18 - 15:26:43
Calibrated time:	01/01/18 - 15:26:43
Event: FDS_EVT_DEL_RECORD received (FDS_SUCCESS) 

firstly, in the main routine,

if(password[0] == 250)
	password[0] = 0;
	
password[0]++;

yps_storage_password_delete();
					
err_code = yps_storage_password_write(password);
APP_ERROR_CHECK(err_code);

uint8_t yps_password[YPS_PASSWORD_LENGTH];
err_code = yps_storage_password_read(yps_password);	    
APP_ERROR_CHECK(err_code);
printf("yps_password = %d \r\n", yps_password[0]);

_1ms_count = 0;	

and then each functions of FDS,

#include <string.h>
#include "hal_fds_module.h"
#include "fds.h"
#include "app_error.h"
#include "app_util.h"
#include "app_util_bds.h"

//#pragma pack(push, 4)
//
//struct schedule_db_schema {
//	uint8_t day;
//	uint8_t hour;
//	uint8_t minute;
//};
//static struct schedule_db_schema db_schema;// __ALIGN(4);
//#pragma pack(pop)
 
static volatile bool 		m_fds_evt_init = false;
static volatile bool 		m_pending_write  = false;      /**< Flag used to preserve write request during Garbage Collector activity. */
static volatile bool 		m_pending_update = false;      /**< Flag used to preserve update request during Garbage Collector activity. */

static uint16_t	       		m_pending_msg_key    = 0;//love_0124
static uint32_t        		m_pending_msg_size   = 0;    /**< Pending write/update request data size. */
static uint8_t const * 		m_p_pending_msg_buff = NULL; /**< Pending write/update request data pointer. */

static fds_record_desc_t  	record_desc;            /**< Record descriptor. */
static fds_record_t       	rec;                 /**< Record description used for writes. */

/* Array to map FDS return values to strings. */
char const * fds_err_str[] =
{
    "FDS_SUCCESS",
    "FDS_ERR_OPERATION_TIMEOUT",
    "FDS_ERR_NOT_INITIALIZED",
    "FDS_ERR_UNALIGNED_ADDR",
    "FDS_ERR_INVALID_ARG",
    "FDS_ERR_NULL_ARG",
    "FDS_ERR_NO_OPEN_RECORDS",
    "FDS_ERR_NO_SPACE_IN_FLASH",
    "FDS_ERR_NO_SPACE_IN_QUEUES",
    "FDS_ERR_RECORD_TOO_LARGE",
    "FDS_ERR_NOT_FOUND",
    "FDS_ERR_NO_PAGES",
    "FDS_ERR_USER_LIMIT_REACHED",
    "FDS_ERR_CRC_CHECK_FAILED",
    "FDS_ERR_BUSY",
    "FDS_ERR_INTERNAL",
};

/* Array to map FDS events to strings. */
static char const * fds_evt_str[] =
{
    "FDS_EVT_INIT",
    "FDS_EVT_WRITE",
    "FDS_EVT_UPDATE",//love_0124 Event for fds_record_update.
    "FDS_EVT_DEL_RECORD",//love_0124 Event for fds_record_delete.
    "FDS_EVT_DEL_FILE",
    "FDS_EVT_GC",
};

/**
 * @brief   Prepare record structure for write or update request.
 *
 * @details Configures file ID, record KEY, data to be written and message length.
 *
 * @param[in] buff  Pointer to the YPS message to be stored in FLASH.
 * @param[in] size  Size of YPS message.
 */
static void yps_file_prepare_record(uint16_t key, uint8_t const * p_buff, uint32_t size)
{
	// Set up record.
	rec.file_id           = FILE_ID;
	rec.key               = key;
	rec.data.p_data       = p_buff;
	rec.data.length_words = BYTES_TO_WORDS(size); // Align data length to 4 bytes.
}

static void yps_storage_fds_evt_handler(fds_evt_t const * const p_fds_evt)
{
	ret_code_t ret;
		
	printf("Event: %s received (%s) \r\n", fds_evt_str[p_fds_evt->id], fds_err_str[p_fds_evt->result]);
                  	
	switch (p_fds_evt->id)
	{
		case FDS_EVT_INIT:
//			printf("FDS_EVT_INIT %x\r\n", p_fds_evt->result);
			
			if (p_fds_evt->result != FDS_SUCCESS)
				printf("FDS_EVT_INIT Initialization failed\r\n");
			else
				m_fds_evt_init = true;
		break;
		case FDS_EVT_WRITE:
//			printf("FDS_EVT_WRITE %x\r\n", p_fds_evt->result);
//			
//			if (p_fds_evt->result == FDS_SUCCESS)
//				printf("FDS_EVT_WRITE FDS_SUCCESS\r\n");
//			else 
//				printf("FDS_EVT_WRITE Failed!!!\r\n");
		break;
		
		case FDS_EVT_UPDATE:
//			printf("FDS_EVT_UPDATE %x\r\n",p_fds_evt->result);
		break;
		case FDS_EVT_DEL_RECORD:
//			printf("FDS_EVT_DEL_RECORD %x\r\n", p_fds_evt->result);
//			ret = fds_gc();
//			APP_ERROR_CHECK(ret);
		break;
		case FDS_EVT_DEL_FILE:
//			printf("FDS_EVT_DEL_FILE %x\r\n",p_fds_evt->result);
		break;
		case FDS_EVT_GC:
			//Perform pending write/update.
			if (m_pending_write)
			{
				printf("FDS_EVT_GC m_pending_write %x\r\n", p_fds_evt->result);
				m_pending_write = false;
				// Prepare record structure.
				yps_file_prepare_record(m_pending_msg_key, m_p_pending_msg_buff, m_pending_msg_size);				
				ret = fds_record_write(&record_desc, &rec);
				printf("FDS_EVT_GC fds_record_write : %x %d \r\n", ret, record_desc.record_id);
				APP_ERROR_CHECK(ret);
			}
			else if (m_pending_update)
			{
				printf("FDS_EVT_GC m_pending_update %x\r\n", p_fds_evt->result);				
				m_pending_update = false;
				// Prepare record structure.
				yps_file_prepare_record(m_pending_msg_key, m_p_pending_msg_buff, m_pending_msg_size);				
				ret = fds_record_update(&record_desc, &rec);
				printf("FDS_EVT_GC fds_record_update : %x %d \r\n", ret, record_desc.record_id);				
				APP_ERROR_CHECK(ret);
			}
		
		break;
		
		default:
		break;
	}
}

ret_code_t yps_storage_init(void) {
	
	ret_code_t ret = fds_register(yps_storage_fds_evt_handler);

	if (ret != FDS_SUCCESS) {
		return ret;
	}

	ret = fds_init();

	if (ret != FDS_SUCCESS) {
		return ret;
	}

	fds_stat_t stat = {0};
	
	ret = fds_stat(&stat);//love_0123
	APP_ERROR_CHECK(ret);
    
	return NRF_SUCCESS;
}

bool yps_storage_find(uint16_t file_id, uint16_t key, fds_record_desc_t * p_desc) {
	
	ret_code_t ret;
	fds_find_token_t ftok;
	
	memset(&ftok, 0, sizeof(fds_find_token_t));
	
	ret = fds_record_find(file_id, key, p_desc, &ftok);
//	APP_ERROR_CHECK(ret);
	
	if (ret == FDS_SUCCESS) {
		printf("fds_record_find Record ID = %d\r\n", p_desc->record_id);
		return true;
	}

	printf("yps_storage_find not found 0x%x 0x%x\r\n", file_id, key);

	return false;
}

ret_code_t yps_storage_write(uint16_t file_id, uint16_t key, uint8_t const * p_data, uint16_t length_bytes) {
	
	ret_code_t ret;
//	fds_record_desc_t record_desc;
//	
//	memset(&record_desc, 0, sizeof(record_desc));

//	fds_record_t const rec =
//	{
//		.file_id           = file_id,
//		.key               = key,
//		.data.p_data       = &p_data,
//		.data.length_words = 1//BYTES_TO_WORDS(length_bytes)
//	};

	// Prepare record structure.
	yps_file_prepare_record(key, p_data, length_bytes);
	
	if(yps_storage_find(file_id, key, &record_desc)) 
	{
		// update
		ret = fds_record_update(&record_desc, &rec);
		printf("fds_record_update : %x %d \r\n", ret, record_desc.record_id);
		
		if (ret == FDS_ERR_NO_SPACE_IN_FLASH)//love_0124
		{
		        // If there is no space, preserve write request and call Garbage Collector.
		        m_pending_update     = true;
		        m_pending_msg_key    = key;//love_0124
		        m_pending_msg_size   = length_bytes;
		        m_p_pending_msg_buff = p_data;

			ret = fds_gc();
			APP_ERROR_CHECK(ret);
		}
	} 
	else 
	{
		// new
		ret = fds_record_write(&record_desc, &rec);
		printf("fds_record_write : %x %d \r\n", ret, record_desc.record_id);

		if (ret == FDS_ERR_NO_SPACE_IN_FLASH)//love_0124
		{
			// If there is no space, preserve update request and call Garbage Collector.
		        m_pending_write      = true;
		        m_pending_msg_key    = key;//love_0124		        
		        m_pending_msg_size   = length_bytes;
		        m_p_pending_msg_buff = p_data;
			
			ret = fds_gc();
			APP_ERROR_CHECK(ret);
		}
	}

	return ret;
}

ret_code_t yps_storage_read(uint16_t file_id, uint16_t key, uint8_t * p_data) {
	
	ret_code_t ret;
	fds_flash_record_t  	flash_record = {0};
	fds_find_token_t 	ftok = {0};

//	memset(&ftok, 0x00, sizeof(fds_find_token_t));
		
	ret = fds_record_find(file_id, key, &record_desc, &ftok);
//	APP_ERROR_CHECK(ret);
	
	if(ret != FDS_SUCCESS)
	{
		printf("yps_storage_read not found 0x%x 0x%x\r\n", file_id, key);
		return ret;
	}
	
	ret = fds_record_open(&record_desc, &flash_record);
	APP_ERROR_CHECK(ret);
	
	if (ret != FDS_SUCCESS)
		return ret;
	// Access the record through the flash_record structure.
	memcpy(p_data, flash_record.p_data, (flash_record.p_header->length_words * sizeof(uint32_t)));//sizeof(uint32_t) BYTES_PER_WORD
	
	// Close the record when done.
	ret = fds_record_close(&record_desc);
	APP_ERROR_CHECK(ret);
	
	if (ret != FDS_SUCCESS)
		return ret;

	return ret;
}

ret_code_t yps_storage_find_and_delete(uint16_t file_id, uint16_t key)
{
	ret_code_t ret;
//	fds_record_desc_t record_desc;
	fds_find_token_t ftok;
	
//	memset(&record_desc, 0, sizeof(fds_record_desc_t));	
	memset(&ftok, 0, sizeof(fds_find_token_t));
	
	// Loop and find records with same ID and rec key and mark them as deleted.
	while (fds_record_find(file_id, key, &record_desc, &ftok) == FDS_SUCCESS)
	{
		ret = fds_record_delete(&record_desc);
		printf("Deleted record ID: %d 0x%x\r\n", record_desc.record_id, (unsigned int) ret);
	}

//	// call the garbage collector to empty them, don't need to do this all the time, this is just for demonstration
//	ret = fds_gc();
//
//	if (ret != FDS_SUCCESS)
//	{
//		return ret;
//	}

	return NRF_SUCCESS;
}

ret_code_t yps_storage_id_write(uint8_t * p_id) {
	return yps_storage_write(FILE_ID, REC_KEY_ID, p_id, YPS_ID_LENGTH);
}

ret_code_t yps_storage_password_write(uint8_t * p_password) {
	return yps_storage_write(FILE_ID, REC_KEY_PASSWORD, p_password, YPS_PASSWORD_LENGTH);
}

ret_code_t yps_storage_id_read(uint8_t * p_id) {
	ret_code_t ret = yps_storage_read(FILE_ID, REC_KEY_ID, p_id);
	return ret;
}

ret_code_t yps_storage_password_read(uint8_t * p_password) {
	ret_code_t ret = yps_storage_read(FILE_ID, REC_KEY_PASSWORD, p_password);
	return ret;
}

ret_code_t yps_storage_id_delete(void) {
	return yps_storage_find_and_delete(FILE_ID, REC_KEY_ID);
}

ret_code_t yps_storage_password_delete(void) {
	return yps_storage_find_and_delete(FILE_ID, REC_KEY_PASSWORD);
}

bool get_fds_evt_init(void) {
	return m_fds_evt_init;
}

[Petter Myhre]

You should always check the returned errors, but not all errors are unrecoverable. I'm not sure why you wouldn't check the error?

[edmond.yun]

Hi, Petter Myhre,

I refer your guide, and understand the reason that the FDS_ERR_NOT_FOUND error is happened, 

after deleting record repeatedly and if trying to read and write data, maybe the fds management system can have the timing issue... so I think that if only using to update data (not using deleting record), the issue is not happened, but need to confirm the necessary memory with DFU_APP_DATA_RESERVED in sdk_config.h... is it right ?

if the size of data is not over to the memory, most errors are not happened...is it right ?

if I have mis-understanding, please help to provide the perfect example to use for mass-production,

when I check the example of sdk 14.1.0, it is not enough to use for production,

ex) if failing to update or write data, the handler that received the result need to try the behavior again after checking result...

thankful for your support in advance, (can see the case with FDS_EVT_GC in the attached code).

thanks.

[Petter Myhre]

I'm sorry, I don't understand. What does DFU_APP_DATA_RESERVED have to do with this? Do you still have a problem? If you do, could you try to clearly explain what it is?

[edmond.yun]

Hi, Petter, 

I am  very thankful for your care and concern, I need to really use fstorage with exact information,

I will explain what I know for the function, please refer below,

1. I searched for app data, I can keep user data in the region of memory map and I can allocate the size of the memory with DFU_APP_DATA_RESERVED in sdk_config.h...

you can find more information in site of nordic development description like api description so on....

I need to keep the user data with fstorage in the region of app data memory, it is a fundamental question for this agenda.... How can I confirm the usage of app data memory with fstorage ? how can define the address of start/end ?

the data should be kept after doing dfu...it is also important for mass-production, please refer it,

please help to provide the useful example of fstorage by modifying my code,

thankful for your support in advance,

thanks.

[edmond.yun]

Hi, Petter,

sorry for any confusion, I confused this agenda with other...sorry...

for only this agenda, currently I am not using for delete_record function, 

instead, I confirmed the size of memory by using  DFU_APP_DATA_RESERVED, it can be defined by user in sdk_config.h... and it is related with the region of app data, when verifying by myself, the data can kept in the region of app data(maybe) after doing DFU... I tried to verify for many times by firstly writing some data with field value, after that, I tried to read the data with the same field data by using fds function(not fstorage function). so I learned that if can confirm the size of necessary data with DFU_APP_DATA_RESERVED in the region of app data (maybe, I am'not sure that the region of DFU_APP_DATA_RESERVED  is the region of app data),  I don't need to delete the record every time (when writing new data in the same field) because new data will be updated (not written) by update_fds function.. is it right ?

so currently I can't find the error related with find function because I am not using delete_record_fds function. how about my usage ? is it right for mass production ? 

please help to let me have the confirmation about my code...

thankful for your support in advance,

thanks.

[Petter Myhre]

You can check the start and end address used by Fstorage by putting breakpoint in flash_bounds_set() in fds.c. If you have a bootloader flash_end_addr() will account for that and use the flash pages below. The SDK examples typically reserves 3 flash pages.

DFU_APP_DATA_RESERVED determines how many flash pages below the bootloader that will be reserved (not erased when doing DFU). This should match the number of pages used by Fstorage, typically 3.