RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 15:00:56 GMT

Yup I tested with the original code but their also same delete function is there and after calling delete i am in while loop with idle state handler. Any in delete function it is not showing proper stat but in while loop where i currently running after delete_all() there i ran the stat then it was coming properly(i.e zero).

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 14:51:59 GMT

Thank Vidar Berg I made changes as you said in the example fortunately it is working fine. i will check with my original code and let you know.

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 14:45:02 GMT

I am sorry to say that i forgot to add fds_gc_complete part in this code but it was their in my original project code already. so i don't think this is the issue. okay any have i will test with fds_gc_complete and let you know as soon.

RE: FDS garbage collection not clearing dirty records

Vidar Berg — Mon, 27 Jul 2020 14:05:28 GMT

Thanks, but you're calling fds_stat() immediately after calling fds_gc() so the garbage collection does not get time any to complete.

Code using fds_gc_complete flag:

/**
 * Copyright (c) 2017 - 2019, Nordic Semiconductor ASA
 *
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without modification,
 * are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright notice, this
 *    list of conditions and the following disclaimer.
 *
 * 2. Redistributions in binary form, except as embedded into a Nordic
 *    Semiconductor ASA integrated circuit in a product or a software update for
 *    such product, must reproduce the above copyright notice, this list of
 *    conditions and the following disclaimer in the documentation and/or other
 *    materials provided with the distribution.
 *
 * 3. Neither the name of Nordic Semiconductor ASA nor the names of its
 *    contributors may be used to endorse or promote products derived from this
 *    software without specific prior written permission.
 *
 * 4. This software, with or without modification, must only be used with a
 *    Nordic Semiconductor ASA integrated circuit.
 *
 * 5. Any software provided in binary form under this license must not be reverse
 *    engineered, decompiled, modified and/or disassembled.
 *
 * THIS SOFTWARE IS PROVIDED BY NORDIC SEMICONDUCTOR ASA "AS IS" AND ANY EXPRESS
 * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
 * OF MERCHANTABILITY, NONINFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ARE
 * DISCLAIMED. IN NO EVENT SHALL NORDIC SEMICONDUCTOR ASA OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
 * GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
 * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
 */

#include <stdint.h>
#include <string.h>
#include "nrf.h"
#include "nordic_common.h"
#ifdef SOFTDEVICE_PRESENT
#include "nrf_sdh.h"
#include "nrf_sdh_ble.h"
#else
#include "nrf_drv_clock.h"
#endif
#include "fds.h"
#include "app_timer.h"
#include "app_error.h"
#include "nrf_cli.h"
#include "fds_example.h"

#define NRF_LOG_MODULE_NAME app
#include "nrf_log.h"
#include "nrf_log_ctrl.h"
#include "nrf_delay.h"


/* A tag identifying the SoftDevice BLE configuration. */
#define APP_BLE_CONN_CFG_TAG    1

bool volatile fds_delete_complete = false;
bool volatile fds_gc_complete = false;

static void delete_data(uint16_t file_id, uint16_t record_key);
/* Array to map FDS return values to strings. */
char const * fds_err_str[] =
{
    "FDS_SUCCESS",
    "FDS_ERR_OPERATION_TIMEOUT",
    "FDS_ERR_NOT_INITIALIZED",
    "FDS_ERR_UNALIGNED_ADDR",
    "FDS_ERR_INVALID_ARG",
    "FDS_ERR_NULL_ARG",
    "FDS_ERR_NO_OPEN_RECORDS",
    "FDS_ERR_NO_SPACE_IN_FLASH",
    "FDS_ERR_NO_SPACE_IN_QUEUES",
    "FDS_ERR_RECORD_TOO_LARGE",
    "FDS_ERR_NOT_FOUND",
    "FDS_ERR_NO_PAGES",
    "FDS_ERR_USER_LIMIT_REACHED",
    "FDS_ERR_CRC_CHECK_FAILED",
    "FDS_ERR_BUSY",
    "FDS_ERR_INTERNAL",
};

/* Array to map FDS events to strings. */
static char const * fds_evt_str[] =
{
    "FDS_EVT_INIT",
    "FDS_EVT_WRITE",
    "FDS_EVT_UPDATE",
    "FDS_EVT_DEL_RECORD",
    "FDS_EVT_DEL_FILE",
    "FDS_EVT_GC",
};

/* Dummy configuration data. */
static configuration_t m_dummy_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "dummy",
};

/* A record containing dummy configuration data. */
static fds_record_t const m_dummy_record =
{
    .file_id           = CONFIG_FILE,
    .key               = CONFIG_REC_KEY,
    .data.p_data       = &m_dummy_cfg,
    /* The length of a record is always expressed in 4-byte units (words). */
    .data.length_words = (sizeof(m_dummy_cfg) + 3) / sizeof(uint32_t),
};

/* Keep track of the progress of a delete_all operation. */
static struct
{
    bool delete_next;   //!< Delete next record.
    bool pending;       //!< Waiting for an fds FDS_EVT_DEL_RECORD event, to delete the next record.
} m_delete_all;

/* Flag to check fds initialization. */
static bool volatile m_fds_initialized;


static void fds_evt_handler(fds_evt_t const * p_evt)
{
    NRF_LOG_GREEN("Event: %s received (%s)",
                  fds_evt_str[p_evt->id],
                  fds_err_str[p_evt->result]);

    switch (p_evt->id)
    {
        case FDS_EVT_INIT:
            if (p_evt->result == FDS_SUCCESS)
            {
                m_fds_initialized = true;
            }
            break;

        case FDS_EVT_WRITE:
        {
            if (p_evt->result == FDS_SUCCESS)
            {
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->write.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->write.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->write.record_key);
            }
        } break;

        case FDS_EVT_DEL_RECORD:
        {
            if (p_evt->result == FDS_SUCCESS)
            {
                fds_delete_complete = true;
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->del.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->del.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->del.record_key);
            }
            m_delete_all.pending = false;
        } break;

        case FDS_EVT_GC:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                fds_gc_complete = true;
            }
        } break;

        default:
            break;
    }
}


/**@brief   Begin deleting all records, one by one. */
void delete_all_begin(void)
{
    m_delete_all.delete_next = true;
}


/**@brief   Process a delete all command.
 *
 * Delete records, one by one, until no records are left.
 */
void delete_all_process(void)
{
    if (   m_delete_all.delete_next
        & !m_delete_all.pending)
    {
        NRF_LOG_INFO("Deleting next record.");

        m_delete_all.delete_next = record_delete_next();
        if (!m_delete_all.delete_next)
        {
            NRF_LOG_CYAN("No records left to delete.");
        }
    }
}


#ifdef SOFTDEVICE_PRESENT
/**@brief   Function for initializing the SoftDevice and enabling the BLE stack. */
static void ble_stack_init(void)
{
    ret_code_t rc;
    uint32_t   ram_start;

    /* Enable the SoftDevice. */
    rc = nrf_sdh_enable_request();
    APP_ERROR_CHECK(rc);

    rc = nrf_sdh_ble_default_cfg_set(APP_BLE_CONN_CFG_TAG, &ram_start);
    APP_ERROR_CHECK(rc);

    rc = nrf_sdh_ble_enable(&ram_start);
    APP_ERROR_CHECK(rc);
}
#else
static void clock_init(void)
{
    /* Initialize the clock. */
    ret_code_t rc = nrf_drv_clock_init();
    APP_ERROR_CHECK(rc);

    nrf_drv_clock_lfclk_request(NULL);

    /* Wait for the clock to be ready. */
    while (!nrf_clock_lf_is_running()) {;}
}
#endif


/**@brief   Initialize the timer. */
static void timer_init(void)
{
    ret_code_t err_code = app_timer_init();
    APP_ERROR_CHECK(err_code);
}


/**@brief   Initialize logging. */
static void log_init(void)
{
    ret_code_t rc = NRF_LOG_INIT(NULL);
    APP_ERROR_CHECK(rc);
}


/**@brief   Sleep until an event is received. */
static void power_manage(void)
{
#ifdef SOFTDEVICE_PRESENT
    (void) sd_app_evt_wait();
#else
    __WFE();
#endif
}


/**@brief   Wait for fds to initialize. */
static void wait_for_fds_ready(void)
{
    while (!m_fds_initialized)
    {
        power_manage();
    }
}

#define FILE_ID         0x0001  /* The ID of the file to write the records into. */
#define RECORD_KEY_1    0x1111  /* A key for the first record. */
#define RECORD_KEY_2    0x2222  /* A key for the second record. */
static uint32_t   const m_deadbeef = 0xDEADBEEF;
static char       const m_hello[]  = "Hello, world 2!";


static void write_data();



//#define FILE_ID     0x1111
//#define RECORD_KEY  0x2222

static void read_data();


int main(void)
{
    ret_code_t rc;

#ifdef SOFTDEVICE_PRESENT
    ble_stack_init();
#else
    clock_init();
#endif

    timer_init();
    log_init();
    cli_init();

    NRF_LOG_INFO("FDS example started.")

    /* Register first to receive an event when initialization is complete. */
    (void) fds_register(fds_evt_handler);

    NRF_LOG_INFO("Initializing fds...");

    rc = fds_init();
    APP_ERROR_CHECK(rc);

    /* Wait for fds to initialize. */
    wait_for_fds_ready();

    NRF_LOG_INFO("Available commands:");
    NRF_LOG_INFO("- print all\t\tprint records");
    NRF_LOG_INFO("- print config\tprint configuration");
    NRF_LOG_INFO("- update\t\tupdate configuration");
    NRF_LOG_INFO("- stat\t\tshow statistics");
    NRF_LOG_INFO("- write\t\twrite a new record");
    NRF_LOG_INFO("- delete\t\tdelete a record");
    NRF_LOG_INFO("- delete_all\tdelete all records");
    NRF_LOG_INFO("- gc\t\trun garbage collection");

    NRF_LOG_INFO("Reading flash usage statistics...");

    fds_stat_t stat = {0};

    rc = fds_stat(&stat);
    APP_ERROR_CHECK(rc);

    NRF_LOG_INFO("Found %d valid records.", stat.valid_records);
    NRF_LOG_INFO("Found %d dirty records (ready to be garbage collected).", stat.dirty_records);



//    fds_record_desc_t desc = {0};
//    fds_find_token_t  tok  = {0};
//
//    rc = fds_record_find(CONFIG_FILE, CONFIG_REC_KEY, &desc, &tok);
//
//    if (rc == FDS_SUCCESS)
//    {
//        /* A config file is in flash. Let's update it. */
//        fds_flash_record_t config = {0};
//
//        /* Open the record and read its contents. */
//        rc = fds_record_open(&desc, &config);
//        APP_ERROR_CHECK(rc);
//
//        /* Copy the configuration from flash into m_dummy_cfg. */
//        memcpy(&m_dummy_cfg, config.p_data, sizeof(configuration_t));
//
//        NRF_LOG_INFO("Config file found, updating boot count to %d.", m_dummy_cfg.boot_count);
//
//        /* Update boot count. */
//        m_dummy_cfg.boot_count++;
//
//        /* Close the record when done reading. */
//        rc = fds_record_close(&desc);
//        APP_ERROR_CHECK(rc);
//
//        /* Write the updated record to flash. */
//        rc = fds_record_update(&desc, &m_dummy_record);
//        APP_ERROR_CHECK(rc);
//    }
//    else
//    {
//        /* System config not found; write a new one. */
//        NRF_LOG_INFO("Writing config file...");
//
//        rc = fds_record_write(&desc, &m_dummy_record);
//        APP_ERROR_CHECK(rc);
//    }






cli_start();
uint32_t record_key = 0x0001;
uint32_t val  = 1000;
while(record_key < 0x0A)
{
  write_data(FILE_ID,record_key,val);
  record_key++;
  val++;
}
record_key = 0x0001;
while(record_key < 0x0A){
      read_data(FILE_ID,record_key);
      record_key++;
}
record_key = 0x0001;
while(record_key < 0x0A)
{
  delete_data(FILE_ID,record_key);
  record_key++;
  }


    /* Enter main loop. */
    for (;;)
    {
        if (!NRF_LOG_PROCESS())
        {
            power_manage();
        }
        cli_process();
        delete_all_process();
    }
}


static void write_data(uint32_t file_id,uint32_t record_key,uint32_t data)
{
ret_code_t rc;
fds_record_t        record;
fds_record_desc_t   record_desc = {0};

// Set up record.
record.file_id           = file_id;
record.key               = record_key;
record.data.p_data       = &data;
/* The following calculation takes into account any eventual remainder of the division. */
record.data.length_words = 1/*(sizeof(data) + 3) / 4*/;
rc = fds_record_write(&record_desc, &record);

if (rc == FDS_SUCCESS)
{
    NRF_LOG_INFO("data success %d",record_key);
    while (!NRF_LOG_PROCESS());
    cli_process();
    // Handle error.
}
else{
NRF_LOG_INFO("error writing");
}
}


static void read_data(uint32_t file_id,uint32_t record_key)
{
ret_code_t rc;
fds_flash_record_t  flash_record;
fds_record_desc_t   record_desc = {0};
fds_find_token_t    ftok = {0};
  /* It is required to zero the token before first use. */
memset(&ftok, 0x00, sizeof(fds_find_token_t));
/* Loop until all records with the given key and file ID have been found. */
if (fds_record_find(file_id, record_key, &record_desc, &ftok) == FDS_SUCCESS)
{
     //NRF_LOG_INFO("record_found");
    if (fds_record_open(&record_desc, &flash_record) != FDS_SUCCESS)
    {
        NRF_LOG_INFO("opening_error");
        /* Handle error. */
    }
//    char data[16];
//    memcpy(&data,flash_record.p_data,sizeof(data));
//    for(int i=0;i<sizeof(data);i++)
//    {
//      NRF_LOG_INFO("%c",data[i]);
//    }
    uint32_t recorded_data = 0;
    memcpy(&recorded_data,flash_record.p_data,sizeof(recorded_data));
    NRF_LOG_INFO("recorded_data : %d",recorded_data);
    while (!NRF_LOG_PROCESS());
    cli_process();
    /* Access the record through the flash_record structure. */
    /* Close the record when done. */
    if (fds_record_close(&record_desc) != FDS_SUCCESS)
    {
        NRF_LOG_INFO("error closing the file");
        /* Handle error. */
    }
}
else{
NRF_LOG_INFO("no data ");
}
}
/**
 * @}
 */


static void delete_data(uint16_t file_id, uint16_t record_key) {
  NRF_LOG_INFO("deleting records...");
  ret_code_t rc;
  fds_flash_record_t flash_record;
  fds_record_desc_t record_desc = {0};
  fds_find_token_t ftok = {0};
  /* It is required to zero the token before first use. */
  memset(&ftok, 0x00, sizeof(fds_find_token_t));
  /* Loop until all records with the given key and file ID have been found. */
  int err = fds_record_find(file_id, record_key, &record_desc, &ftok);
  if (err == FDS_SUCCESS) {
    ret_code_t ret = fds_record_delete(&record_desc);
    while (!fds_delete_complete);
    fds_delete_complete = false;
    if (ret == FDS_SUCCESS) {
      NRF_LOG_INFO("delete suc..");
    }
    fds_gc();
    while (!fds_gc_complete);
    fds_gc_complete = false;
    fds_stat_t stat = {0};
     rc = fds_stat(&stat);
     APP_ERROR_CHECK(rc);
     NRF_LOG_INFO("dirty records after gc %d", stat.dirty_records);
  }
}

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 09:27:27 GMT

devzone.nordicsemi.com/.../8688.main.cI attached Modified flash_fds example from SDK 15.3.0 to reproduce the issue. But one thing i am not able to understand was if i send 'stat' command via putty-Terminal then it gives dirty records as '0'. How come?

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 09:16:10 GMT

This is the project I am doing so its difficult to disclose in public. If any other source is there to upload my project privately then it is fine for me or else i will re-produce the issue with some code snippet and post it here within some couple of minutes.

RE: FDS garbage collection not clearing dirty records

Vidar Berg — Mon, 27 Jul 2020 08:51:38 GMT

Are you able to upload your project so I can try to debug it here?

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 08:47:27 GMT

Yes exactly the same whatever you mentioned i was doing.

RE: FDS garbage collection not clearing dirty records

Vidar Berg — Mon, 27 Jul 2020 08:44:42 GMT

But do you check the event result in the FDS_EVENT_GC case?

e.g.,

        case FDS_EVT_GC:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                fds_gc_complete=true
            }

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 08:35:40 GMT

Yes fds_gc_complete is declared as volatile and in event handler i am making it true under FDS_EVENT_GC switch case. Even i put some delay for the fds to complete its operation properly but no use.

RE: FDS garbage collection not clearing dirty records

Vidar Berg — Mon, 27 Jul 2020 08:31:57 GMT

Ok, thanks for confirming. The issue I had in mind was fixed before SDK v15.3.0 so it must be something else. A couple of follow up questions: is the 'fds_gc_complete' declared volatile and do you check that the event result is NRF_SUCCESS before you set fds_gc_complete=true ?

RE: FDS garbage collection not clearing dirty records

Yogeshwaran — Mon, 27 Jul 2020 08:24:45 GMT

SDK 15.3.0

RE: FDS garbage collection not clearing dirty records

Vidar Berg — Mon, 27 Jul 2020 08:17:28 GMT

Hello,

Which SDK version are you using? Early versions of the FDS had a bug that caused the record statistics to not get updated after GC if I remember correctly.

Thanks.