Storing Data to flash memory

RE: Storing Data to flash memory

Edvin — Mon, 06 Jan 2020 13:19:48 GMT

Hello,

Desc is the descriptor which is used to find and store values in records. config is the variable (fds_flash_record_t) that holds the actual flash data. You can check the definitions of these variable types in fds.h for an explanation.

Have you seen the usage section on infocenter? It contains some pointers on how to use the records. So check out the variable types and this guide.

Best regards,

Edvin

RE: Storing Data to flash memory

Thomas — Sun, 05 Jan 2020 13:21:42 GMT

Im using segger.

On the line:

  rc = fds_record_open(&desc, &config);//Read data

How do I read the last entry. I presume the config that is in the flash is the default dummy one.

So my question is what is desc? Is this the specfic instance that is read?

Because I want to call a function that I can read the last record from. I will later need to read all the data. But I am thinking of doing this in a for loop from the first entry to the last.

RE: Storing Data to flash memory

Edvin — Thu, 19 Dec 2019 13:05:28 GMT

If you set a breakpoint and then resume, the softdevice will assert. That is fine, you just need to reset.

Just set a breakpoint at the line memcpy, and right click config.p_data and click "add to watch". What this button is called depends on your IDE.

If you are not able to see the content of the pointer, then you need to disable optimization in your project.If you are not sure how to do that, let me know what IDE you are using.

RE: Storing Data to flash memory

Thomas — Thu, 19 Dec 2019 11:40:30 GMT

When I place a breakpoint on memcpy(&m_sensor_cfg, config.p_data, sizeof(configuration_t));

I run over it and, it goes to NRF_BREAKPOINT_COND.

So I am not sure how to test it.

I presume I am writing the information incorrectly and reading it incorrectly.

Given the aforementioned line of code should print out "sensor" as this is what I believe I write to the configuration file.

However, I don't want to read the configuration file. I want to read the latest data.

But, I can't find an example of where to do that in the sdk example code. Or the documentation.

RE: Storing Data to flash memory

Edvin — Thu, 19 Dec 2019 10:52:04 GMT

What does p_data look like in the flash_storage_read_data() function right before you copy it over to the m_sensor_cfg?

Try to set a breakpoint and watch that data.

Did you wait for the write complete event before you read it?

RE: Storing Data to flash memory

Thomas — Thu, 19 Dec 2019 09:20:34 GMT

That is the line of code that should pull the data. I want that line of code to output the latest data I have entered into my flash storage.

The line of code you highlighted always prints "Reading data dummy" even though the data I am pointing at in the line of code is the m_sensor_cfg. This has a device name set to "Sensor" not "dummy" thus my confusion.

I am writing a new entry via my flash_storage_write_data function. This is the data I wish to read (latest).

RE: Storing Data to flash memory

Edvin — Thu, 19 Dec 2019 09:16:57 GMT

I don't know what to read out of your snippets. Is it the lines in the log saying "reading data dummy" that you refer to?

Beause the only place I can see that you print that is from the line:

NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);//Print out the configurations bootcount member variable

What do you expect it to say? Where does "dummy" come from, and where do you update it?

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 14:47:03 GMT

I think it might be because I am always reading the same data? Which, is set by the code before mine. The dummy code. Therefore for reading the data how do I point it to the latest data inside of the flash storage?

Also are multiple configurations allowed in the fds_example.h?

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 14:12:36 GMT

Ok thanks I won't call fds_gs too often,.

I still have the issue of my code not reading from the correct location.

I have setup two data configurations. One is the dummy it is unchanged.

The other is custom see below:

/* Sensor configuration data. */
static configuration_t_sensor m_sensor_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "sensor",
    .count_id    = 0,
    .t1_data     = 0,
    .t2_data     = 0,
    .t3_data     = 0,
    .p1_data     = 0,
    .p2_data     = 0,
};

/* A record containing dummy configuration data. */
static fds_record_t const m_sensor_record =
{
    .file_id           = CONFIG_FILE,
    .key               = CONFIG_REC_KEY,
    .data.p_data       = &m_sensor_cfg,
    /* The length of a record is always expressed in 4-byte units (words). */
    .data.length_words = (sizeof(m_sensor_cfg) + 3) / sizeof(uint32_t),
};

When I print my data to the RTT view it always reads the dummy data.

NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);

I am going to assume this is a configuration / setup issue.

Also when running the code and looping a function to add. I fill up the flash very fast.

0> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x0087
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010
00> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x0088
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010
00> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: No space in flash, delete some records to update the config file.
00> <info> app: Received string: "1,1191,1,1172
00> "

Note the t1 and t2 data are coming in from a UART this is the data I wish to store.

RE: Storing Data to flash memory

Edvin — Wed, 18 Dec 2019 13:33:16 GMT

Sorry, I forgot to comment your last post.

Well. The easiest would probably to erase the entire flash if you end up with a lot of records. The easiest way is to use Nordic Command Line Tools, and the command "nrfjprog -e". This will delete everything on the chip.

However, You can see from your log that you have 100 dirty records, and only 4 valid records. This means that you can run a garbage collection.

Whenever you either update a record or delete a record, it is marked for deletion. This is what we call a dirty record. The reason it isn't deleted and freeing up the space immediately is because of the nature of the flash. You need to erase large chunks at once, and you can only write one time to a specific location once between deleting it again (in fact, you can only write 1's to 0's in a flash, before erasing it, which will reset it to 1's. This is why an empty flash is always 0xFF's).

So, when either fds_record_write or fds_record_update returns NRF_ERROR_NO_MEM, you need to run a garbage collection by calling fds_gc(). This is a complex call, moving the records around to free space (one flash page is always reserved for this in FDS), and deleting the old pages, so that it has room for new records. Remember to wait for the callback for this call before you try to read or write something else.

The event that you should add to your fds_event handler is called FDS_EVT_GC. Check that it returned with p_evt->result == NRF_SUCCESS before setting a flag saying it is ok to continue.

Also, don't fall for the temptation of running fds_gc() on every startup. This will beat up the flash. The flash on the nRFs are guaranteeing 10000 flash erase/write cycles. If you run fds_gc() on every startup, this will be limited to 10000 resets.

The flash will not stop working after 10 001 cycles, but this is what we can say for sure that will work.

It is not unusual that it will work for 40-50 000 cycles, or even 100 000 cycles, but still, there is no need to beat up the flash by running an fds_gc() before it is necessary.

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 13:19:19 GMT

Thanks for the Edvin,

I've written that in now.

But I cant test it because my flash is full.

00> <info> app: Initializing fds...
00> <info> app: Event: FDS_EVT_INIT received (NRF_SUCCESS)
00> <info> app: Found 4 valid records.
00> <info> app: Found 100 dirty records (ready to be garbage collected).
00> <info> app: Writing config file...
00> <info> app: No space in flash, delete some records to update the config file.
00> <info> app: Event: FDS_EVT_DEL_FILE received (NRF_SUCCESS)
00> <info> app: No space in flash, delete some records to update the config file.

Therefore, I need to delete the data. How do I delete all of the data?

I've tried fds_file_delete(&desc); which does not work.

Thanks,

Thomas

RE: Storing Data to flash memory

Edvin — Wed, 18 Dec 2019 13:12:21 GMT

I didn't understand what exactly you are trying to tell me, but I guess you are not reading the same as you are writing?

[quote user="ThomasMorris"]If so is there a statement that says something like wait till fds ready.[/quote]

Yes.

Do you have an event handler for your FDS instance? You probably have a call like this somewhere:

fds_register(fds_evt_handler);

If so, fds_evt_handler is your fds callback function.

What you need to do is to wait after you have changed any of the flash content. You can do so by using a flag:

//Top of main.c

volatile bool fds_write_complete_flag = false;


static void fds_evt_handler(fds_evt_t const * p_evt)
{
    if (p_evt->result == NRF_SUCCESS)
    {
        NRF_LOG_GREEN("Event: %s received (NRF_SUCCESS)",
                      fds_evt_str[p_evt->id]);
    }
    else
    {
        NRF_LOG_GREEN("Event: %s received (%s)",
                      fds_evt_str[p_evt->id],
                      fds_err_str(p_evt->result));
    }

    switch (p_evt->id)
    {
        case FDS_EVT_INIT:
            if (p_evt->result == NRF_SUCCESS)
            {
                m_fds_initialized = true;
            }
            break;

        case FDS_EVT_WRITE:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->write.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->write.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->write.record_key);
                fds_write_complete_flag = true;
            }
        } break;

        case FDS_EVT_DEL_RECORD:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->del.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->del.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->del.record_key);
            }
            m_delete_all.pending = false;
        } break;

        default:
            break;
    }
}

int main(void)
{
    (void) fds_register(fds_evt_handler);
    
    flash_storage_read_data();
    flash_storage_write_data(00,00,00,00,00,00,00,00,00);//Write the information to the flash storage
    while (fds_write_complete_flag == false)
    {
        // Wait...
    }
    fds_write_complete_flag = false;
    flash_storage_read_data();

See how I use the FDS_EVT_WRITE event to set the fds_write_complete_flag to true, and how I wait in the main() function for this event to set the flag before I read it.

BR,

Edvin

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 12:18:17 GMT

Also when I run the write code in a loop i quickly ran out of space.

It comes up with :No space in flash, delete some records to update the config file.

How do I delete the entrants? ( All of them)

I presume it has something to do with fds_record_delete within the fds.c code.

Thanks,

Thomas

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 12:09:41 GMT

Cheers Edvin, I've decided not to use CLI.

I want to be able to store data and then read it.

At the moment I have some code that adds an entry to the flash storage and then reads it.

This is called after the dummy entry has been added.

flash_storage_read_data();
flash_storage_write_data(00,00,00,00,00,00,00,00,00);//Write the information to the flash storage
flash_storage_read_data();

These functions are written as below

static void flash_storage_write_data(int count, int T1_value,int T2_value,
                                       int t3_value_1,int t3_value_2, int p1_value_1, int p1_value_2,
                                       int p2_value_1,int p2_value_2){
    ret_code_t rc;
    fds_record_desc_t desc = {0};
    //Need to modify the sensor record here
    rc = fds_record_write(&desc, &m_sensor_record);//Write data to the sensor record
        if ((rc != NRF_SUCCESS) && (rc == FDS_ERR_NO_SPACE_IN_FLASH))
        {
            NRF_LOG_INFO("No space in flash, delete some records to update the config file.");
        }
        else
        {
            APP_ERROR_CHECK(rc);
            NRF_LOG_INFO("Data added successfully");
        }
}

static void flash_storage_read_data()//Function to read the data from the flash storage
{
    ret_code_t rc;
    fds_record_desc_t desc = {0};
    fds_find_token_t  tok  = {0};

    rc = fds_record_find(CONFIG_FILE, CONFIG_REC_KEY, &desc, &tok);

    if (rc == NRF_SUCCESS)
    {
        /* A config file is in flash. Let's update it. */
        fds_flash_record_t config = {0};

        /* Open the record and read its contents. */
        rc = fds_record_open(&desc, &config);//NEed to ensure it opens sensor config
        APP_ERROR_CHECK(rc);

        /* Copy the configuration from flash into m_sensor_cfg. */
        memcpy(&m_sensor_cfg, config.p_data, sizeof(configuration_t_sensor));

        NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);//Print out the configurations bootcount member variable

        /* Update boot count. */
        m_dummy_cfg.boot_count++;

        /* Close the record when done reading. */
        rc = fds_record_close(&desc);
        APP_ERROR_CHECK(rc);
     }
}

I have defined my own configuration in the fds_example.h to contain sensor data information:

//Sensor Config data
typedef struct
{
    uint32_t boot_count;
    char     device_name[16];
    bool     config1_on;
    bool     config2_on;
    
    int count_id;
    int t1_data;
    int t2_data;
    int t3_data;
    int p1_data;
    int p2_data;
}configuration_t_sensor;

However, when I run my code I receive:

00> <info> app: UART example started.
00> <info> app: Initializing fds...
00> <info> app: Event: FDS_EVT_INIT received (NRF_SUCCESS)
00> <info> app: Found 54 valid records.
00> <info> app: Found 18 dirty records (ready to be garbage collected).
00> <info> app: Config file found, updating boot count to 8.
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Reading data dummy.
00> <info> app: Event: FDS_EVT_UPDATE received (NRF_SUCCESS)
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x004C
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010

Even though I have configured my sensor data configuration to be:

static configuration_t_sensor m_sensor_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "Sensor",
    .count_id    = 0,//Int to to be passed in
    .t1_data     = 0,//Int to to be passed in
    .t2_data     = 0,//Int to to be passed in
    .t3_data     = 0,//Int to to be passed in
    .p1_data     = 0,//Int to to be passed in
    .p2_data     = 0,//Int to to be passed in
};

I am not sure why this is. Could it have something to do with the flash storage taking a long time as you stated? If so is there a statement that says something like wait till fds ready. However, surely it should read the configuration file I have selected.

Cheers,

Thomas

RE: Storing Data to flash memory

Edvin — Wed, 18 Dec 2019 11:53:48 GMT

You can store data to flash without the CLI library. What you can do is to look at what the different CLI commands. Unfortunately, it is a bit tricky to trace what these commands to, which is why I find it a bit unfortunately that the FDS examples use this.

Basically, you can use UART to send a string of data to the nRF, then use the fds_record_write() or fds_record_update() to save this data.

If you want to dig into why it crashes with "fatal error", this means that one of your APP_ERROR_CHECK(err_code) receives an err_code != 0. If you want to find out which one, I suggest that you define DEBUG in your preprocessor defines (let me know if you are not sure how to do that, and what SDK version you use, and what IDE you use).

Remember that if you write or update a record using fds_record_write/update(), then you need to wait for the FDS callback before you can read it. This is typically not a problem in the CLI case, since the time you use to type the different commands is very long compared to the time it takes to save the data to flash, but if you do it sequentially in an application, make sure to wait for the callback before you try to read the flash record after it is updated.

Best regards,

Edvin

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 10:17:42 GMT

Cheers I got it to build and, it crashes with fatal error. I don't need the cli but need to be able to write data to the flash which, in the fds_storage application i was using the command line to test.

My question therefore, is how do I store the data?

Note if I uncomment the cli_init it does not crash. And if I had it in and changed my pin definitions from my UART peripheral to the default the behaviour is the same as the fds_flash storage example.

Hope I explained this well.

I presume I do something like this:

static configuration_t m_dummy_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "dummy",
};

/* A record containing dummy configuration data. */
static fds_record_t const m_dummy_record =
{
    .file_id           = CONFIG_FILE,
    .key               = CONFIG_REC_KEY,
    .data.p_data       = &m_dummy_cfg,
    /* The length of a record is always expressed in 4-byte units (words). */
    .data.length_words = (sizeof(m_dummy_cfg) + 3) / sizeof(uint32_t),
};
//In main loop somewhere

rc = fds_record_write(&desc, &m_dummy_record);
//I want to set this as a string and as a new record

But can I pass in a string?

I want to set this as a string and as a new record.

If not is it best to store the data as a struct or something of the like?

Also how do I read this data as in the actual data inside the flash storage. Using the boot count value works but I would like to see all of the data stored.

Also each time I reboot the dirty records increase by 1.

RE: Storing Data to flash memory

Edvin — Wed, 18 Dec 2019 10:08:54 GMT

If I were you, I'd try to stay clear of the CLI library, unless you intentionally are trying to use this. There is a uart library that e.g. the ble_app_uart example uses, that doesn't use the CLI. CLI tend to make things a bit complicated.

However, if you need the CLI, then it is possible. Your compiler doesn't find the function nrf_cli_uart_transport_api(). You are probably missing the .c file that has this function. Then you also need the define:

NRF_CLI_UART_ENABLED to be defined to 1 in sdk_config.h.

But try to use the uart library that is used in the ble_app_uart example.

RE: Storing Data to flash memory

Thomas — Wed, 18 Dec 2019 08:36:09 GMT

Hi Edvin, I'll look into that I have had a lot of trouble with the UART example though from a previous issue. https://devzone.nordicsemi.com/f/nordic-q-a/54979/receiving-data-from-uart-via-a-peripheral-device

For more details.

I am trying to merge the flash_fds code and my ble_app_beacon code. Presently I have an undefined reference to fds_register and many other error see below.

  Linking ble_app_beacon_pca10056_s140.elf
    Output/ble_app_beacon_pca10056_s140 Release/Obj/cli.o:D:\Segger projects\Segger Projects\DeviceDownload2\examples\ble_peripheral\ble_app_beacon/cli.c:87: undefined reference to `nrf_cli_uart_transport_api'
Build failed

However, I have included in main:

#ifdef SOFTDEVICE_PRESENT
#include "nrf_sdh.h"
#include "nrf_sdh_ble.h"
#else
#include "nrf_drv_clock.h"
#endif
#include "fds.h"
#include "app_timer.h"
#include "app_error.h"
#include "nrf_cli.h"
#include "fds_example.h"

#define NRF_LOG_MODULE_NAME app
#include "nrf_log.h"
#include "nrf_log_ctrl.h"


/* A tag identifying the SoftDevice BLE configuration. */
#define APP_BLE_CONN_CFG_TAG    1

Also in my processor includes I have added:

../../../../../../components/libraries/cli/uart
../../../../../../examples/peripheral/flash_fds
../../../../../components/libraries/uart
../../../../../../components/libraries/fifo
../../../../../../components/libraries/queue

I have also added a bunch of the c files required.

I have changed these lines of code in the sdk to match the flash storage sdk config

CRC16_ENABLED 1//was 0
FS_ENABLED 1//was 0
FDS_CRC_CHECK_ON_READ 1//was 0
NRF_FSTORAGE_ENABLED 1//was 0
NRF_CLI_ENABLED 1//was 0
NRF_QUEUE_ENABLED 1 // Was 0
NRF_QUEUE_CLI_CMDS 1// Was 0

Presently I have undefined references to nrf_queue_generic_pop and some more see above.

I am not sure how to solve the nrf_cli_uart_transport api being undefined.

If within nrf_cli_uart.c I uncomment the line if NRF_module_enabled(NRF_CLI_UART) it builds. But that therefore, means that the nrf_module is not enabled. All the code after that line is greyed out because it is not defined.

From looking at this : undefined reference I ensured NRF_CLI_ENABLED 1 but, it already was.

RE: Storing Data to flash memory

Edvin — Tue, 17 Dec 2019 16:50:56 GMT

You need to use a BLE service + characteristic to transfer this data. The characteristic you need to use probably look a lot like the service that is implemented in the ble_app_uart example. This uses something called notifications to transfer data. In that example, it uses ble_nus_data_send() to send a text string, (or any other array of bytes).

If you look into this example, which you will find in SDK\examples\ble_peripheral\ble_app_uart, you can see that it usually calls ble_nus_data_send() inside the uart_event_handle(), but you can send it at any other place in the code as well.

Also note that you can queue several packets using ble_nus_data_send(), until it returns NRF_ERROR_RESOURCES. Once that is returned, you must wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event (which you can add to ble_evt_handler() in main.c), that means that one packet is ACKed by the connected device, and there is some space free in the softdevice queue, and you can start queuing new packets.

Best regards,

Edvin

RE: Storing Data to flash memory

Thomas — Tue, 17 Dec 2019 14:50:39 GMT

Cheers Edvin I'll look into those. Also would you therefore, have a recommendation about how to transmit data from the flash storage to my mobile application. As in what method should I use to do it?

RE: Storing Data to flash memory

Edvin — Tue, 17 Dec 2019 14:48:38 GMT

If by "bluetooth" you mean "bluetooth classic", then no, the nRF52840 doesn't support it. But it does support Bluetooth Low Energy. (I guess you can call this a GATT Connection).

For the different fields, such as Record Key and File ID, see this page.

Also, see the guide in the usage section on infocenter. This explains how to both write records, and to read records. Also, the Functionality section describes how to use record keys and File IDs.

RE: Storing Data to flash memory

Thomas — Tue, 17 Dec 2019 14:29:32 GMT

Hi Edvin,

Just to confirm then the nordic board is not capable of bluetooth connections then, meaning to transmit this data I will need a GATT connection?
The data I want to store will be a string e.g 1,1100,1,1100,1,1000 something like that, this is my sensor data

Also then how do I print a single data log entry.

I presume something like this:

read file_id key_id. And it should return that data.

Edit:

From looking into it further I think the rec.id self increments and the file_id is passed into it.

How do i therefore, read it.

Also I need to merge this functionality of adding data to the flash storage with my project within the ble_peripheral folder. This however, uses UART with differently defined pins. I want the data to be added when new data comes in from this UART device not via the terminal.

Thanks,

Thomas

RE: Storing Data to flash memory

Edvin — Tue, 17 Dec 2019 14:10:16 GMT

Hello,

The Nordic Board is capable of Bluetooth Low Energy connections, just to answer your last question first.

So you want to store data that is sampled every second. While this is possible, as you say, it will require 86400 entries every day. How large are these samples?

As I see it, you have two options:

Either you can use the fstorage module, or you can use the FDS module (which uses fstorage module).

The difference between the two is that fstorage is more "down to metal", where you need to decide to what address you want to write the data. Remember that you need to delete an entire flash page before you can write to the same address once more.

FDS will handle this for you, and always write to a free address. It operates with records that you can write, update and delete.

Try to read a bit about each of them. The advantage is that this will make sure of an even flash wear distribution, which will make your flash last longer. The disadvantage is that it has more overhead per record, so you may need to consider to gather up data for e.g. one minute, and save it all in one record.

Best regards,

Edvin