Storing Data to flash memory

[Thomas]

Hi

I am using the Nordic NRF52840_dk with SDK 16.

I want to store data to flash memory with Ble and uart enabled. I am using a external UART peripheral to receive data which, I then parse and broadcast via a ble advertisement.

However, I wish to store this data locally along with a start time for the logging. The data is received once per second. I want this stored inside a log which, I can later send to a mobile application.

I therefore, presume I need to use the flashwrite example. And my code is based on RTT.

I have a value stored as a string that comes in from a UART. I need to store this string in flash memory. I am presuming having a String array is a poor way to do it. Given that I have a requirement for logging for a day meaning 86400 entries of data.

What I want to happen is a command to be sent from the android app to start a log. And then a command to stop it.

Also for transmitting the data that is stored on Nordic board(when working). I am not sure if the nordic board is capable of making a bluetooth connection and, if not will making a Gatt connection.

[Edvin]

Hello,

The Nordic Board is capable of Bluetooth Low Energy connections, just to answer your last question first.

So you want to store data that is sampled every second. While this is possible, as you say, it will require 86400 entries every day. How large are these samples?

As I see it, you have two options:

Either you can use the fstorage module, or you can use the FDS module (which uses fstorage module).

The difference between the two is that fstorage is more "down to metal",  where you need to decide to what address you want to write the data. Remember that you need to delete an entire flash page before you can write to the same address once more.

FDS will handle this for you, and always write to a free address. It operates with records that you can write, update and delete.

Try to read a bit about each of them. The advantage is that this will make sure of an even flash wear distribution, which will make your flash last longer. The disadvantage is that it has more overhead per record, so you may need to consider to gather up data for e.g. one minute, and save it all in one record.

Best regards,

Edvin

[Thomas]

Hi Edvin,

• Just to confirm then the nordic board is not capable of bluetooth connections then, meaning to transmit this data I will need a GATT connection?
• The data I want to store will be a string e.g 1,1100,1,1100,1,1000 something like that, this is my sensor data

Also then how do I print a single data log entry.

I presume something like this:

read file_id key_id. And it should return that data.

Edit:

From looking into it further I think the rec.id self increments and the file_id is passed into it.

How do i therefore, read it.

Also I need to merge this functionality of adding data to the flash storage with my project within the ble_peripheral folder. This however, uses UART with differently defined pins. I want the data to be added when new data comes in from this UART device not via the terminal.

Thanks,

Thomas

[Thomas]

Thanks for the Edvin,

I've written that in now.

But I cant test it because my flash is full.

00> <info> app: Initializing fds...
00> <info> app: Event: FDS_EVT_INIT received (NRF_SUCCESS)
00> <info> app: Found 4 valid records.
00> <info> app: Found 100 dirty records (ready to be garbage collected).
00> <info> app: Writing config file...
00> <info> app: No space in flash, delete some records to update the config file.
00> <info> app: Event: FDS_EVT_DEL_FILE received (NRF_SUCCESS)
00> <info> app: No space in flash, delete some records to update the config file.

Therefore, I need to delete the data. How do I delete all of the data?

I've tried fds_file_delete(&desc); which does not work.

Thanks,

Thomas

[Edvin]

Sorry, I forgot to comment your last post.

Well. The easiest would probably to erase the entire flash if you end up with a lot of records. The easiest way is to use Nordic Command Line Tools, and the command "nrfjprog -e". This will delete everything on the chip.

However, You can see from your log that you have 100 dirty records, and only 4 valid records. This means that you can run a garbage collection. 

Whenever you either update a record or delete a record, it is marked for deletion. This is what we call a dirty record. The reason it isn't deleted and freeing up the space immediately is because of the nature of the flash. You need to erase large chunks at once,  and you can only write one time to a specific location once between deleting it again (in fact, you can only write 1's to 0's in a flash, before erasing it, which will reset it to 1's. This is why an empty flash is always 0xFF's).

So, when either fds_record_write or fds_record_update returns NRF_ERROR_NO_MEM, you need to run a garbage collection by calling fds_gc(). This is a complex call, moving the records around to free space (one flash page is always reserved for this in FDS), and deleting the old pages, so that it has room for new records. Remember to wait for the callback for this call before you try to read or write something else.

The event that you should add to your fds_event handler is called FDS_EVT_GC. Check that it returned with p_evt->result == NRF_SUCCESS before setting a flag saying it is ok to continue.

Also, don't fall for the temptation of running fds_gc() on every startup. This will beat up the flash. The flash on the nRFs are guaranteeing 10000 flash erase/write cycles. If you run fds_gc() on every startup, this will be limited to 10000 resets.

The flash will not stop working after 10 001 cycles, but this is what we can say for sure that will work.

It is not unusual that it will work for 40-50 000 cycles, or even 100 000 cycles, but still, there is no need to beat up the flash by running an fds_gc() before it is necessary.

[Thomas]

Ok thanks I won't call fds_gs too often,.

I still have the issue of my code not reading from the correct location.

I have setup two data configurations. One is the dummy it is unchanged.

The other is custom see below:

/* Sensor configuration data. */
static configuration_t_sensor m_sensor_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "sensor",
    .count_id    = 0,
    .t1_data     = 0,
    .t2_data     = 0,
    .t3_data     = 0,
    .p1_data     = 0,
    .p2_data     = 0,
};

/* A record containing dummy configuration data. */
static fds_record_t const m_sensor_record =
{
    .file_id           = CONFIG_FILE,
    .key               = CONFIG_REC_KEY,
    .data.p_data       = &m_sensor_cfg,
    /* The length of a record is always expressed in 4-byte units (words). */
    .data.length_words = (sizeof(m_sensor_cfg) + 3) / sizeof(uint32_t),
};

When I print my data to the RTT view it always reads the dummy data.

 NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);

I am going to assume this is a configuration / setup issue.

Also when running the code and looping a function to add. I fill up the flash very fast.

0> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x0087
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010
00> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x0088
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010
00> <info> app: Received string: "1,1191,1,1172
00> "
00> <info> app: T1 is:"1191"
00> <info> app: T2 is:"1172"
00> <info> app: Reading data dummy.
00> <info> app: No space in flash, delete some records to update the config file.
00> <info> app: Received string: "1,1191,1,1172
00> "

Note the t1 and t2 data are coming in from a UART this is the data I wish to store.

[Thomas]

I think it might be because I am always reading the same data? Which, is set by the code before mine. The dummy code. Therefore for reading the data how do I point it to the latest data inside of the flash storage?

Also are multiple configurations allowed in the fds_example.h?

[Edvin]

I don't know what to read out of your snippets. Is it the lines in the log saying "reading data dummy" that you refer to?

Beause the only place I can see that you print that is from the line:

NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);//Print out the configurations bootcount member variable

What do you expect it to say? Where does "dummy" come from, and where do you update it?

[Thomas]

That is the line of code that should pull the data. I want that line of code to output the latest data I have entered into my flash storage.

The line of code you highlighted always prints "Reading data dummy" even though the data I am pointing at in the line of code is the m_sensor_cfg. This has a device name set to "Sensor" not "dummy" thus my confusion.

I am writing a new entry via my flash_storage_write_data function. This is the data I wish to read (latest).

[Edvin]

What does p_data look like in the flash_storage_read_data() function right before you copy it over to the m_sensor_cfg?

Try to set a breakpoint and watch that data.

Did you wait for the write complete event before you read it?

[Thomas]

When I place a breakpoint on  memcpy(&m_sensor_cfg, config.p_data, sizeof(configuration_t));

I run over it and, it goes to NRF_BREAKPOINT_COND.

So I am not sure how to test it.

I presume I am writing the information incorrectly and reading it incorrectly.

Given the aforementioned line of code should print out "sensor" as this is what I believe I write to the configuration file.

However, I don't want to read the configuration file. I want to read the latest data.

But, I can't find an example of where to do that in the sdk example code. Or the documentation.

[Edvin]

If you set a breakpoint and then resume,  the softdevice will assert. That is fine, you just need to reset.

Just set a breakpoint at the line memcpy, and right click config.p_data and click "add to watch". What this button is called depends on your IDE.

If you are not able to see the content of the pointer, then you need to disable optimization in your project.If you are not sure how to do that, let me know what IDE you are using.

[Thomas]

Im using segger.

On the line:

  rc = fds_record_open(&desc, &config);//Read data

How do I read the last entry. I presume the config that is in the flash is the default dummy one.

So my question is what is desc? Is this the specfic instance that is read?

Because I want to call a function that I can read the last record from. I will later need to read all the data. But I am thinking of doing this in a for loop from the first entry to the last.