Storing Data to flash memory

[Thomas]

Hi

I am using the Nordic NRF52840_dk with SDK 16.

I want to store data to flash memory with Ble and uart enabled. I am using a external UART peripheral to receive data which, I then parse and broadcast via a ble advertisement.

However, I wish to store this data locally along with a start time for the logging. The data is received once per second. I want this stored inside a log which, I can later send to a mobile application.

I therefore, presume I need to use the flashwrite example. And my code is based on RTT.

I have a value stored as a string that comes in from a UART. I need to store this string in flash memory. I am presuming having a String array is a poor way to do it. Given that I have a requirement for logging for a day meaning 86400 entries of data.

What I want to happen is a command to be sent from the android app to start a log. And then a command to stop it.

Also for transmitting the data that is stored on Nordic board(when working). I am not sure if the nordic board is capable of making a bluetooth connection and, if not will making a Gatt connection.

[Edvin]

Hello,

The Nordic Board is capable of Bluetooth Low Energy connections, just to answer your last question first.

So you want to store data that is sampled every second. While this is possible, as you say, it will require 86400 entries every day. How large are these samples?

As I see it, you have two options:

Either you can use the fstorage module, or you can use the FDS module (which uses fstorage module).

The difference between the two is that fstorage is more "down to metal",  where you need to decide to what address you want to write the data. Remember that you need to delete an entire flash page before you can write to the same address once more.

FDS will handle this for you, and always write to a free address. It operates with records that you can write, update and delete.

Try to read a bit about each of them. The advantage is that this will make sure of an even flash wear distribution, which will make your flash last longer. The disadvantage is that it has more overhead per record, so you may need to consider to gather up data for e.g. one minute, and save it all in one record.

Best regards,

Edvin

[Thomas]

Hi Edvin,

• Just to confirm then the nordic board is not capable of bluetooth connections then, meaning to transmit this data I will need a GATT connection?
• The data I want to store will be a string e.g 1,1100,1,1100,1,1000 something like that, this is my sensor data

Also then how do I print a single data log entry.

I presume something like this:

read file_id key_id. And it should return that data.

Edit:

From looking into it further I think the rec.id self increments and the file_id is passed into it.

How do i therefore, read it.

Also I need to merge this functionality of adding data to the flash storage with my project within the ble_peripheral folder. This however, uses UART with differently defined pins. I want the data to be added when new data comes in from this UART device not via the terminal.

Thanks,

Thomas

[Thomas]

Cheers Edvin I'll look into those. Also would you therefore, have a recommendation about how to transmit data from the flash storage to my mobile application. As in what method should I use to do it?

[Edvin]

You need to use a BLE service + characteristic to transfer this data. The characteristic you need to use probably look a lot like the service that is implemented in the ble_app_uart example. This uses something called notifications to transfer data. In that example, it uses ble_nus_data_send() to send a text string, (or any other array of bytes). 

If you look into this example,  which you will find in SDK\examples\ble_peripheral\ble_app_uart, you can see that it usually calls ble_nus_data_send() inside the uart_event_handle(),  but you can send it at any other place in the code as well.

Also note that you can queue several packets using ble_nus_data_send(), until it returns NRF_ERROR_RESOURCES. Once that is returned, you must wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event (which you can add to ble_evt_handler() in main.c),  that means that one packet is ACKed by the connected device, and there is some space free in the softdevice queue, and you can start queuing new packets.

Best regards,

Edvin

[Thomas]

Hi Edvin, I'll look into that I have had a lot of trouble with the UART example though from a previous issue. https://devzone.nordicsemi.com/f/nordic-q-a/54979/receiving-data-from-uart-via-a-peripheral-device

For more details.

I am trying to merge the flash_fds code and my ble_app_beacon code. Presently I have an undefined reference to fds_register and many other error see below.

  Linking ble_app_beacon_pca10056_s140.elf
    Output/ble_app_beacon_pca10056_s140 Release/Obj/cli.o:D:\Segger projects\Segger Projects\DeviceDownload2\examples\ble_peripheral\ble_app_beacon/cli.c:87: undefined reference to `nrf_cli_uart_transport_api'
Build failed

However, I have included in main:

#ifdef SOFTDEVICE_PRESENT
#include "nrf_sdh.h"
#include "nrf_sdh_ble.h"
#else
#include "nrf_drv_clock.h"
#endif
#include "fds.h"
#include "app_timer.h"
#include "app_error.h"
#include "nrf_cli.h"
#include "fds_example.h"

#define NRF_LOG_MODULE_NAME app
#include "nrf_log.h"
#include "nrf_log_ctrl.h"


/* A tag identifying the SoftDevice BLE configuration. */
#define APP_BLE_CONN_CFG_TAG    1

Also in my processor includes I have added:

../../../../../../components/libraries/cli/uart
../../../../../../examples/peripheral/flash_fds
../../../../../components/libraries/uart
../../../../../../components/libraries/fifo
../../../../../../components/libraries/queue

I have also added a bunch of the c files required.

I have changed these lines of code in the sdk to match the flash storage sdk config

CRC16_ENABLED 1//was 0
FS_ENABLED 1//was 0
FDS_CRC_CHECK_ON_READ 1//was 0
NRF_FSTORAGE_ENABLED 1//was 0
NRF_CLI_ENABLED 1//was 0
NRF_QUEUE_ENABLED 1 // Was 0
NRF_QUEUE_CLI_CMDS 1// Was 0

Presently I have undefined references to nrf_queue_generic_pop and some more see above.

I am not sure how to solve the nrf_cli_uart_transport api being undefined.

If within nrf_cli_uart.c I uncomment the line if NRF_module_enabled(NRF_CLI_UART) it builds. But that therefore, means that the nrf_module is not enabled. All the code after that line is greyed out because it is not defined.

From looking at this : undefined reference I ensured NRF_CLI_ENABLED 1 but, it already was.

[Edvin]

If I were you, I'd try to stay clear of the CLI library, unless you intentionally are trying to use this. There is a uart library that e.g. the ble_app_uart example uses, that doesn't use the CLI. CLI tend to make things a bit complicated.

However, if you need the CLI, then it is possible. Your compiler doesn't find the function nrf_cli_uart_transport_api(). You are probably missing the .c file that has this function. Then you also need the define:

NRF_CLI_UART_ENABLED  to be defined to 1 in sdk_config.h.

But try to use the uart library that is used in the ble_app_uart example.

[Thomas]

Cheers I got it to build and, it crashes with fatal error. I don't need the cli but need to be able to write data to the flash which, in the fds_storage application i was using the command line to test.

My question therefore, is how do I store the data?

Note if I uncomment the cli_init it does not crash. And if I had it in and changed my pin definitions from my UART peripheral to the default the behaviour is the same as the fds_flash storage example.

Hope I explained this well.

I presume I do something like this:

static configuration_t m_dummy_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "dummy",
};

/* A record containing dummy configuration data. */
static fds_record_t const m_dummy_record =
{
    .file_id           = CONFIG_FILE,
    .key               = CONFIG_REC_KEY,
    .data.p_data       = &m_dummy_cfg,
    /* The length of a record is always expressed in 4-byte units (words). */
    .data.length_words = (sizeof(m_dummy_cfg) + 3) / sizeof(uint32_t),
};
//In main loop somewhere

rc = fds_record_write(&desc, &m_dummy_record);
//I want to set this as a string and as a new record

But can I pass in a string?

I want to set this as a string and as a new record.

If not is it best to store the data as a struct or something of the like?

Also how do I read this data as in the actual data inside the flash storage. Using the boot count value works but I would like to see all of the data stored.

Also each time I reboot the dirty records increase by 1.

[Edvin]

You can store data to flash without the CLI library. What you can do is to look at what the different CLI commands. Unfortunately, it is a bit tricky to trace what these commands to, which is why I find it a bit unfortunately that the FDS examples use this.

Basically, you can use UART to send a string of data to the nRF, then use the fds_record_write() or fds_record_update() to save this data.

If you want to dig into why it crashes with "fatal error", this means that one of your APP_ERROR_CHECK(err_code) receives an err_code != 0. If you want to find out which one, I suggest that you define DEBUG in your preprocessor defines (let me know if you are not sure how to do that, and what SDK version you use, and what IDE you use).

Remember that if you write or update a record using fds_record_write/update(), then you need to wait for the FDS callback before you can read it. This is typically not a problem in the CLI case, since the time you use to type the different commands is very long compared to the time it takes to save the data to flash,  but if you do it sequentially in an application, make sure to wait for the callback before you try to read the flash record after it is updated.

Best regards,

Edvin 

[Thomas]

Cheers Edvin, I've decided not to use CLI.

I want to be able to store data and then read it.

At the moment I have some code that adds an entry to the flash storage and then reads it.

This is called after the dummy entry has been added.

flash_storage_read_data();
flash_storage_write_data(00,00,00,00,00,00,00,00,00);//Write the information to the flash storage
flash_storage_read_data();

These functions are written as below

static void flash_storage_write_data(int count, int T1_value,int T2_value,
                                       int t3_value_1,int t3_value_2, int p1_value_1, int p1_value_2,
                                       int p2_value_1,int p2_value_2){
    ret_code_t rc;
    fds_record_desc_t desc = {0};
    //Need to modify the sensor record here
    rc = fds_record_write(&desc, &m_sensor_record);//Write data to the sensor record
        if ((rc != NRF_SUCCESS) && (rc == FDS_ERR_NO_SPACE_IN_FLASH))
        {
            NRF_LOG_INFO("No space in flash, delete some records to update the config file.");
        }
        else
        {
            APP_ERROR_CHECK(rc);
            NRF_LOG_INFO("Data added successfully");
        }
}

static void flash_storage_read_data()//Function to read the data from the flash storage
{
    ret_code_t rc;
    fds_record_desc_t desc = {0};
    fds_find_token_t  tok  = {0};

    rc = fds_record_find(CONFIG_FILE, CONFIG_REC_KEY, &desc, &tok);

    if (rc == NRF_SUCCESS)
    {
        /* A config file is in flash. Let's update it. */
        fds_flash_record_t config = {0};

        /* Open the record and read its contents. */
        rc = fds_record_open(&desc, &config);//NEed to ensure it opens sensor config
        APP_ERROR_CHECK(rc);

        /* Copy the configuration from flash into m_sensor_cfg. */
        memcpy(&m_sensor_cfg, config.p_data, sizeof(configuration_t_sensor));

        NRF_LOG_INFO("Reading data %s.", m_sensor_cfg.device_name);//Print out the configurations bootcount member variable

        /* Update boot count. */
        m_dummy_cfg.boot_count++;

        /* Close the record when done reading. */
        rc = fds_record_close(&desc);
        APP_ERROR_CHECK(rc);
     }
}

I have defined my own configuration in the fds_example.h to contain sensor data information:

//Sensor Config data
typedef struct
{
    uint32_t boot_count;
    char     device_name[16];
    bool     config1_on;
    bool     config2_on;
    
    int count_id;
    int t1_data;
    int t2_data;
    int t3_data;
    int p1_data;
    int p2_data;
}configuration_t_sensor;

However, when I run my code I receive: 

00> <info> app: UART example started.
00> <info> app: Initializing fds...
00> <info> app: Event: FDS_EVT_INIT received (NRF_SUCCESS)
00> <info> app: Found 54 valid records.
00> <info> app: Found 18 dirty records (ready to be garbage collected).
00> <info> app: Config file found, updating boot count to 8.
00> <info> app: Reading data dummy.
00> <info> app: Data added successfully
00> <info> app: Reading data dummy.
00> <info> app: Event: FDS_EVT_UPDATE received (NRF_SUCCESS)
00> <info> app: Event: FDS_EVT_WRITE received (NRF_SUCCESS)
00> <info> app: Record ID:  0x004C
00> <info> app: File ID:  0x8010
00> <info> app: Record key:  0x7010

Even though I have configured my sensor data  configuration to be:

static configuration_t_sensor m_sensor_cfg =
{
    .config1_on  = false,
    .config2_on  = true,
    .boot_count  = 0x0,
    .device_name = "Sensor",
    .count_id    = 0,//Int to to be passed in
    .t1_data     = 0,//Int to to be passed in
    .t2_data     = 0,//Int to to be passed in
    .t3_data     = 0,//Int to to be passed in
    .p1_data     = 0,//Int to to be passed in
    .p2_data     = 0,//Int to to be passed in
};

I am not sure why this is. Could it have something to do with the flash storage taking a long time as you stated? If so is there a statement that says something like wait till fds ready. However, surely it should read the configuration file I have selected.

Cheers,

Thomas

[Thomas]

Also when I run the write code in a loop i quickly ran out of space.

It comes up with :No space in flash, delete some records to update the config file.

How do I delete the entrants? ( All of them)

I presume it has something to do with fds_record_delete within the fds.c code.

Thanks,

Thomas

[Edvin]

I didn't understand what exactly you are trying to tell me, but I guess you are not reading the same as you are writing?

 

Thomas said:

If so is there a statement that says something like wait till fds ready.

 Yes.

Do you have an event handler for your FDS instance? You probably have a call like this somewhere:

fds_register(fds_evt_handler);

If so, fds_evt_handler is your fds callback function.

What you need to do is to wait after you have changed any of the flash content. You can do so by using a flag:

//Top of main.c

volatile bool fds_write_complete_flag = false;


static void fds_evt_handler(fds_evt_t const * p_evt)
{
    if (p_evt->result == NRF_SUCCESS)
    {
        NRF_LOG_GREEN("Event: %s received (NRF_SUCCESS)",
                      fds_evt_str[p_evt->id]);
    }
    else
    {
        NRF_LOG_GREEN("Event: %s received (%s)",
                      fds_evt_str[p_evt->id],
                      fds_err_str(p_evt->result));
    }

    switch (p_evt->id)
    {
        case FDS_EVT_INIT:
            if (p_evt->result == NRF_SUCCESS)
            {
                m_fds_initialized = true;
            }
            break;

        case FDS_EVT_WRITE:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->write.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->write.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->write.record_key);
                fds_write_complete_flag = true;
            }
        } break;

        case FDS_EVT_DEL_RECORD:
        {
            if (p_evt->result == NRF_SUCCESS)
            {
                NRF_LOG_INFO("Record ID:\t0x%04x",  p_evt->del.record_id);
                NRF_LOG_INFO("File ID:\t0x%04x",    p_evt->del.file_id);
                NRF_LOG_INFO("Record key:\t0x%04x", p_evt->del.record_key);
            }
            m_delete_all.pending = false;
        } break;

        default:
            break;
    }
}

int main(void)
{
    (void) fds_register(fds_evt_handler);
    
    flash_storage_read_data();
    flash_storage_write_data(00,00,00,00,00,00,00,00,00);//Write the information to the flash storage
    while (fds_write_complete_flag == false)
    {
        // Wait...
    }
    fds_write_complete_flag = false;
    flash_storage_read_data();
    
    

See how I use the FDS_EVT_WRITE event to set the fds_write_complete_flag to true, and how I wait in the main() function for this event to set the flag before I read it.

BR,

Edvin

[Thomas]

Thanks for the Edvin,

I've written that in now.

But I cant test it because my flash is full.

00> <info> app: Initializing fds...
00> <info> app: Event: FDS_EVT_INIT received (NRF_SUCCESS)
00> <info> app: Found 4 valid records.
00> <info> app: Found 100 dirty records (ready to be garbage collected).
00> <info> app: Writing config file...
00> <info> app: No space in flash, delete some records to update the config file.
00> <info> app: Event: FDS_EVT_DEL_FILE received (NRF_SUCCESS)
00> <info> app: No space in flash, delete some records to update the config file.

Therefore, I need to delete the data. How do I delete all of the data?

I've tried fds_file_delete(&desc); which does not work.

Thanks,

Thomas