Flash Storage nRF52832

[connorshannon]

Hello,

I'm using an nRF52832 and an accelerometer/gyro to record data and send it over Bluetooth. My end goal is to store this data in large chunks, and then send it all at once (as opposed to collecting data and immediately sending it over Bluetooth). I'm not quite sure what the best way to do this is. From what I can tell, FDS is something that might work for what I want, but I can't find sample code that implements it. 

I suppose my questions are these:

1. Is FDS likely the best method of storing data, retrieving it, and writing over it again for this chip?

2. If so, where would be a good place to start as far as sample code or documentation for this? Everything I find seems to be outdated or uses a different method to store data in memory.

Thank you!

[tesc]

Hi,

1. Is FDS likely the best method of storing data, retrieving it, and writing over it again for this chip?

It depends a bit on the use case, but on nRF5 SDK the Flash Data Storage (FDS) is clearly a good alternative for many applications. The other main alternative on nRF5 SDK is more direct flash usage, through Flash Storage (fstorage).

2. If so, where would be a good place to start as far as sample code or documentation for this? Everything I find seems to be outdated or uses a different method to store data in memory.

nRF5 SDK itself is in maintenance mode, and has been for quite some time. That might be the reason why most resources on FDS looks old. See nRF Connect SDK and nRF5 SDK statement for details on our SDKs and when to use which SDK. In nRF Connect SDK, there are other options for storing data in non-volatile memory.

If you have based your project off of nRF5 SDK, then you will find examples both for fstorage and for FDS.

If you use nRF Connect SDK, then you have several options. I recommend having a look at the persistent storage of keys and data using the nRF Connect SDK blogpost.

Regards,
Terje

[connorshannon]

I am using the nRF connect sdk, and found some sample code that I'll attach here.

/*
 * NVS Sample for Zephyr using high level API, the sample illustrates the usage
 * of NVS for storing data of different kind (strings, binary blobs, unsigned
 * 32 bit integer) and also how to read them back from flash. The reading of
 * data is illustrated for both a basic read (latest added value) as well as
 * reading back the history of data (previously added values). Next to reading
 * and writing data it also shows how data can be deleted from flash.
 *
 * The sample stores the following items:
 * 1. A string representing an IP-address: stored at id=1, data="192.168.1.1"
 * 2. A binary blob representing a key: stored at id=2, data=FF FE FD FC FB FA
 *    F9 F8
 * 3. A reboot counter (32bit): stored at id=3, data=reboot_counter
 * 4. A string: stored at id=4, data="DATA" (used to illustrate deletion of
 * items)
 *
 * At first boot the sample checks if the data is available in flash and adds
 * the items if they are not in flash.
 *
 * Every reboot increases the values of the reboot_counter and updates it in
 * flash.
 *
 * At the 10th reboot the string item with id=4 is deleted (or marked for
 * deletion).
 *
 * At the 11th reboot the string item with id=4 can no longer be read with the
 * basic nvs_read() function as it has been deleted. It is possible to read the
 * value with nvs_read_hist()
 *
 * At the 78th reboot the first sector is full and a new sector is taken into
 * use. The data with id=1, id=2 and id=3 is copied to the new sector. As a
 * result of this the history of the reboot_counter will be removed but the
 * latest values of address, key and reboot_counter is kept.
 *
 * Copyright (c) 2018 Laczen
 *
 * SPDX-License-Identifier: Apache-2.0
 */


#include <zephyr/kernel.h>
#include <zephyr/sys/reboot.h>
#include <zephyr/device.h>
#include <string.h>
#include <zephyr/drivers/flash.h>
#include <zephyr/storage/flash_map.h>
#include <zephyr/fs/nvs.h>

static struct nvs_fs fs;

#define NVS_PARTITION		storage_partition
#define NVS_PARTITION_DEVICE	FIXED_PARTITION_DEVICE(NVS_PARTITION)
#define NVS_PARTITION_OFFSET	FIXED_PARTITION_OFFSET(NVS_PARTITION)

/* 1000 msec = 1 sec */
#define SLEEP_TIME      100
/* maximum reboot counts, make high enough to trigger sector change (buffer */
/* rotation). */
#define MAX_REBOOT 400

#define ADDRESS_ID 1
#define KEY_ID 2
#define RBT_CNT_ID 3
#define STRING_ID 4
#define LONG_ID 5


int main(void)
{
	int rc = 0, cnt = 0, cnt_his = 0;
	char buf[16];
	uint8_t key[8], longarray[128];
	uint32_t reboot_counter = 0U, reboot_counter_his;
	struct flash_pages_info info;

	/* define the nvs file system by settings with:
	 *	sector_size equal to the pagesize,
	 *	3 sectors
	 *	starting at NVS_PARTITION_OFFSET
	 */
	fs.flash_device = NVS_PARTITION_DEVICE;
	if (!device_is_ready(fs.flash_device)) {
		printk("Flash device %s is not ready\n", fs.flash_device->name);
		return 0;
	}
	fs.offset = NVS_PARTITION_OFFSET;
	rc = flash_get_page_info_by_offs(fs.flash_device, fs.offset, &info);
	if (rc) {
		printk("Unable to get page info\n");
		return 0;
	}
	fs.sector_size = info.size;
	fs.sector_count = 3U;

	rc = nvs_mount(&fs);
	if (rc) {
		printk("Flash Init failed\n");
		return 0;
	}

	/* ADDRESS_ID is used to store an address, lets see if we can
	 * read it from flash, since we don't know the size read the
	 * maximum possible
	 */
	rc = nvs_read(&fs, ADDRESS_ID, &buf, sizeof(buf));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Address: %s\n", ADDRESS_ID, buf);
	} else   {/* item was not found, add it */
		strcpy(buf, "192.168.1.1");
		printk("No address found, adding %s at id %d\n", buf,
		       ADDRESS_ID);
		(void)nvs_write(&fs, ADDRESS_ID, &buf, strlen(buf)+1);
	}
	/* KEY_ID is used to store a key, lets see if we can read it from flash
	 */
	rc = nvs_read(&fs, KEY_ID, &key, sizeof(key));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Key: ", KEY_ID);
		for (int n = 0; n < 8; n++) {
			printk("%x ", key[n]);
		}
		printk("\n");
	} else   {/* item was not found, add it */
		printk("No key found, adding it at id %d\n", KEY_ID);
		key[0] = 0xFF;
		key[1] = 0xFE;
		key[2] = 0xFD;
		key[3] = 0xFC;
		key[4] = 0xFB;
		key[5] = 0xFA;
		key[6] = 0xF9;
		key[7] = 0xF8;
		(void)nvs_write(&fs, KEY_ID, &key, sizeof(key));
	}
	/* RBT_CNT_ID is used to store the reboot counter, lets see
	 * if we can read it from flash
	 */
	rc = nvs_read(&fs, RBT_CNT_ID, &reboot_counter, sizeof(reboot_counter));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Reboot_counter: %d\n",
			RBT_CNT_ID, reboot_counter);
	} else   {/* item was not found, add it */
		printk("No Reboot counter found, adding it at id %d\n",
		       RBT_CNT_ID);
		(void)nvs_write(&fs, RBT_CNT_ID, &reboot_counter,
			  sizeof(reboot_counter));
	}
	/* STRING_ID is used to store data that will be deleted,lets see
	 * if we can read it from flash, since we don't know the size read the
	 * maximum possible
	 */
	rc = nvs_read(&fs, STRING_ID, &buf, sizeof(buf));
	if (rc > 0) {
		/* item was found, show it */
		printk("Id: %d, Data: %s\n",
			STRING_ID, buf);
		/* remove the item if reboot_counter = 10 */
		if (reboot_counter == 10U) {
			(void)nvs_delete(&fs, STRING_ID);
		}
	} else   {
		/* entry was not found, add it if reboot_counter = 0*/
		if (reboot_counter == 0U) {
			printk("Id: %d not found, adding it\n",
			STRING_ID);
			strcpy(buf, "DATA");
			(void)nvs_write(&fs, STRING_ID, &buf, strlen(buf) + 1);
		}
	}

	/* LONG_ID is used to store a larger dataset ,lets see if we can read
	 * it from flash
	 */
	rc = nvs_read(&fs, LONG_ID, &longarray, sizeof(longarray));
	if (rc > 0) {
		/* item was found, show it */
		printk("Id: %d, Longarray: ", LONG_ID);
		for (int n = 0; n < sizeof(longarray); n++) {
			printk("%x ", longarray[n]);
		}
		printk("\n");
	} else   {
		/* entry was not found, add it if reboot_counter = 0*/
		if (reboot_counter == 0U) {
			printk("Longarray not found, adding it as id %d\n",
			       LONG_ID);
			for (int n = 0; n < sizeof(longarray); n++) {
				longarray[n] = n;
			}
			(void)nvs_write(
				&fs, LONG_ID, &longarray, sizeof(longarray));
		}
	}

	cnt = 5;
	while (1) {
		k_msleep(SLEEP_TIME);
		if (reboot_counter < MAX_REBOOT) {
			if (cnt == 5) {
				/* print some history information about
				 * the reboot counter
				 * Check the counter history in flash
				 */
				printk("Reboot counter history: ");
				while (1) {
					rc = nvs_read_hist(
						&fs, RBT_CNT_ID,
						&reboot_counter_his,
						sizeof(reboot_counter_his),
						cnt_his);
					if (rc < 0) {
						break;
					}
					printk("...%d", reboot_counter_his);
					cnt_his++;
				}
				if (cnt_his == 0) {
					printk("\n Error, no Reboot counter");
				} else {
					printk("\nOldest reboot counter: %d",
					       reboot_counter_his);
				}
				printk("\nRebooting in ");
			}
			printk("...%d", cnt);
			cnt--;
			if (cnt == 0) {
				printk("\n");
				reboot_counter++;
				(void)nvs_write(
					&fs, RBT_CNT_ID, &reboot_counter,
					sizeof(reboot_counter));
				if (reboot_counter == MAX_REBOOT) {
					printk("Doing last reboot...\n");
				}
				sys_reboot(0);
			}
		} else {
			printk("Reboot counter reached max value.\n");
			printk("Reset to 0 and exit test.\n");
			reboot_counter = 0U;
			(void)nvs_write(&fs, RBT_CNT_ID, &reboot_counter,
			  sizeof(reboot_counter));
			break;
		}
	}
	return 0;
}

I'm not entirely familiar with the nvs code or procedures, so if anyone had any tips I would greatly appreciate it. I'm just trying to store large chunks of data (either int or float values), read from the memory/send over bluetooth, and then delete what I just send from the memory and rewrite the memory location with new data.

Thanks!

[tesc]

Hi,

Documentation wise, there are the documentation for Non-Volatile Storage (NVS), the Non-volatile Storage APIs, and the Non-Volatile Storage (NVS) sample which looks like the code you have found and shared here.

For your particular use case, you might want to look into Flash Circular Buffer (FCB) instead of NVS. While there are less resources available for FCB (there does not seem to be any samples for it in the SDK), it does sound like a closer match to what you are looking for. See also the FCB API documentation.

I assume that you cannot simply store your values in an array in RAM, due to either size constraints, a need to store the data past device reset, or for other reasons.

Please note that flash has a limited number of write/erase cycles, with 10 000 cycles guaranteed for the nRF52833. This means that each flash block, 4 kB of size, can be erased a total ten thousand times before you might start to see glitches on some devices. For this reason, RAM will often be a better alternative for buffering data, provided that the RAM can be kept alive. When using Flash, it is important to set aside enough flash pages relative to the write frequency, that you will not exhaust the write/erase cycles during the expected lifetime of the device.

Regards,
Terje

[connorshannon]

Thank you, that helps. I'm planning on storing ~60 double values per second for some amount of time (may be dependent on storage space limitations). I had heard that flash storage would work better for this application than RAM, but if you think RAM might be better please let me know. I don't need to store data past device reset. I basically just need to store data for some amount of time (the longer the better), and then take a break in reading data and send everything that was stored over Bluetooth.

[tesc]

Hi,

At that cadence you would fill one kB roughly every two seconds, or spend 8-10 seconds filling a Flash page (4 kB on the nRF52 series.) Headers (for FDS) would come in addition. I would definitely go for a fixed buffer or circular buffer approach, and not use a key-value or similar data storage.

Flash has mainly two big advantages over RAM. First, it is persistent, although this is not something you need. Second, it has larger capacity. On the nRF52832, 512 kB of Flash, v.s. 64 kB of RAM. Of course both Flash and RAM have other uses, so the space available for storing measured values is also less. The remaining available storage depends on the application size and memory usage, but it should be easier to find (or save) more space in the larger Flash.

RAM has some advantages over Flash as well. First, notably, unlimited read/write cycles. It will not wear out during the lifetime of the device. Second, speed. In particular very low write time, and no need to do time costly erase operations.

Some quick calculations for the Flash option:

Assuming no overhead, for every flash page used for storage you get roughly 10 (seconds to fill the page) times 10 000 (guaranteed minimum number of flash erase/write cycles) = 100 000 seconds worth of data, before you might start to see glitches in Flash performance. This is somewhat more than one day of continuous data collection (27.7 hours). Accounting for some overhead, and for easier calculation, let's assume one day's worth of data. If, then, you want the lifetime of the product to be at least one year of data collection, then we are already at a Flash storage capacity need of around 1.5 MB (one Flash page of 4 kB per day's worth of lifetime, times 365 days in a full year), or 3x what is available on the nRF52832 altogether.

Now, use of Flash could still be viable, depending on how much of the time the device would be actively recording. For instance, if it records for only one minute every hour, the lifetime would be 13 to 14 months if you use 10 Flash pages (40 kB of Flash). In other words: If the duty cycle of gathering data is 1/60, Flash can be feasible, but if the duty cycle is closer to 100 % then it is most likely unfeasible. The limiting factor there, is the Flash lifetime of 10 000 write/erase cycles.

For the RAM option, I recommend having a look at Ring Buffers. The concepts are pretty well described on that documentation page, along with example code snippets.

Regards,
Terje