Flash Storage nRF52832

[connorshannon]

Hello,

I'm using an nRF52832 and an accelerometer/gyro to record data and send it over Bluetooth. My end goal is to store this data in large chunks, and then send it all at once (as opposed to collecting data and immediately sending it over Bluetooth). I'm not quite sure what the best way to do this is. From what I can tell, FDS is something that might work for what I want, but I can't find sample code that implements it. 

I suppose my questions are these:

1. Is FDS likely the best method of storing data, retrieving it, and writing over it again for this chip?

2. If so, where would be a good place to start as far as sample code or documentation for this? Everything I find seems to be outdated or uses a different method to store data in memory.

Thank you!

[tesc]

Hi,

1. Is FDS likely the best method of storing data, retrieving it, and writing over it again for this chip?

It depends a bit on the use case, but on nRF5 SDK the Flash Data Storage (FDS) is clearly a good alternative for many applications. The other main alternative on nRF5 SDK is more direct flash usage, through Flash Storage (fstorage).

2. If so, where would be a good place to start as far as sample code or documentation for this? Everything I find seems to be outdated or uses a different method to store data in memory.

nRF5 SDK itself is in maintenance mode, and has been for quite some time. That might be the reason why most resources on FDS looks old. See nRF Connect SDK and nRF5 SDK statement for details on our SDKs and when to use which SDK. In nRF Connect SDK, there are other options for storing data in non-volatile memory.

If you have based your project off of nRF5 SDK, then you will find examples both for fstorage and for FDS.

If you use nRF Connect SDK, then you have several options. I recommend having a look at the persistent storage of keys and data using the nRF Connect SDK blogpost.

Regards,
Terje

[connorshannon]

I am using the nRF connect sdk, and found some sample code that I'll attach here.

/*
 * NVS Sample for Zephyr using high level API, the sample illustrates the usage
 * of NVS for storing data of different kind (strings, binary blobs, unsigned
 * 32 bit integer) and also how to read them back from flash. The reading of
 * data is illustrated for both a basic read (latest added value) as well as
 * reading back the history of data (previously added values). Next to reading
 * and writing data it also shows how data can be deleted from flash.
 *
 * The sample stores the following items:
 * 1. A string representing an IP-address: stored at id=1, data="192.168.1.1"
 * 2. A binary blob representing a key: stored at id=2, data=FF FE FD FC FB FA
 *    F9 F8
 * 3. A reboot counter (32bit): stored at id=3, data=reboot_counter
 * 4. A string: stored at id=4, data="DATA" (used to illustrate deletion of
 * items)
 *
 * At first boot the sample checks if the data is available in flash and adds
 * the items if they are not in flash.
 *
 * Every reboot increases the values of the reboot_counter and updates it in
 * flash.
 *
 * At the 10th reboot the string item with id=4 is deleted (or marked for
 * deletion).
 *
 * At the 11th reboot the string item with id=4 can no longer be read with the
 * basic nvs_read() function as it has been deleted. It is possible to read the
 * value with nvs_read_hist()
 *
 * At the 78th reboot the first sector is full and a new sector is taken into
 * use. The data with id=1, id=2 and id=3 is copied to the new sector. As a
 * result of this the history of the reboot_counter will be removed but the
 * latest values of address, key and reboot_counter is kept.
 *
 * Copyright (c) 2018 Laczen
 *
 * SPDX-License-Identifier: Apache-2.0
 */


#include <zephyr/kernel.h>
#include <zephyr/sys/reboot.h>
#include <zephyr/device.h>
#include <string.h>
#include <zephyr/drivers/flash.h>
#include <zephyr/storage/flash_map.h>
#include <zephyr/fs/nvs.h>

static struct nvs_fs fs;

#define NVS_PARTITION		storage_partition
#define NVS_PARTITION_DEVICE	FIXED_PARTITION_DEVICE(NVS_PARTITION)
#define NVS_PARTITION_OFFSET	FIXED_PARTITION_OFFSET(NVS_PARTITION)

/* 1000 msec = 1 sec */
#define SLEEP_TIME      100
/* maximum reboot counts, make high enough to trigger sector change (buffer */
/* rotation). */
#define MAX_REBOOT 400

#define ADDRESS_ID 1
#define KEY_ID 2
#define RBT_CNT_ID 3
#define STRING_ID 4
#define LONG_ID 5


int main(void)
{
	int rc = 0, cnt = 0, cnt_his = 0;
	char buf[16];
	uint8_t key[8], longarray[128];
	uint32_t reboot_counter = 0U, reboot_counter_his;
	struct flash_pages_info info;

	/* define the nvs file system by settings with:
	 *	sector_size equal to the pagesize,
	 *	3 sectors
	 *	starting at NVS_PARTITION_OFFSET
	 */
	fs.flash_device = NVS_PARTITION_DEVICE;
	if (!device_is_ready(fs.flash_device)) {
		printk("Flash device %s is not ready\n", fs.flash_device->name);
		return 0;
	}
	fs.offset = NVS_PARTITION_OFFSET;
	rc = flash_get_page_info_by_offs(fs.flash_device, fs.offset, &info);
	if (rc) {
		printk("Unable to get page info\n");
		return 0;
	}
	fs.sector_size = info.size;
	fs.sector_count = 3U;

	rc = nvs_mount(&fs);
	if (rc) {
		printk("Flash Init failed\n");
		return 0;
	}

	/* ADDRESS_ID is used to store an address, lets see if we can
	 * read it from flash, since we don't know the size read the
	 * maximum possible
	 */
	rc = nvs_read(&fs, ADDRESS_ID, &buf, sizeof(buf));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Address: %s\n", ADDRESS_ID, buf);
	} else   {/* item was not found, add it */
		strcpy(buf, "192.168.1.1");
		printk("No address found, adding %s at id %d\n", buf,
		       ADDRESS_ID);
		(void)nvs_write(&fs, ADDRESS_ID, &buf, strlen(buf)+1);
	}
	/* KEY_ID is used to store a key, lets see if we can read it from flash
	 */
	rc = nvs_read(&fs, KEY_ID, &key, sizeof(key));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Key: ", KEY_ID);
		for (int n = 0; n < 8; n++) {
			printk("%x ", key[n]);
		}
		printk("\n");
	} else   {/* item was not found, add it */
		printk("No key found, adding it at id %d\n", KEY_ID);
		key[0] = 0xFF;
		key[1] = 0xFE;
		key[2] = 0xFD;
		key[3] = 0xFC;
		key[4] = 0xFB;
		key[5] = 0xFA;
		key[6] = 0xF9;
		key[7] = 0xF8;
		(void)nvs_write(&fs, KEY_ID, &key, sizeof(key));
	}
	/* RBT_CNT_ID is used to store the reboot counter, lets see
	 * if we can read it from flash
	 */
	rc = nvs_read(&fs, RBT_CNT_ID, &reboot_counter, sizeof(reboot_counter));
	if (rc > 0) { /* item was found, show it */
		printk("Id: %d, Reboot_counter: %d\n",
			RBT_CNT_ID, reboot_counter);
	} else   {/* item was not found, add it */
		printk("No Reboot counter found, adding it at id %d\n",
		       RBT_CNT_ID);
		(void)nvs_write(&fs, RBT_CNT_ID, &reboot_counter,
			  sizeof(reboot_counter));
	}
	/* STRING_ID is used to store data that will be deleted,lets see
	 * if we can read it from flash, since we don't know the size read the
	 * maximum possible
	 */
	rc = nvs_read(&fs, STRING_ID, &buf, sizeof(buf));
	if (rc > 0) {
		/* item was found, show it */
		printk("Id: %d, Data: %s\n",
			STRING_ID, buf);
		/* remove the item if reboot_counter = 10 */
		if (reboot_counter == 10U) {
			(void)nvs_delete(&fs, STRING_ID);
		}
	} else   {
		/* entry was not found, add it if reboot_counter = 0*/
		if (reboot_counter == 0U) {
			printk("Id: %d not found, adding it\n",
			STRING_ID);
			strcpy(buf, "DATA");
			(void)nvs_write(&fs, STRING_ID, &buf, strlen(buf) + 1);
		}
	}

	/* LONG_ID is used to store a larger dataset ,lets see if we can read
	 * it from flash
	 */
	rc = nvs_read(&fs, LONG_ID, &longarray, sizeof(longarray));
	if (rc > 0) {
		/* item was found, show it */
		printk("Id: %d, Longarray: ", LONG_ID);
		for (int n = 0; n < sizeof(longarray); n++) {
			printk("%x ", longarray[n]);
		}
		printk("\n");
	} else   {
		/* entry was not found, add it if reboot_counter = 0*/
		if (reboot_counter == 0U) {
			printk("Longarray not found, adding it as id %d\n",
			       LONG_ID);
			for (int n = 0; n < sizeof(longarray); n++) {
				longarray[n] = n;
			}
			(void)nvs_write(
				&fs, LONG_ID, &longarray, sizeof(longarray));
		}
	}

	cnt = 5;
	while (1) {
		k_msleep(SLEEP_TIME);
		if (reboot_counter < MAX_REBOOT) {
			if (cnt == 5) {
				/* print some history information about
				 * the reboot counter
				 * Check the counter history in flash
				 */
				printk("Reboot counter history: ");
				while (1) {
					rc = nvs_read_hist(
						&fs, RBT_CNT_ID,
						&reboot_counter_his,
						sizeof(reboot_counter_his),
						cnt_his);
					if (rc < 0) {
						break;
					}
					printk("...%d", reboot_counter_his);
					cnt_his++;
				}
				if (cnt_his == 0) {
					printk("\n Error, no Reboot counter");
				} else {
					printk("\nOldest reboot counter: %d",
					       reboot_counter_his);
				}
				printk("\nRebooting in ");
			}
			printk("...%d", cnt);
			cnt--;
			if (cnt == 0) {
				printk("\n");
				reboot_counter++;
				(void)nvs_write(
					&fs, RBT_CNT_ID, &reboot_counter,
					sizeof(reboot_counter));
				if (reboot_counter == MAX_REBOOT) {
					printk("Doing last reboot...\n");
				}
				sys_reboot(0);
			}
		} else {
			printk("Reboot counter reached max value.\n");
			printk("Reset to 0 and exit test.\n");
			reboot_counter = 0U;
			(void)nvs_write(&fs, RBT_CNT_ID, &reboot_counter,
			  sizeof(reboot_counter));
			break;
		}
	}
	return 0;
}

I'm not entirely familiar with the nvs code or procedures, so if anyone had any tips I would greatly appreciate it. I'm just trying to store large chunks of data (either int or float values), read from the memory/send over bluetooth, and then delete what I just send from the memory and rewrite the memory location with new data.

Thanks!

[tesc]

Hi,

Documentation wise, there are the documentation for Non-Volatile Storage (NVS), the Non-volatile Storage APIs, and the Non-Volatile Storage (NVS) sample which looks like the code you have found and shared here.

For your particular use case, you might want to look into Flash Circular Buffer (FCB) instead of NVS. While there are less resources available for FCB (there does not seem to be any samples for it in the SDK), it does sound like a closer match to what you are looking for. See also the FCB API documentation.

I assume that you cannot simply store your values in an array in RAM, due to either size constraints, a need to store the data past device reset, or for other reasons.

Please note that flash has a limited number of write/erase cycles, with 10 000 cycles guaranteed for the nRF52833. This means that each flash block, 4 kB of size, can be erased a total ten thousand times before you might start to see glitches on some devices. For this reason, RAM will often be a better alternative for buffering data, provided that the RAM can be kept alive. When using Flash, it is important to set aside enough flash pages relative to the write frequency, that you will not exhaust the write/erase cycles during the expected lifetime of the device.

Regards,
Terje

[connorshannon]

Thank you, that helps. I'm planning on storing ~60 double values per second for some amount of time (may be dependent on storage space limitations). I had heard that flash storage would work better for this application than RAM, but if you think RAM might be better please let me know. I don't need to store data past device reset. I basically just need to store data for some amount of time (the longer the better), and then take a break in reading data and send everything that was stored over Bluetooth.

[tesc]

Hi,

At that cadence you would fill one kB roughly every two seconds, or spend 8-10 seconds filling a Flash page (4 kB on the nRF52 series.) Headers (for FDS) would come in addition. I would definitely go for a fixed buffer or circular buffer approach, and not use a key-value or similar data storage.

Flash has mainly two big advantages over RAM. First, it is persistent, although this is not something you need. Second, it has larger capacity. On the nRF52832, 512 kB of Flash, v.s. 64 kB of RAM. Of course both Flash and RAM have other uses, so the space available for storing measured values is also less. The remaining available storage depends on the application size and memory usage, but it should be easier to find (or save) more space in the larger Flash.

RAM has some advantages over Flash as well. First, notably, unlimited read/write cycles. It will not wear out during the lifetime of the device. Second, speed. In particular very low write time, and no need to do time costly erase operations.

Some quick calculations for the Flash option:

Assuming no overhead, for every flash page used for storage you get roughly 10 (seconds to fill the page) times 10 000 (guaranteed minimum number of flash erase/write cycles) = 100 000 seconds worth of data, before you might start to see glitches in Flash performance. This is somewhat more than one day of continuous data collection (27.7 hours). Accounting for some overhead, and for easier calculation, let's assume one day's worth of data. If, then, you want the lifetime of the product to be at least one year of data collection, then we are already at a Flash storage capacity need of around 1.5 MB (one Flash page of 4 kB per day's worth of lifetime, times 365 days in a full year), or 3x what is available on the nRF52832 altogether.

Now, use of Flash could still be viable, depending on how much of the time the device would be actively recording. For instance, if it records for only one minute every hour, the lifetime would be 13 to 14 months if you use 10 Flash pages (40 kB of Flash). In other words: If the duty cycle of gathering data is 1/60, Flash can be feasible, but if the duty cycle is closer to 100 % then it is most likely unfeasible. The limiting factor there, is the Flash lifetime of 10 000 write/erase cycles.

For the RAM option, I recommend having a look at Ring Buffers. The concepts are pretty well described on that documentation page, along with example code snippets.

Regards,
Terje

[connorshannon]

Thank you, this answer was exceptionally helpful. I do plan on using ring buffers, and am in the midst of figuring out the best way to make the largest buffer possible (given the amount of available RAM). I have a discussion post about this here, but I thought I would keep this thread open in case you have any insights on the best way to do that.

[tesc]

Hi,

I'm afraid I don't have any specific hints or tips regarding ring buffers, other than the ones already given to you in the other thread and in the documentation.

However, regarding data rate, please note that the 2M PHY does not provide 2 Mbps payload throughput. There is overhead for any data transmission, including for BLE. For instance, during connection events packets are sent back and forth both ways between central and peripheral, leaving only half the time for transfers in one particular direction. In addition, there are short pauses between each back and forth, and there are pauses in-between connection events.

To minimize overhead, and maximize throughput, in addition to using 2M phy, you should increase ATT MTU size and data length. See Optimize the connection for throughput - NUS throughput example for details.

Depending on your use case and surroundings, 448,000 bps (i.e. around 450 kbps) should typically be achievable, but not necessarily with default data length, ATT MTU size, connection event length and connection interval.

Please note that if throughput is too low, any ring buffer will fill up regardless its size. Therefore, the most important part of your design, is to make sure that you are able to consistently send data at preferably a somewhat higher rate than produced. The size of the buffer will mostly depend on the jitter (variability in throughput, so that you get occasional bursts of data build-up (producing faster than you can send/consume) and occasional bursts of sending more than is produced (to reduce the amount of buffered data again.))

Regards,
Terje

[hmolesworth]

One other thing to keep in mind, in case it was overlooked: each 0x1000 byte FLASH page is divided into 8 x 0x200 byte blocks. Best to minimise the number of writes to each block, ie suggest buffer 512 bytes in RAM prior to the sequence of 32-bit word flash writes.

"The same block in the Flash can only be written nWRITE number of times before an erase must be performed using ERASEPAGE or ERASEALL"

nWRITE = 181 on the nRF52832, so 128 x 32-bit writes is ok but not 512 x incremental byte writes using '1's for unused bytes. No information is available on pattern sensitivity on adjacent blocks.