Problems writing/reading struct to NVS

I've been slowly adding NVS functionality to my code.  Has been working fine, up until today when I attempted to write a struct I have for storing historical data to flash.

This is the struct:

struct log_data {
	// data structure to store individual strike events
	uint8_t	strike_level;  // SML = 1, MED = 2 or LRG = 4
	uint16_t	total_strike_count;  // Total number of strikes recorded to date
	struct bt_cts_current_time time_detected;  //Time stamp for each strike
};

I then have an array of this struct.  At this point, MAX_STRIKE_RECORDS = 10, but ultimately I want to see this at 250.

struct log_data lsr_strike_log[MAX_STRIKE_RECORDS];

In my code to read the data in from Flash, I have the following:

	uint32_t rc = 0;
	rc = nvs_read(&fs, HISTORY_ID, &lsr_strike_log, sizeof(lsr_strike_log));
	printk("sizeof = %u\n", sizeof(lsr_strike_log));
	printk("strike history rc = %u\n", rc);

The return value from nvs_read() is supposed to be the amount of data read in, if everything works as expected.  In my case, this should be equal to sizeof(lsr_strike_log), which is 160 bytes.  If the return value is larger than this, it apparently means there is more data to read in.  This is what I get from my two printk() statements:

sizeof = 160
strike history rc = 4294967294

So, its like the nvs_read() is going outside my defined flash area, or something - I'm not sure, hence why I'm posting here to see if someone can help me understand what's going on better.
My flash is initialised as per the following, and I don't get any of the error messages output via UART, so it appears to be initialising OK.
flash_dev = FLASH_AREA_DEVICE(STORAGE_NODE_LABEL);
	if (!device_is_ready(flash_dev)) {
		printk("Flash device %s is not ready\n", flash_dev->name);
		return -EINVAL;
	}
	fs.offset = FLASH_AREA_OFFSET(storage);
	rc = flash_get_page_info_by_offs(flash_dev, fs.offset, &info);
	if (rc) 
	{
		printk("Unable to get page info\n");
		return rc;
	}
	fs.sector_size = info.size;
	fs.sector_count = 3U;

	rc = nvs_init(&fs, flash_dev->name);
	if (rc) 
	{
		printk("Flash Init failed\n");
		return rc;
	}
If I enable CONFIG_LOG=y in my proj.config file, I see the following output:
I suspect I'm doing something obvious wrong. Problem is, its not obvious to me.
Thanks and regards,
Mike
Parents
  • Thanks Haakonsh.

    If I simplify what I'm trying to write to NVS, it seems to work OK.  I changed my data structure to just a simple array:

    uint16_t lsr_strike_log[9];

    And if I write/read this to NVS, I can see that the data is being stored.  I'm still getting weird values for the return value from nvs_write() and nvs_read() though.

    This is the log info I see when writing the simple array:

    If I try and write my struct to NVS in the form I ultimately want to, I crash my DK and it keeps rebooting.

    In my code I initialse NVS before I initialise anything to do with BLE, so not sure if the BLE initialisation then changes what I've done when setting up NVS?

    Thanks and regards,

    Mike

  • Hello Mike,

    What is the current state on this issue? 

    Best regards,

    Simon

  • Hi Simon,

    I’m just about to get back into this part of it after struggling with issues around GPIO/System OFF/MCUBoot for nearly a month.

    I’ll update this thread in a week or so when I’ve had a chance to do some more testing

    Cheers,

    Mike

  • Mike Austin (LPI) said:
    I’ll update this thread in a week or so when I’ve had a chance to do some more testing

    Sounds goodThumbsup

  • Hi Simon,

    I seem to have this working now.  Only problem is, I'm not able to get everything I want stored in NVS as I am running out of room.  And I'm not sure how to make more space, or indeed if I can.

    My struct takes up 12 bytes per array element, and I am trying to have capacity for up to 250 array elements.  At the moment, I hit a wall around 167.  I've got a few other bytes of info I store in NVS that are separate to this array - they take up about 15 bytes in total.

    This is how the flash is allocated in my .dts file (this is the default for the nRF52-DK)

    &flash0 {
    
    	partitions {
    		compatible = "fixed-partitions";
    		#address-cells = <1>;
    		#size-cells = <1>;
    
    		boot_partition: partition@0 {
    			label = "mcuboot";
    			reg = <0x00000000 0xc000>;
    		};
    		slot0_partition: partition@c000 {
    			label = "image-0";
    			reg = <0x0000C000 0x32000>;
    		};
    		slot1_partition: partition@3e000 {
    			label = "image-1";
    			reg = <0x0003E000 0x32000>;
    		};
    		scratch_partition: partition@70000 {
    			label = "image-scratch";
    			reg = <0x00070000 0xa000>;
    		};
    		storage_partition: partition@7a000 {
    			label = "storage";
    			reg = <0x0007a000 0x00006000>;
    		};
    	};
    };

    And this is how I have my flash device set up in my code (based on the example code)

    int16_t flash_initialise(void)
    {
        int16_t rc;
    
    	flash_dev = FLASH_AREA_DEVICE(STORAGE_NODE_LABEL);
    	if (!device_is_ready(flash_dev)) {
    		#ifdef DEBUG_NVS
    			printk("Flash device %s is not ready\n", flash_dev->name);
    		#endif
    		return -EINVAL;
    	}
    	printk("Flash device %s is ready\n",flash_dev->name);
    
    	fs.offset = FLASH_AREA_OFFSET(storage);
    	rc = flash_get_page_info_by_offs(flash_dev, fs.offset, &info);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS		
    			printk("Unable to get page info\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Page info OK\n");
    	#endif
    	fs.sector_size = info.size;
    	fs.sector_count = 2U;
    
    	rc = nvs_init(&fs, flash_dev->name);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS
    			printk("Flash Init failed\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Flash initialised OK\n");
    	#endif
    
    	// Check if data is stored in flash
    	rc = check_for_nvs_key();
    	return rc;
    	
    }

    As I understand it, the available flash storage size is 0x6000, which is about 24kB.  And I have allocated two sectors, each of 4096 bytes.

    So, I'm not sure why I'm running into storage problems.

    Sorry if this sounds like a pretty basic question, but I can't seem to find any info that would help me understand why I'm running out of storage space and how to increase it

    Cheers,

    Mike

  • Sorry for the delay, I was gone on Friday.

    Could you check if the partition manager is enabled?

    You can do that by searching for CONFIG_PARTITION_MANAGER_ENABLED in <sample>build/zephyr/.config and see if it's set to 'y'.

    Partition Manager is enabled when you're including child images (like mcuboot) and the dts partitions will be ignored.

    Best regards,

    Simon

  • Hi Simon,

    Yes, I have partition manager enabled via:

    #
    # Partition Manager
    #
    CONFIG_PARTITION_MANAGER_ENABLED=y
    CONFIG_FLASH_MAP_CUSTOM=y
    CONFIG_SRAM_SIZE=64
    CONFIG_SRAM_BASE_ADDRESS=0x20000000
    Cheers,
    Mike
Reply Children
  • Ah, okay.

    Check the note in the documentation about the Partition Manager:

    "When you build a multi-image application using the Partition Manager, the Device Tree Source flash partitions are ignored."

    So you should create the storage partition through a pm_static.yml file

    Vidar provided one example how to go about this in this reply:  RE: Problem to read back flash with NVS when concurrent use with Bluetooth 

    Then you should be able to access the size and address of the user storage from your application (generated defines will be located in <sample>\build\zephyr\include\generated\pm_config.h I believe), which you can use to set up NVS to store custom data.

    By the way, Bluetooth bonding uses NVS as well (settings_storage), so that may interfere with your custom NVS data, in the ticket above that was resolved by creating a separate partition (user_storage).

    However, I think it should be possible to use the same partition for both cases, see  Concurrent usage of NV and settings on the same partition 

    Best regards,

    Simon 

  • Hi Simon,

    I'm definitely entering unchartered waters here now!

    OK, so this is what my partitions.yml file looked like before:

    mcuboot_primary_app:
      address: 0xc200
      end_address: 0x45000
      orig_span: &id002
      - app
      region: flash_primary
      size: 0x38e00
      span: *id002
    mcuboot_secondary:
      address: 0x45000
      end_address: 0x7e000
      placement:
        after:
        - mcuboot_primary
        align:
          start: 0x1000
      region: flash_primary
      share_size:
      - mcuboot_primary
      size: 0x39000
    settings_storage:
      address: 0x7e000
      end_address: 0x80000
      placement:
        align:
          start: 0x1000
        before:
        - end
      region: flash_primary
      size: 0x2000
    sram_primary:
      address: 0x20000000
      end_address: 0x20010000
      region: sram_primary
      size: 0x10000

    So, I created a pm_static.yml file as you suggested.  A simple copy and paste of the one Vidar listed threw up some errors associated with gaps in the memory (or something).  So, I made my pm_static.yml look like this so that the gaps in memory, as I saw it, were "plugged"

    user_storage:
      address: 0x80000
      size: 0x2000
      end_address: 0x82000
      placement:
        before: 
        - settings_storage
      region: flash_primary
    settings_storage:
      address: 0x82000
      size: 0x2000
      end_address: 0x84000
      placement:
        before:
        - end
      region: flash_primary

    This is giving me 0x2000 of settings_storage and an additional 0x2000 of user_storage, which in theory should be enough.

    Then, where I define my flash device, I have:

    #define STORAGE_NODE_LABEL user_storage
    const struct device *flash_dev;
    static struct nvs_fs fs;
    
    // Initialise NVS and check for NVS key
    // Presence of key means there is valid data stored in NVS
    int16_t flash_initialise(void)
    {
        int16_t rc;
    
    	flash_dev = FLASH_AREA_DEVICE(STORAGE_NODE_LABEL);
    	if (!device_is_ready(flash_dev)) {
    		#ifdef DEBUG_NVS
    			printk("Flash device %s is not ready\n", flash_dev->name);
    		#endif
    		return -EINVAL;
    	}
    	
    	#ifdef DEBUG_NVS
    		printk("Flash device %s is ready\n",flash_dev->name);
    	#endif
    
    	fs.offset = FLASH_AREA_OFFSET(STORAGE_NODE_LABEL);
    	rc = flash_get_page_info_by_offs(flash_dev, fs.offset, &info);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS		
    			printk("Unable to get page info\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Page info OK\n");
    	#endif
    	fs.sector_size = info.size;
    	fs.sector_count = NVS_SECTOR_COUNT;
    
    	rc = nvs_init(&fs, flash_dev->name);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS
    			printk("Flash Init failed\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Flash initialised OK\n");
    	#endif
    
    	// Check if data is stored in flash
    	rc = check_for_nvs_key();
    	return rc;
    	
    }

    I use my function flash_initialise to set up my storage*

    That all builds OK, and I can flash it to my DK.  Problem is, when my code fires up BLE advertising, the advertising fails to start, and I get error code "-EAGAIN" from the call to bt_le_adv_start_legacy(), which gets called as part of bt_le_adv_start().

    So, I'm doing something wrong, as when I go back to the dynamic memory allocation using the DTS, the problem goes away.

    Can you see anything obvious that I'm doing that would affect the BLE functionality?

    * As an aside, the nvs_init() that I call in flash_initialise() shows as deprecated when I build, but my code crashes when I replace that with the suggested alternative nvs_mount().  Not sure why.

    Cheers,

    Mike

  • Hi Simon,

    Did a bit more digging.  The issue is in fact when I try and get the flash page info.  I think the BLE thing is a side issue.

    I trawled through the DevZone and found this, which seems to be along similar lines to what I'm doing:

     Issue with flash partitions in NCS 1.7.0 on nRF9160 

    This is what my flash initialisation function now looks like:

    int16_t flash_initialise(void)
    {
        int16_t rc;
    
    	//flash_dev = DEVICE_DT_GET(FLASH_NODE);
    	flash_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_flash_controller));
    	if (!device_is_ready(flash_dev)) {
    		#ifdef DEBUG_NVS
    			printk("Flash device %s is not ready\n", flash_dev->name);
    		#endif
    		return -EINVAL;
    	}
    	
    	#ifdef DEBUG_NVS
    		printk("Flash device %s is ready\n",flash_dev->name);
    	#endif
    
    	fs.offset = FLASH_AREA_OFFSET(storage);
    	rc = flash_get_page_info_by_offs(flash_dev, fs.offset, &info);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS		
    			printk("Unable to get page info\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Page info OK\n");
    	#endif
    	fs.sector_size = info.size;
    	fs.sector_count = NVS_SECTOR_COUNT;
    
    	rc = nvs_init(&fs, flash_dev->name);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS
    			printk("Flash Init failed\n");
    		#endif
    		return rc;
    	}
    	#ifdef DEBUG_NVS
    		printk("Flash initialised OK\n");
    	#endif
    
    	// Check if data is stored in flash
    	rc = check_for_nvs_key();
    	return rc;
    	
    }

    So, I'm now getting the device via:

    flash_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_flash_controller));

    And that appears to be working.  But when I then attempt to get the offset for my NVS file system struct, via:

    	fs.offset = FLASH_AREA_OFFSET(user_storage);
    	rc = flash_get_page_info_by_offs(flash_dev, fs.offset, &info);
    	if (rc !=0) 
    	{
    		#ifdef DEBUG_NVS		
    			printk("Unable to get page info\n");
    		#endif
    		return rc;
    	}

    I'm still not successfully getting the page info, as I get the error message "Unable to get page info"

    Clearly, I'm not really understanding how, with the static partitions definition as opposed to those defined in the .DTS, I should be initialising my flash device.

    Cheers,

    Mike

  • I played around with the NVS sample and made it use the Partition Manager (PM) instead of DTS (did this by setting CONFIG_BOOTLOADER_MCUBOOT=y, since PM will be used when child images are added).

    I was able to make it work with nrf52dk_nrf52832 and NCS v2.0.0

    nvs_prt_mngr.zip

    If the PM is enabled the macros used (FLASH_AREA_OFFSET() for example), will get info from PM instead of DTS. See 

    One thing you should be aware of, don't let NVS use the area outside the custom partition. Make sure (sector size)*(sector count) is smaller than user_storage.

    Best regards,

    Simon

  • Hi Simon,

    Thanks for sticking with me on this!

    OK, I'm getting things working, but I'm still coming unstuck when I try and push the size of my data logging array up towards the size I want it.  I'm clearly not understanding how the flash is arranged/configured. Or there is something else at play

    At the moment, I have the static partitions configured as follows:

    user_storage:
      address: 0x7c000
      size: 0x2000
      placement:
        before: 
        - settings_storage
      region: flash_primary
    settings_storage:
      address: 0x7e000
      size: 0x2000
      placement:
        before:
        - end
      region: flash_primary

    I'm outputting my flash and user info data via UART so I can check everything is OK.  This is what I get with the above configuration, and with the number of sectors set to 2.

    Sector size = 4096
    Sector count = 2
    Sector size*count = 8192
    User storage = 8192

    Now, what I'm writing to flash consists of the following:

    	(void)nvs_write(&fs, TOTAL_STRIKE_COUNT_ID, &strike_count_total, sizeof(strike_count_total));
    	(void)nvs_write(&fs, STRIKE_SML_ID, &strike_count_sml, sizeof(strike_count_sml));
    	(void)nvs_write(&fs, STRIKE_MED_ID, &strike_count_med, sizeof(strike_count_med));
    	(void)nvs_write(&fs, STRIKE_LRG_ID, &strike_count_lrg, sizeof(strike_count_lrg));
    	(void)nvs_write(&fs, LOG_DATA_ID, &lsr_strike_log, sizeof(lsr_strike_log));

    These are defined as:

    #define MAX_STRIKE_RECORDS 150
    
    uint16_t strike_count_total = 0;
    uint16_t strike_count_sml = 0;
    uint16_t strike_count_med = 0;
    uint16_t strike_count_lrg = 0;
    
    struct log_data {
    	uint8_t	strike_level;  // SML = 1, MED = 2 or LRG = 4
    	uint16_t year;
    	uint8_t month;
    	uint8_t day;
    	uint8_t hours;
    	uint8_t minutes;
    	uint8_t seconds;
    };
    
    struct log_data lsr_strike_log[MAX_STRIKE_RECORDS];
    

    What I'm trying to do is make enough flash space available to be able to store all that, but with MAX_STRIKE_RECORDS = 250.  At the moment, things go pear shaped with MAX_STRIKE_RECORDS > 195.  And I'm not sure why, nor what to do about it.  By my reckoning, I have more than enough space, but it almost seems like what I'm trying to store is consuming twice what I think it is.

    From here: https://docs.zephyrproject.org/latest/services/storage/nvs/nvs.html it looks like each of the elements being stored in flash needs:

    8 bytes of meta-data + 1 byte of id + sizeof(data).

    I have 8 elements in total (key, batch, serial, total, sml, med, lrg and log)

    Each one of my lsr_strike_log array elements seems to use up 10 bytes of data.

    So, by my calcs, the total amount of flash storage for 250 records is:

    key = 8 + 1 + 3 = 12

    batch = 8 + 1 + 4 = 13

    serial = 8 + 1 + 2 = 11

    total = 8 + 1 + 2 = 11

    sml = 8 + 1 + 2 = 11

    med = 8 + 1 + 2 = 11

    lrg = 8 + 1 + 2 = 11

    log = 8 + 1 + 250x10 = 2509

    Which gives me 2589 bytes.  I have 4096 in each sector, which seems ample to me.  But things bomb out when I go past MAX_STRIKE_RECORDS = 195.  This requires 2039 bytes in total.  Which is almost exactly half the size of one sector (4096 bytes).

    If I set MAX_STRIKE_RECORDS > 195, then when I try and write the log data to flash via:

    val = nvs_write(&fs, LOG_DATA_ID, &lsr_strike_log, sizeof(lsr_strike_log));
    if (val < 0)
    {
    	printk("Return val on log data write  = %d\n", val);
    }

    I get a return value of -28, which equates to:

    ENOSPC 28       /**< No space left on device */

    So, my questions are:

    1. Why, when it looks like I've got more than enough user storage allocated, do I run out of room?

    2. How can I increase my user storage so I have enough room for 250 log array elements?

    Cheers,

    Mike

Related