Frequent NVS Writes

[Neil Benn]

Hello,

  I'm working up an application where tens of kilobytes are sent via bluetooth to a device which then stores the data on the internal flash (nrf52832).  I've worked on defining a larger section of memory for the flash and the data is sent over in chunks (write without response).  This is then stored until it reaches a certain arbitrary size and this is stored on the NVS.  For the first couple of writes this works just fine but after that I get an error of -22 which seems to point to invalis argument.  Anyway I have reduced this to a minimal use which is a struct as shown below:

/**
 * @brief This struct is used to store a slice of compressed data
 * @param data the compressed data
 * @param len the length of the compressed data
 * @param uncompressed_size the size of the uncompressed data
 */
typedef struct compressed_data_slice{
    uint8_t *data;
    uint32_t len;
    uint32_t uncompressed_size;
}COMPRESSED_DATA_SLICE;

  This is then used to store the data with the following code:

int test_store_slices(){
    COMPRESSED_DATA_SLICE cds;
    cds.len = 1056;
    cds.uncompressed_size = 10240;
    cds.data = k_malloc(cds.len);
    for (int i = 0; i < cds.len; i++){
        cds.data[i] = i;
    }
    for (int i = 0; i < 10; i++){
        store_slice(cds);
    }
    return 0;
}

int store_slice(COMPRESSED_DATA_SLICE cds){
    k_mutex_lock(&flash_mutex, K_FOREVER);
    int ret = 0;
    vid_file_id++;
    uint16_t slice_id = vid_file_id;
    //bw_settings_write("last_slice_id", &slice_id, sizeof(slice_id));
    LOG_INF("Storage of slice of len %" PRIu32 " bytes with uncompressed size of %" PRIu32 " bytes with file_id of %i", 
        cds.len, cds.uncompressed_size, slice_id);
    uint8_t *store_data;
    store_data = k_malloc(cds.len + 4);
    memcpy(store_data, &cds.uncompressed_size, 4);
    memcpy(store_data + 4, cds.data, cds.len);

    ret = nvs_delete(&fs, slice_id);
    if (ret < 0){
        LOG_ERR("Flash delete failed, rc=%d\n", ret);
    }
    
    ret = nvs_write(&fs, slice_id, store_data, cds.len + 4);
    if (ret < 0) {
        LOG_ERR("Flash write failed, rc=%d\n", ret);
    }
    
    ssize_t free_bytes = nvs_calc_free_space(&fs);
    LOG_INF("Free space is %" PRIu32 " bytes", free_bytes);
    k_mutex_unlock(&flash_mutex);
    return ret;
}

  This shows the same pattern as when the data is sent over the BLE connection, the first two writes work and the rest fail:

*** Booting nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
[00:00:00.008,850] <inf> brikwiz_base: Starting BrikWiz Base Peripheral

[00:00:00.008,850] <inf> brikwiz_storage: Init storage
[00:00:00.008,880] <inf> brikwiz_storage: Storage size is 4096 index is 112
[00:00:00.008,880] <inf> brikwiz_storage: Partition size is 65536 offset is 458752
[00:00:00.010,528] <inf> fs_nvs: 16 Sectors of 4096 bytes
[00:00:00.010,559] <inf> fs_nvs: alloc wra: 5, f50
[00:00:00.010,559] <inf> fs_nvs: data wra: 5, efc
[00:00:00.023,498] <inf> brikwiz_storage: Free space is 55480 bytes
[00:00:00.023,620] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 1
[00:00:00.064,361] <inf> brikwiz_storage: Free space is 54404 bytes
[00:00:00.064,392] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 2
[00:00:00.100,219] <inf> brikwiz_storage: Free space is 53684 bytes
[00:00:00.100,219] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 3
[00:00:00.101,043] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.115,509] <inf> brikwiz_storage: Free space is 54536 bytes
[00:00:00.115,539] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 4
[00:00:00.116,088] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.130,737] <inf> brikwiz_storage: Free space is 54536 bytes
[00:00:00.130,737] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 5
[00:00:00.131,317] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.145,965] <inf> brikwiz_storage: Free space is 54536 bytes
[00:00:00.145,965] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 6
[00:00:00.146,789] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.161,315] <inf> brikwiz_storage: Free space is 54932 bytes
[00:00:00.161,315] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 7
[00:00:00.162,139] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.176,727] <inf> brikwiz_storage: Free space is 55072 bytes
[00:00:00.176,757] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 8
[00:00:00.177,551] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.192,321] <inf> brikwiz_storage: Free space is 55628 bytes
[00:00:00.192,352] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 9
[00:00:00.192,932] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.207,885] <inf> brikwiz_storage: Free space is 55628 bytes
[00:00:00.207,916] <inf> brikwiz_storage: Storage of slice of len 1056 bytes with uncompressed size of 10240 bytes with file_id of 10
[00:00:00.208,496] <err> brikwiz_storage: Flash write failed, rc=-22

[00:00:00.223,480] <inf> brikwiz_storage: Free space is 55628 bytes

  Can you please advise as to the source of the problem?  It can't be space as there is a lot of space but I'm at a loss as to the cause - especially with such a simple routine.

  Thanks.

Cheers,

Neil

[Neil Benn]

I put some code in to check the sector space and increased the size of each sector but that is not the cause, it still fails, interestingly all write under ~400 bytes work with everything over 550 bytes failing and intermittent failures between those two levels.

[Håkon Alseth]

Hi,

  

I see that you are allocating memory, but there's no free-call? Where do you call k_free()?

You do not check the return ptr from malloc, which I strongly recommend that you do.

 

What is the configured system heap size, ie. CONFIG_HEAP_MEM_POOL_SIZE?

 

Kind regards,

Håkon

[Neil Benn]

Hello,

  Good point on the memory freeing thanks - though this is just a test case and I'll throw the code away once I can work out how to store larger entries; on the memory I've got:

CONFIG_HEAP_MEM_POOL_SIZE=4026

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=9048

  I increased the stack size because in the real use case I'll be taking data in from a BLE callback.

Cheers,

Neil

[Håkon Alseth]

Hi Neil,

 

Were you able to get past the second iteration after free'ing the data?

 

Kind regards,

Håkon

[Håkon Alseth]

Hi,

 

Is the function store_slice_bw() also calling k_malloc()?

for (int i = 0; i < 10; i++){
    delete_slice(i + 1);
    store_slice_bw(cds, true);
}

 

if it is based on the original "store_slice" function, it seems to allocate cds.len+4 bytes within a for-loop, draining the HEAP.

 

Kind regards,

Håkon