How store large amount data 73Kbytes into flash using FDS?

[vishal]

We are using nRF52840 with sdk 15.0 version & segger embedded studio. We are able to store some small amount of configuration data into flash using FDS. But now we want to store large data array near about 73Kbytes into flash. I am writing example code for storing this data and created one dummy array of size 73728 bytes. But I getting following ERROR:

FDS_ERR_RECORD_TOO_LARGE

I am confused for configuration of FDS for storing 73Kbytes 

Here is bellow edited in sdk_config.h

FDS_VIRTUAL_PAGES 10

FDS_VIRTUAL_PAGE_SIZE 2048

FDS_OP_QUEUE_SIZE 4

Will you please help me how i use this above configuration for storing 73KB data. & how i fixed FDS_ERR_RECORD_TOO_LARGE ERROR?

Is it possible to store this data in one fds_record_write (...) call request?

Thanks in advanced..!!

[Ashish]

The number that you give for the FDS_VIRTUAL_PAGE_SIZE is actually 4-byte words. So you can save at a maximum of (2048-5)*4 = 8172 bytes. So you will need to split up your data if you want to save more than that.

[vishal]

Ok, Is it possible to increase FDS_VIRTUAL_PAGE_SIZE as per required data storage above of 2048. If i increase this i got ERROR 11 i.e. NRF_ERROR_INVALID_DATA when fds_flash_init();

We will split our data with smaller chunks with multiple of above maximum limit. But still confused for is it needed to change record key every request of that.

here is bellow my sample code snippet for storing dummy array of 73728bytes.

 

void fds_data_write() {
  uint32_t length = 74000;
  uint8_t data_write[length];
//  memset(data_write,2,sizeof(data_write));
  for (int i = 0; i<=length; i++) {
   data_write[i] = i;
  }
//  NRF_LOG_INFO("fds_data_write");
  ret_code_t ret = fds_write(FILE_ID_READING_1, REC_KEY_READING_1,(char *)data_write,length);
  if (ret == FDS_SUCCESS) {
    NRF_LOG_DEBUG("FDS data stored");
  } else {
    NRF_LOG_DEBUG("FDS failed error code:%d", ret);
  }
}

Will you please provide me program snippet for how i call multiple request for storing 73KB data with FILE-ID and REC_KEY. Because after stored this data we want to also retrieve this data as required time.

Lets say from above whole array first 8172 bytes stored of 1 request with specific FILE_ID & RECKEY, Then i should need to call next chunk data for that is it necessary to change RECKEY.

Because if KECKEY is not change for second call request it will overwritten data on old. Is any other way to store this large data with one FILEID & KECKEY?

Thanks.....

[Einar Thorsrud]

What are the connection parameters in this case? FDS will try to schedule flash operations between BLE events, but it might be difficult if you have a short connection interval.

[vishal]

Hi Ashish & Einar, 

vishal said:

Yes i am using nrf_fstorage_sd.c lib 

Sorry for this wrong library said, actually this lib included in my project not used for write data into flash. But for writing data into flash i am using fds.c & fds.h library. 

Here is below screen shot of  our set connection parameter:

#define NUS_SERVICE_UUID_TYPE BLE_UUID_TYPE_VENDOR_BEGIN /**< UUID type for the Nordic UART Service (vendor specific). */

#define APP_BLE_OBSERVER_PRIO 3 /**< Application's BLE observer priority. You shouldn't need to modify this value. */
#define APP_ADV_INTERVAL 64     /**< The advertising interval (in units of 0.625 ms. This value corresponds to 40 ms). */

#define APP_ADV_DURATION 0   /* Set 0 for contineous advertisng time & handle their timeout by sperate app timer.*/ 

#define MIN_CONN_INTERVAL MSEC_TO_UNITS(7.5, UNIT_1_25_MS)   //20           /**< Minimum acceptable connection interval (20 ms), Connection interval uses 1.25 ms units. */
#define MAX_CONN_INTERVAL MSEC_TO_UNITS(7.5, UNIT_1_25_MS)   //75           /**< Maximum acceptable connection interval (75 ms), Connection interval uses 1.25 ms units. */
#define SLAVE_LATENCY 0                                      /**< Slave latency. */
#define CONN_SUP_TIMEOUT MSEC_TO_UNITS(4000, UNIT_10_MS)     /**< Connection supervisory timeout (4 seconds), Supervision Timeout uses 10 ms units. */
#define FIRST_CONN_PARAMS_UPDATE_DELAY APP_TIMER_TICKS(5000) /**< Time from initiating event (connect or start of notification) to first time sd_ble_gap_conn_param_update is called (5 seconds). */
#define NEXT_CONN_PARAMS_UPDATE_DELAY APP_TIMER_TICKS(90000) /**< Time between each call to sd_ble_gap_conn_param_update after the first call (30 seconds). */
#define MAX_CONN_PARAMS_UPDATE_COUNT 3                       /**< Number of attempts before giving up the connection parameter negotiation. */

#define DEAD_BEEF 0xDEADBEEF      /**< Value used as error code on stack dump, can be used to identify stack location on stack unwind. */
#define BOOTLOADER_DFU_START 0xB1 // Bootloader DFU

#define UART_TX_BUF_SIZE 2048 /**< UART TX buffer size. */
#define UART_RX_BUF_SIZE 2048 /**< UART RX buffer size. */

Thanks..

[vishal]

Looking forward your response..!!!

Thanks,

[Einar Thorsrud]

Hi vishal,

I am sorry for the late reply. Somehow I forgot about this thread.

I see you configure 7.5 ms connection interval. I assume you are making a peripheral, so weather this is used or not I don't know. the same for slave latency and supervision timeout. Can you check what is actually used on the link when you test? After all the peripheral can only request it, the central decides. If it is as low as 7.5 ms and the supervision timeout is short (not 4 seconds as shown here, then it might not be possible for the SoftDevice to schedule a flash operation.

Reading over the tread again the connection interval question might be irrelevant, as you do not even enable the SoftDevice in the code you posted earlier. At the same time there is talk about NUS etc. In fact, I am not able to understand the context here. Can you write current question in a new post and include all relevant information there so that we know in detail how your application looks like, what it is supposed to do and how it does not work. Please add all relevant code. Please make that post stand-alone so that it contains all relevant information.

[vishal]

Thanks Einar for your suggestion..!!

I will create another thread for this problem.. But now i have question regarding flash data storage Is it necessary to define fds write data buffer as static. Because my data length wan to write is 74000 If i define as static then RAM used more. In ca if not define this buffer as static then some last data is not write correctly. 

Will you suggest other way instead of using static buffer.

Thanks....

[Einar Thorsrud]

Hi,

You cannot write that much in one go, so there is no need to have that large write buffer. The maximum record size is the size of a virtual page less a few bytes for header etc. For a typical a virtual page size of 1024 words, the maximum record size is 1018 words, which is 4072 bytes.

The write buffer does not strictly has to be static, but in practice that is almost always sensible, as the write buffer has to be valid for the entire duration of the write operation, which is asynchronous. The key here is that the buffer must be valid until you get the callback indicating that the write operation has finished. How you do that is up to you.

[vishal]

Yes you are right, we are not writing all data 74KB at once. We have write with 4068 bytes of chunk data in to flash memory.  Also successfully write into flash here is our below function: Will you please review this function and is it correct way...

ret_code_t fds_reading_write(uint16_t file_ID, uint8_t const *data_to_write, const uint32_t data_length) {
  uint32_t fds_write_counter = data_length;
  ret_code_t err_code;
  fds_record_t record;
  for (uint16_t pages = 1; pages < 20; pages++) {
    fds_record_desc_t record_desc = {0};
    fds_find_token_t ftoken;
    memset(&ftoken, 0x00, sizeof(fds_find_token_t));
    memset(&record_desc, 0x00, sizeof(fds_record_desc_t));
    // Set up record.
    record.file_id = file_ID;
    record.key = pages;
    if (fds_write_counter < LENGTH_TO_SEND) {
      record.data.length_words = ((fds_write_counter + 3) / sizeof(uint32_t));
    } else {
      record.data.length_words = ((LENGTH_TO_SEND + 3) / sizeof(uint32_t));
    }
    record.data.p_data = &data_to_write[DATA_STORE_LEN - fds_write_counter]; //DATA_STORE_LEN = 74000
    err_code = fds_record_write(&record_desc, &record);
    APP_ERROR_CHECK(err_code);
    wait_for_next_write();
    if (err_code == FDS_SUCCESS) {
      NRF_LOG_DEBUG("Write Sucess:%d", fds_write_counter);
      if (fds_write_counter < LENGTH_TO_SEND) {
        fds_write_counter = 0;
        free((uint8_t *)data_to_write);
      } else {
        fds_write_counter -= LENGTH_TO_SEND;
      }
    } else {
      NRF_LOG_DEBUG("Write Failed, id %d", err_code);
    }
  }
  return NRF_SUCCESS;
}

Thanks for your great supporting..!!

[Einar Thorsrud]

Hi,

I am missing the whole picture, this looks OK. It seems like you have a buffer containing the whole 74 kB (pointed to by data_to_write), and you do separate writes until it succeeds. I want to mention that this function must run in a low priority (typically main context), so that the event indicating that write operation is successful can run, which I assume updates some state variable or similar used by your wait_for_next_write() function. Assuming that is OK, it should work.

(As a side note I do not personally like that you call free inside this write function when you have written the last chunk, as it is not obvious for the caller code that calling this function would eventually free the input buffer).

[vishal]

Thank you so much for your suggestion..!!

Now you are saying in this above function free((uint8_t *)data_to_write); is not necessary to do when last chunk is successfully write. can i remove this will not affected on run time memory.

Thanks... 

[Einar Thorsrud]

vishal said:

Now you are saying in this above function free((uint8_t *)data_to_write); is not necessary to do when last chunk is successfully write. can i remove this will not affected on run time memory.

Not exactly. I have no idea where you allocate the memory. I just commented that it looks a bit strange to free the memory there, as there is no link to where it was allocated. It is not readable. I don't have full overview of your application, but if you allocate memory dynamically then you probably want to free it somewhere as well (unless you only allocate it once and can live with it not being freed, but then you should have just used a static buffer instead).