Dynamic arrays and FDS

[mazis]

Hello Nordic!

I need to save dynamic arrays in NRF52840 flash using FDS, but after some days of trial and error I need to beg for help.

What I need to achieve is to have a dynamic static global array of a structure I created where I will be recording the access credentials of my doorlock. So when more credentials arrive at my device, I append this data in my previous dynamic array and store it in flash. I've been using the door_lock_nus example as a guide because there is an example of FDS usage so I made something similar.

First of all, I define the structure, with an uint16, two uint8 and a uint32 variables, so 64 bits in total per array position. And then I define my array of structs but without initialization. Something like this:

 

static user_credentials_t credentials[];

When I initialize the device, I read the saved credentials by using fds_record_find, fds_record_open and memcpy. For the memcpy byte size parameter I have a previous atribute that I'm storing and reading too in the flash which is the "size" of the array. As the Zigbee Door Lock Cluster has an attribute called "Number of total users" I'm using it as my index array. So first I read the "Number of total users" and then the credentials array.

Then, when new credentials are sent, I read them and append in the last position of my array. I also increase the number of users and set the attribute using zb_zcl_set_attr_val. When I store the new credentials in the flash, I am updating too the .data.p_data and .data.length_words of my fds_record_t.

Somewhere in between theese steps is failing. I have noticed that when the stored data has more than 3 credentials stored fails at loading but I can't figure out what is failing. I have also tried allocating memory but then I came up wit this link. So I remove my malloc and realloc instructions and change my approach.

I hope everything is well explained as I am from Spain. I haven't provide any code snippet because it is written in spanish, but I could if you wish.

Thank you all!

Cheers!

 

[tesc]

Hi,

It is hard to tell without more details, but I suspect that your approach involving the incomplete array type "static user_credentials_t credentials[];" is not good.

Can you share the code? Even if it uses variable names, function names, comments, etc. in a different language, I think I should be able to understand what it does.

Also, I can have a look at the FDS data as stored in flash. You can use nrfjprog for that, with the following command:

nrfjprog --readcode flashdump.hex

It will read out the flash contents, and store it in a file named flashdump.hex.

Please note that this is a public thread. We can turn it into a private case before you attach anything, if you want to keep your source code and/or flash dump confidential. In that case let me know.

Regards,
Terje

[mazis]

Hello tesc!

I can share the code, no problem. Please, find the involved functions below.

This is the structure I have defined according to the Zigbee Cluster Reference and how the array of credentials is defined:

typedef  struct credenciales_acceso_s
{
    zb_uint16_t user_ID;
    zb_uint8_t  user_status;
    zb_uint8_t  user_type;
    zb_uint32_t pin_code;
} credenciales_acceso_t;

static credenciales_acceso_t credenciales[];

Here is the way I am reading my credentials record from flash, only the first time the device is started. It's just a part of the function that initializes the FDS and reads the other parameters saved in flash (one of them is the Number of total users that I am using as the array lenght):

    // CARGAMOS LAS CREDENCIALES
    fds_record_desc_t desc1 = {0};
    fds_find_token_t  tok1  = {0};
    fds_flash_record_t flash_record = {0};
    //contexto_cerradura.mis_atributos_cerradura.num_usuarios = 0;
    zb_uint16_t nUsuarios = contexto_cerradura.mis_atributos_cerradura.num_usuarios;
    // CONFIGURAMOS EL ARCHIVO FDS
    m_credenciales_record.file_id           = CREDENCIALES_FILE;
    m_credenciales_record.key               = CREDENCIALES_STATE_KEY;
    m_credenciales_record.data.p_data       = &credenciales;
    m_credenciales_record.data.length_words = (sizeof(credenciales_acceso_t) * nUsuarios) / sizeof(uint32_t);
    
    if (fds_record_find(CREDENCIALES_FILE, CREDENCIALES_STATE_KEY, &desc1, &tok1) == NRF_SUCCESS)
    {
        if (nUsuarios > 0)
        {
            error_code = fds_record_open(&desc1, &flash_record);
            APP_ERROR_CHECK(error_code);

            /* Copy the configuration from flash into the configuration in the RAM. */
            memcpy(&credenciales, flash_record.p_data, sizeof(credenciales_acceso_t) * nUsuarios);

            error_code = fds_record_close(&desc1);
            APP_ERROR_CHECK(error_code);
        }                
    }
    else    
    {
        NRF_LOG_INFO("Previous configuration not found, creating one");

        error_code = fds_record_write(&desc, &m_credenciales_record);
        APP_ERROR_CHECK(error_code);
    }

And this is the function that appends new credentials to my credentials array and stores it in the flash. Right now I am calling the function from the buttons handler to simulate a new credential has arrived. As I mentioned in the first post. I thougth about using malloc in the write function to increase the array size but it is discouraged. I really think that the failure is in this funcion because the loading function seems correct to me:

zb_void_t set_pin_code(zb_uint16_t userID, zb_uint8_t userStatus, zb_uint8_t userType, zb_uint32_t pinCode)
{
    zb_uint16_t nUsuarios = contexto_cerradura.mis_atributos_cerradura.num_usuarios;

    credenciales[nUsuarios].user_ID = userID + nUsuarios;
    credenciales[nUsuarios].pin_code = pinCode + nUsuarios;
    credenciales[nUsuarios].user_status = userStatus + nUsuarios;
    credenciales[nUsuarios].user_type = userType + nUsuarios;
    nUsuarios+=1;
    // ACTUALIZAMOS EL VALOR DEL ATRIBUTO
    UNUSED_RETURN_VALUE(zb_zcl_set_attr_val(CERRADURA_ENDPOINT,
                                            ZB_ZCL_CLUSTER_ID_DOOR_LOCK,
                                            ZB_ZCL_CLUSTER_SERVER_ROLE,
                                            ZB_ZCL_ATTR_DOOR_LOCK_DOOR_STATE_ID,
                                            &nUsuarios,
                                            ZB_FALSE));
    contexto_cerradura.mis_atributos_cerradura.num_usuarios = nUsuarios;
    // ACTUALIZAMOS EL VALOR DE LAS CREDENCIALES GUARDADAS EN LA FLASH
    ret_code_t        error_code;
    fds_record_desc_t desc = {0};
    fds_find_token_t  tok  = {0};
    
    m_credenciales_record.data.p_data       = &credenciales;
    m_credenciales_record.data.length_words = (sizeof(credenciales_acceso_t) * nUsuarios) / sizeof(uint32_t);

    error_code = fds_record_find(CREDENCIALES_FILE, CREDENCIALES_STATE_KEY, &desc, &tok);
    APP_ERROR_CHECK(error_code);

    error_code = fds_record_update(&desc, &m_credenciales_record);
    APP_ERROR_CHECK(error_code);
    /* Set the garbage flag to dispatch the garbage collector. */
    //m_garbage_flag = true;

    fds_record_desc_t desc1 = {0};
    fds_find_token_t  tok1  = {0};    

    error_code = fds_record_find(DOOR_LOCK_CONFIG_FILE, DOOR_LOCK_CONFIG_STATE_KEY, &desc1, &tok1);
    APP_ERROR_CHECK(error_code);
    error_code = fds_record_update(&desc1, &m_configuration_record);
    APP_ERROR_CHECK(error_code);
    /* Set the garbage flag to dispatch the garbage collector. */
    m_garbage_flag = true;
}

About the fds_record_t that I am using to read and save the data from flash it is defined but not initialized until the FDS initialization function.

I wanted to ask you how do I use the nrfjprog --readcode flashdump.hex. Do I need some additional SW?

Thank you so much!

Cheers!

[tesc]

Hi,

I do not see that you initialize the incomplete array "credentiales[]" anywhere. It cannot be used as a variable size array. I would expect undefined behaviour if you try to write and/or read any indexes other than index 0. Rather, you should decide on a maximum number of credentials, and initialize the array as e.g. "credentiales[10]" (if 10 is the max.)

nrfjprog is part of nRF Command Line Tools. You need to be connected through a J-Link programmer, the same way as when programming the board, running debug sessions, etc. For the nRF52840 DK, it means connected to the PC using the USB connection on the short edge of the DK.

By the way, I notice that you set a flag for garbage collection in the function. That is OK if this is a function that is expected to be called seldom. However, if it is expected to be called often, then I recommend retrieving file system statistics through fds_stat(), and only flag for garbage collection if you do not have enough contiguous space for storing new entries (largest_contig) and there is enough space available to be freed (freeable_words). Alternatively, change the approach to run garbage collection if there is not enough space to write the record, then retry writing the record.

Regards,
Terje

[mazis]

Hello!

In the example that I'm using as a guide, when the data is read it is stored in a variable that isn't initialized, so I tried to code the same (similar). Thast's why I don't initialize my array.

To have a maximum number of credentials was exactly what I was trying to avoid. I preffer to have a dynamic array. What about using malloc() cautiously? Is it as bad as I have read? I found online linked lists, I could try that way too...

As you say there is a flag for garbage collecion and, in fact, that funcion is called seldom. Anyway, I wil try to implement the fds_stat() so I call for garbage collection only where it is needed.

Thank you so much!

Cheers!

[tesc]

Hi,

Especially on an embedded system, you will have a maximum number of credentials no matter what, because RAM is limited. At some point you will reach max, and either the heap is full (so the malloc will fail), or if you put it on the stack you get a stack overflow. There will also be a limit to how many credentials you can store in the FDS area of flash, depending on how many flash pages you set aside for FDS and what other data you need to store there. My point is, it may be better to put a max anyway, and that way have full control of maximum possible RAM and flash usage.

Also, are you required to have all of the credentials in RAM at the same time, or would it be acceptable to load just one at the time? With FDS, you can have multiple records with the same File ID and Record Key. That way when you get a new credential, you can store it in a new record, and when you read credentials you can read records sequentially from flash, always keeping one credential (or a limited number of credentials) in RAM.

Alternatively, you can use the same File ID for all credentials, but different Record Key. The Record Key can then act as an index. Then you can seach FDS records only using File ID when you need to look through all the credentials, and search using both File ID and Record Key for getting one particular credential.

Have a look at the FDS API documentation, where you find functions such as fds_record_find_by_key(), fds_record_find_in_file(), etc.

Regards,
Terje