How to read a large file (About 240MB) from SD card

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARDgreater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

Parents
  • Hi 

    Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

    In other words you will have to split up the transaction into smaller chunks, yes. 

    The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

    Best regards
    Torbjørn

  • Hi Torbjørn, thanks for the quick help. 

    Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

    Okay, I understand, and thanks for the thorough explanation. 

    The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

    Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

    Thanks & kind Regards,

    Sami

  • Hi Sami

    Which driver or library are you using to read your SD card? 

    Are you using a file system?

    Best regards
    Torbjørn

  • Hi Torbjørn, thanks for the reply.

    I am using the ff.c library (i don't know whether it is a file system or not) inside the ff.c it is written on the top that "FatFs - Generic FAT file system module  R0.12b", in which the read function is as follows;

                  ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);

    Best Regards,

    Sami

  • Hi Torbjørn, thanks for the reply.

    I am using the ff.c library (i don't know whether it is a file system or not) inside the ff.c it is written on the top that "FatFs - Generic FAT file system module  R0.12b", in which the read function is as follows;

                  ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);

    and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

    Also, I don't understand this part in your first reply. Can you please explain and clarify it more? 

    Best Regards,

    Sami

  • Hi Sami

    Yes, ff.c is an implementation of the FatFS file system provided by Zephyr. If you are using this then you can easily read the file in smaller chunks, to avoid running out of memory. 

    The third argument to the f_read(..) function allows you to set the maximum number of bytes that you want to read, and you can scale this after your buffers. 

    Samiulhaq said:
    Also, I don't understand this part in your first reply. Can you please explain and clarify it more? 

    I guess the point I was trying to make is that you should try to implement the readout from flash and the BLE transmissions in parallel, so that you don't limit the upload speed by having to wait for a flash read for every BLE packet. 

    One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

    Best regards
    Torbjørn

Reply
  • Hi Sami

    Yes, ff.c is an implementation of the FatFS file system provided by Zephyr. If you are using this then you can easily read the file in smaller chunks, to avoid running out of memory. 

    The third argument to the f_read(..) function allows you to set the maximum number of bytes that you want to read, and you can scale this after your buffers. 

    Samiulhaq said:
    Also, I don't understand this part in your first reply. Can you please explain and clarify it more? 

    I guess the point I was trying to make is that you should try to implement the readout from flash and the BLE transmissions in parallel, so that you don't limit the upload speed by having to wait for a flash read for every BLE packet. 

    One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

    Best regards
    Torbjørn

Children
  • Hi Torbjørn, thanks for the reply. 

    Yes, ff.c is an implementation of the FatFS file system provided by Zephyr. If you are using this then you can easily read the file in smaller chunks, to avoid running out of memory. 

    The third argument to the f_read(..) function allows you to set the maximum number of bytes that you want to read, and you can scale this after your buffers. 

    Yes, I have successfully implemented that, I can read data from an SD card and send it to the mobile app continuously. Below is my approach, kindly check if it is efficient or not:

    #define FILE_SIZE_MAX                   BLE_NUS_MAX_DATA_LEN  // file size is same as the packet size
    static uint8_t file_buffer[FILE_SIZE_MAX];
    ff_result = f_open(&file, FILE_NAME, FA_READ);
    
    for (;;)
        {
            if(file_send_to_peripheral)
            {
    
               file_actual_read_size = (f_size(&file));
               uint32_t remaining_bytes = file_actual_read_size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
    
                if(file_actual_read_size > 0)
                {
    
                while(remaining_bytes > 0)
                 {
    
                 ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
                 err_code= ble_nus_data_send(&m_nus, file_buffer, &chunk_length, m_conn_handle);
    
                    if (err_code != NRF_ERROR_RESOURCES)
                    {
                        APP_ERROR_CHECK(err_code);
    
                       remaining_bytes -= chunk_length;
                       if(remaining_bytes < chunk_length)
                        {
                         chunk_length = remaining_bytes;
                        }
                        NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
    
                    }
    
                    }
                    }
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }

    One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

     

    Thanks for this best suggestion, but I haven't used any FIFO or queue before if you refer me to an example/thread/blog, this will be beneficial for me. I will be very grateful to you. 

    Thanks & Best Regards,

    Sami

  • Hello Torbjørn, I hope you will be fine. 

    Updates: I have implemented the nrf_queue as per your suggestion, below is my approach: 

    #define FILE_SIZE_MAX                   10* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */
    static uint8_t file_buffer[FILE_SIZE_MAX];
    
    ff_result = f_open(&file, FILE_NAME, FA_READ);
    size = (f_size(&file)); 
    
    NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);
    size_t queued;
    uint8_t i=1;
    static void test_queue_write()
    {
        uint32_t err_code;
    
        ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
        NRF_LOG_INFO("%d bytes read.", bytes_read);
        err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
        APP_ERROR_CHECK(err_code);
        queued = nrf_queue_utilization_get(&m_queue);
        NRF_LOG_INFO("Queued after %d write: %i",i, queued);
        i++;
    }
    
      for (;;)
        {
            if(file_send_to_peripheral)
            {
    
    
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
               uint8_t j=1;
    
                if(size > 0)
                {
    
                 while(! nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();
                      }
    
                while(remaining_bytes>0)
                 {
    
                 //ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
                 if(!nrf_queue_is_empty(&m_queue))
                 {
      
                    err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out));
    
                       if ( (err_code != NRF_SUCCESS) &&
                            (err_code != NRF_ERROR_INVALID_STATE) &&
                            (err_code != NRF_ERROR_RESOURCES) &&
                            (err_code != NRF_ERROR_BUSY) &&
                            (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING)
                           )
                       {
                         APP_ERROR_CHECK(err_code);
    
                         }
    
                     err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
    
                    if ((err_code != NRF_ERROR_INVALID_STATE) &&
                        (err_code != NRF_ERROR_RESOURCES) &&
                        (err_code != NRF_ERROR_NOT_FOUND))
                       {
                        APP_ERROR_CHECK(err_code);
    
                          remaining_bytes -= chunk_length;
                       if(remaining_bytes < chunk_length)
                         {
                         chunk_length = remaining_bytes;
                         }
                         NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                         //queued = nrf_queue_utilization_get(&m_queue);
                         //NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                         //j++;
    
                      }      
                      }
    
                    else
                     {
                     while(! nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();
                      }
                   
                       }
               }
             
               }
    
    
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }
     

    But it doesn't increase the throughput. The old approach(reading data from an SD card and sending it to App directly without using a Queue) and the new approach(queue involvement)  has almost the same throughput.

    Can you please check what I am doing wrong?

    Best Regards,

    Sami

  • Hi Sami

    What is the throughput that you are achieving, and what kind of phone are you connecting to?

    Do you know how long it takes to read a single packet from the flash? 

    I can't spot an issue in your flow that would explain why the throughput is no better. The critical thing is that you read more data from the flash in parallel while the Bluetooth stack is busy, but just looking at your code it seems like you should be able to do this. 

    Still I would recommend simplifying the flow a bit. Ideally you want a main loop that works something like this:

    while(more_bytes_to_send)
    {
        if(queue not full)
        {
            read_buffer_from_flash_and_store_in_queue();
        }
        if(queue not empty)
        {
            read_buffer_from_queue_and_write_to_nus();
        }
    }

    What complicates matters a bit is how to handle the RESOURCES error when you try to send a packet over BLE.
    When you get this error the last packet you tried to send has not been forwarded to the Bluetooth stack, which means the data stored in the data_out buffer (to reference your code) is not stored anywhere else. You need to call ble_nus_data_send(..) again, and make sure not to overwrite the data_out buffer unti the ble_nus_data_send(..) function returns NRF_SUCCESS, indicating that the data has been successfully forwarded to the stack. 

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    To assist you during the development of this mechanism I suggest setting up some GPIO toggling to measure when the MCU is busy reading data from the flash, and toggle a separate pin whenever a packet is forwarded to the SoftDevice. Then you can check these pins on a scope and try to get an overview of the scheduling and timing of these activities. Then it should be easier to understand why things are running too slow. 

    Best regards
    Torbjørn

  • Hi Torbjorn, 

    What is the throughput that you are achieving, and what kind of phone are you connecting to?

    I am able to send 2284066 bytes in about 40-45 seconds. I have tested on 2 phones, one is VIVO Y51s(Android version 12) and the other is Samsung Galaxy A30 (Android version 11).

    Do you know how long it takes to read a single packet from the flash? 

    I don't know about that, how can I measure it?

    Still I would recommend simplifying the flow a bit. Ideally you want a main loop that works something like this:

    I have implement this approach, but now the program got stuck after 1 write to the queue and 1 read from the queue as you can see in the attached log:

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    To assist you during the development of this mechanism I suggest setting up some GPIO toggling to measure when the MCU is busy reading data from the flash, and toggle a separate pin whenever a packet is forwarded to the SoftDevice. Then you can check these pins on a scope and try to get an overview of the scheduling and timing of these activities. Then it should be easier to understand why things are running too slow. 

    I am trying to implement your valuable suggestions and will let you know if I face any problems. 

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    Should I call ble_nus_data_send() again after BLE_GATTS_EVT_HVN_TX_COMPLET?

    Best Regards,

    Sami

  • Hi Torbjorn

    Updates;

    I change the code a little bit and now the programming is not sticking,

    My approach is:

    1) Check if (! nrf_queue_is_full(&m_queue)), then do read from SD card & writing in queue until the queue get filled

    2) check if (!nrf_queue_is_empty(&m_queue)), then start read from queue until

                    queued = nrf_queue_utilization_get(&m_queue) >=244;

    and do send the data to the mobile app (calling ble_nus_data_send()) until

                     err_code== NRF_ERROR_BUSY

    But the ble_nus_data_send() is not called after every nrf_queue_read(). Can you please help me with that?

    see the below code snippet:

        for (;;)
        {
    
            if(file_send_to_peripheral)
            {
    
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
               uint32_t j=1;
    
                if(size > 0)
                {
    
                while(remaining_bytes>0)
                 {
    
                if(! nrf_queue_is_full(&m_queue))
    
                   {
                    while (!nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();  // Writing to queue
                      }
                    }
    
                if(!nrf_queue_is_empty(&m_queue))
                   {
                     while(queued >= 244)
                    {
                       err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); // Reading from queue
    
                        if ( (err_code != NRF_SUCCESS) &&(err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
                            (err_code != NRF_ERROR_BUSY) &&
                            (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING) )
                          {
                            APP_ERROR_CHECK(err_code);
                          }
                          queued = nrf_queue_utilization_get(&m_queue);
                          NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                           j++;
    
                          // sending to app
                        do{
                        err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
                        if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                        {
                        APP_ERROR_CHECK(err_code);
    
                           remaining_bytes -= chunk_length;
                        if(remaining_bytes < chunk_length)
                        {
                         chunk_length = remaining_bytes;
                        }
                         NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                        }
                        }while(err_code== NRF_ERROR_BUSY);
                       
                        }
                        }
                      }
                  }
    
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }

    But the ble_nus_data_send() is not calling after every read from the queue(nrf_queue_read()) as you can see in the below attached log image. 

    Best Regards,

    Best Regards,

    Sami

Related