How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[ovrebekk]

Hi Sami

Which driver or library are you using to read your SD card? 

Are you using a file system?

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the reply.

I am using the ff.c library (i don't know whether it is a file system or not) inside the ff.c it is written on the top that "FatFs - Generic FAT file system module  R0.12b", in which the read function is as follows;

              ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);

Best Regards,

Sami

[Samiulhaq]

Hi Torbjørn, thanks for the reply.

I am using the ff.c library (i don't know whether it is a file system or not) inside the ff.c it is written on the top that "FatFs - Generic FAT file system module  R0.12b", in which the read function is as follows;

              ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);

ovrebekk said:

and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Also, I don't understand this part in your first reply. Can you please explain and clarify it more? 

Best Regards,

Sami

[ovrebekk]

Hi Sami

Yes, ff.c is an implementation of the FatFS file system provided by Zephyr. If you are using this then you can easily read the file in smaller chunks, to avoid running out of memory. 

The third argument to the f_read(..) function allows you to set the maximum number of bytes that you want to read, and you can scale this after your buffers. 

Samiulhaq said:

Also, I don't understand this part in your first reply. Can you please explain and clarify it more? 

I guess the point I was trying to make is that you should try to implement the readout from flash and the BLE transmissions in parallel, so that you don't limit the upload speed by having to wait for a flash read for every BLE packet. 

One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

Best regards
Torbjørn