How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi Sami

Samiulhaq said:

In the Wireshark/sniffer trace, the last packet has the same data as the actual file. From this, I am concluding that all the data has been sent/received correctly and there is no packet loss, is this the correct way of checking it?  

That is correct. The BLE link layer will automatically retransmit any packet that is lost due to interference, and as long as the link doesn't break down completely you are guaranteed that all the packets make it to the other side intact, and in the same order as they were uploaded. So packet loss in this case only means packets lost on air, not actual data loss.
In a way you can think of a BLE connection as a TCP/IP connection, where data integrity is guaranteed at the cost of data throughput. 

When this happens you can see from the trace that the next payload will have the same data as the previous, and that the SN and NESN values are the same. This is normally an indication of packet loss over the air leading to a retransmit of the same data. 

Samiulhaq said:

This is a very good point but is it possible to compress the file using nRF DK? because I have to store the data of the different sensors(connected to the nRF52) in the SD card in the form of a CSV file and from there send that file(automatically) to the mobile app after every 60 hours.

The nRF52 can certainly do compression, as long as you have some memory to spare for the algorithms. 

Looking at your data from the trace it appears to be ASCII coded values, 17x 16-bit values pluss one timestamp pr record if I am not mistaken?

If this is correct you should be able to store 6 of these records in a 244 byte payload if you encode it to binary. 34 bytes for the 17 16-bit values plus 4 bytes for the timestamp multiplied by 6 equals 228 bytes. Since the binary values are fixed size you don't need to waste any bytes for delimiters or end of record symbols. 

I think this change alone should yield a reduction in the data to transfer of around 3 times (looking at the traces it seems you get around 2 records through pr packet). 

Alternatively you could try to find some compression algorithm that you apply to your file before you send it, such as zip or LZ4. 
The advantage here is that you don't have to manually parse the strings and convert them to binary, but the drawback is that you won't be able to store the entire record (or the entire compressed record) in memory so you probably need to break the compression into smaller steps, which will reduce the efficiency. 

I would also expect the flash/RAM impact to be higher with such an algorithm, compared to a simple ASCII -> binary conversion. 

Samiulhaq said:

I have reduced the conn. interval to 20ms in my project and now I can achieve an average throughput of about 1200kbps while sending 22MB files and now the throughput is not varying too much as I have reduced the conn. interval and my throughput are now about 1200kbps and I am ok what that.

Good to hear that you found a good compromise on the connection interval. 

Samiulhaq said:

How can I check whether the complete file has been received correctly or not? In my opinion, I have to do a small modification in the nRF Toolbox app and add the functionality of storing the data received via UART in a CSV file, and then compare it to the actual file, is this the efficient way?  Have you any other suggestions/alternatives about this? 

Possibly the best solution is to calculate a checksum over the entire file as it is transmitted, and then transmit the checksum at the end. The the app can do the same thing as it is receiving the file and compare the checksum generated on the receiving end to the one sent by the nRF at the end. If the checksum matches you know that the entire transaction went fine. 

Technically this should not be necessary as the link layer will make sure all the data is received, like I mentioned earlier, but it is still possible for application errors to lead to loss of data if there is some error in the buffer handling etc. 

Are you only planning on using the nRF Toolbox app, or will you eventually make a custom application?

Samiulhaq said:

Once Again thank you so much for your valuable time, explanation, and suggestions. I have learned a lot from you, to be honest. 

You welcome, I am glad you found my answers helpful. That's what they pay me for

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thank you so much for these great explanations and suggestions. 

Now everything is clear to me, but I am probably not going to the compression side at this time as per the company CEO's suggestion, if we need the compression then for sure I will generate another ticket for that. 

ovrebekk said:

Are you only planning on using the nRF Toolbox app, or will you eventually make a custom application?

Yes, I will be using the nRF Toolbox app with a little modification of saving the received data (via UART) in a CSV file and then will send that CSV file to the database for synchronization. This is the task of the App developer but currently, we don't have the app developer, so I have to do that too. I don't know how to do it but I have to start working on that. What you will suggest to me for that as I am new to app development?.

Once again thank you so much and Best regards,

Sami

[ovrebekk]

Hi

Samiulhaq said:

Now everything is clear to me, but I am probably not going to the compression side at this time as per the company CEO's suggestion, if we need the compression then for sure I will generate another ticket for that. 

Fair enough. If you do go that route I think simply converting to binary is the best option. It should be relatively simple to make a parser on each side to convert from ASCII to binary, and back again. 

Samiulhaq said:

This is the task of the App developer but currently, we don't have the app developer, so I have to do that too. I don't know how to do it but I have to start working on that. What you will suggest to me for that as I am new to app development?.

Are you doing both Android and iOS development, or just one of them? 

We have various template applications available on Github to help you get started with app development for both of these platforms, as well as a BLE library that handles a lot of the basic Bluetooth setup for you. 

The best place to start is probably the nRF Blinky application, available both for Android and iOS. It is much simpler than the Toolbox, allowing you more easily to build your own interface on top of it. 

Both of these are native applications, written in Java for Android and Swift for iOS. There are some initiatives such as Flutter intended to unify app development for the two platforms, but the Bluetooth support in Flutter is a bit rudimentary unfortunately, so most of our examples will only work on one or the other. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn Sir, thanks for the reply.

ovrebekk said:

Are you doing both Android and iOS development, or just one of them? 

Currently, I am working on the android nRF Toolbox app once I implement it in Android then will maybe go for iOS. 

ovrebekk said:

We have various template applications available on Github to help you get started with app development for both of these platforms, as well as a BLE library that handles a lot of the basic Bluetooth setup for you. 

Yes Sir, I have downloaded a slightly lower version of the nRF Toolbox app from Github and successfully run it in the Android studio and it is working fine. Now I have to make little modifications(as I have mentioned earlier) in this app.

ovrebekk said:

The best place to start is probably the nRF Blinky application, available both for Android and iOS. It is much simpler than the Toolbox, allowing you more easily to build your own interface on top of it. 

Don't you think this will be a lengthy and tedious approach because? this app doesn't have a lot of functionality that I need for example ble-UART functionality through which I will receive the data sent by nRF DK and then I only have to store the data received via ble-uart in a CSV file. So in my opinion, the nRF Toolbox app is the best option to store with, and then I have to do a little modification of saving the UART data in a CSV file, What do you think?

Best Regards,

Sami

[ovrebekk]

Hi Sami

If what you are doing is very close to what the Toolbox app is already doing I agree it might be a better option to start there. 

Normally I find it easier to start with a simple example and build up, but in this case the more complex nRF Toolbox application seems to be a good fit

Best regards
Torbjørn