How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

In the Wireshark/sniffer trace, the last packet has the same data as the actual file. From this, I am concluding that all the data has been sent/received correctly and there is no packet loss, is this the correct way of checking it?  

That is correct. The BLE link layer will automatically retransmit any packet that is lost due to interference, and as long as the link doesn't break down completely you are guaranteed that all the packets make it to the other side intact, and in the same order as they were uploaded. So packet loss in this case only means packets lost on air, not actual data loss.
In a way you can think of a BLE connection as a TCP/IP connection, where data integrity is guaranteed at the cost of data throughput. 

When this happens you can see from the trace that the next payload will have the same data as the previous, and that the SN and NESN values are the same. This is normally an indication of packet loss over the air leading to a retransmit of the same data. 

Samiulhaq said:

This is a very good point but is it possible to compress the file using nRF DK? because I have to store the data of the different sensors(connected to the nRF52) in the SD card in the form of a CSV file and from there send that file(automatically) to the mobile app after every 60 hours.

The nRF52 can certainly do compression, as long as you have some memory to spare for the algorithms. 

Looking at your data from the trace it appears to be ASCII coded values, 17x 16-bit values pluss one timestamp pr record if I am not mistaken?

If this is correct you should be able to store 6 of these records in a 244 byte payload if you encode it to binary. 34 bytes for the 17 16-bit values plus 4 bytes for the timestamp multiplied by 6 equals 228 bytes. Since the binary values are fixed size you don't need to waste any bytes for delimiters or end of record symbols. 

I think this change alone should yield a reduction in the data to transfer of around 3 times (looking at the traces it seems you get around 2 records through pr packet). 

Alternatively you could try to find some compression algorithm that you apply to your file before you send it, such as zip or LZ4. 
The advantage here is that you don't have to manually parse the strings and convert them to binary, but the drawback is that you won't be able to store the entire record (or the entire compressed record) in memory so you probably need to break the compression into smaller steps, which will reduce the efficiency. 

I would also expect the flash/RAM impact to be higher with such an algorithm, compared to a simple ASCII -> binary conversion. 

Samiulhaq said:

I have reduced the conn. interval to 20ms in my project and now I can achieve an average throughput of about 1200kbps while sending 22MB files and now the throughput is not varying too much as I have reduced the conn. interval and my throughput are now about 1200kbps and I am ok what that.

Good to hear that you found a good compromise on the connection interval. 

Samiulhaq said:

How can I check whether the complete file has been received correctly or not? In my opinion, I have to do a small modification in the nRF Toolbox app and add the functionality of storing the data received via UART in a CSV file, and then compare it to the actual file, is this the efficient way?  Have you any other suggestions/alternatives about this? 

Possibly the best solution is to calculate a checksum over the entire file as it is transmitted, and then transmit the checksum at the end. The the app can do the same thing as it is receiving the file and compare the checksum generated on the receiving end to the one sent by the nRF at the end. If the checksum matches you know that the entire transaction went fine. 

Technically this should not be necessary as the link layer will make sure all the data is received, like I mentioned earlier, but it is still possible for application errors to lead to loss of data if there is some error in the buffer handling etc. 

Are you only planning on using the nRF Toolbox app, or will you eventually make a custom application?

Samiulhaq said:

Once Again thank you so much for your valuable time, explanation, and suggestions. I have learned a lot from you, to be honest. 

You welcome, I am glad you found my answers helpful. That's what they pay me for

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thank you so much for these great explanations and suggestions. 

Now everything is clear to me, but I am probably not going to the compression side at this time as per the company CEO's suggestion, if we need the compression then for sure I will generate another ticket for that. 

ovrebekk said:

Are you only planning on using the nRF Toolbox app, or will you eventually make a custom application?

Yes, I will be using the nRF Toolbox app with a little modification of saving the received data (via UART) in a CSV file and then will send that CSV file to the database for synchronization. This is the task of the App developer but currently, we don't have the app developer, so I have to do that too. I don't know how to do it but I have to start working on that. What you will suggest to me for that as I am new to app development?.

Once again thank you so much and Best regards,

Sami

[ovrebekk]

Hi

Samiulhaq said:

Now everything is clear to me, but I am probably not going to the compression side at this time as per the company CEO's suggestion, if we need the compression then for sure I will generate another ticket for that. 

Fair enough. If you do go that route I think simply converting to binary is the best option. It should be relatively simple to make a parser on each side to convert from ASCII to binary, and back again. 

Samiulhaq said:

This is the task of the App developer but currently, we don't have the app developer, so I have to do that too. I don't know how to do it but I have to start working on that. What you will suggest to me for that as I am new to app development?.

Are you doing both Android and iOS development, or just one of them? 

We have various template applications available on Github to help you get started with app development for both of these platforms, as well as a BLE library that handles a lot of the basic Bluetooth setup for you. 

The best place to start is probably the nRF Blinky application, available both for Android and iOS. It is much simpler than the Toolbox, allowing you more easily to build your own interface on top of it. 

Both of these are native applications, written in Java for Android and Swift for iOS. There are some initiatives such as Flutter intended to unify app development for the two platforms, but the Bluetooth support in Flutter is a bit rudimentary unfortunately, so most of our examples will only work on one or the other. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn Sir, thanks for the reply.

ovrebekk said:

Are you doing both Android and iOS development, or just one of them? 

Currently, I am working on the android nRF Toolbox app once I implement it in Android then will maybe go for iOS. 

ovrebekk said:

We have various template applications available on Github to help you get started with app development for both of these platforms, as well as a BLE library that handles a lot of the basic Bluetooth setup for you. 

Yes Sir, I have downloaded a slightly lower version of the nRF Toolbox app from Github and successfully run it in the Android studio and it is working fine. Now I have to make little modifications(as I have mentioned earlier) in this app.

ovrebekk said:

The best place to start is probably the nRF Blinky application, available both for Android and iOS. It is much simpler than the Toolbox, allowing you more easily to build your own interface on top of it. 

Don't you think this will be a lengthy and tedious approach because? this app doesn't have a lot of functionality that I need for example ble-UART functionality through which I will receive the data sent by nRF DK and then I only have to store the data received via ble-uart in a CSV file. So in my opinion, the nRF Toolbox app is the best option to store with, and then I have to do a little modification of saving the UART data in a CSV file, What do you think?

Best Regards,

Sami

[ovrebekk]

Hi Sami

If what you are doing is very close to what the Toolbox app is already doing I agree it might be a better option to start there. 

Normally I find it easier to start with a simple example and build up, but in this case the more complex nRF Toolbox application seems to be a good fit

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, I hope you are fine.

Now I can store the data received via UART(of the nRF Toolbox app) in a CSV file, but the problem is that the receiving data lags a lot in the nRF Toolbox app(as I am confirming from the log in the Android studio). It means that let's say the nRF52 has sent the data/file in around 1-2 minutes but the app is receiving it very slowly (taking more than 10 minutes to finish the receiving) as I am confirming from the log in the Android studio. Even if I erase or power down the nRF DK after sending the data, the app still receives the data due to lagging. As I am storing the log data in a CSV file, therefore, storing the data in a CSV file in the app is not synced with the nRF52.

This lagging problem occurs when the data rate of sending the data (by nRF DK) is high (~1200kbps).

One other problem, the nRF Toolbox app got hanged on the high data rate(~1200kbps) and then I am not able to do anything with the app, what will be the reason in your opinion? 

Any help regarding this will be highly appreciated. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

How do you buffer the data on the phone side? 
If you're not doing it already you could try to store all the incoming data to a RAM buffer temporarily, and write it to a file from a separate thread, to make sure the two operations can run in parallel. 

Our apps guys haven't really done any high speed testing to verify how much data the apps can process, but personally I would expect 1200kbps to be manageable. The question is how much changes you have done to the app in order to implement things such as CSV support. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the reply.

ovrebekk said:

How do you buffer the data on the phone side? 

I am simply saving the log data coming from UART of the nRF Toolbox app in a CSV file using pure Kotlin. Everything is perfect and the data is saved in a CSV file but the coming log data(via UART) is lagging a lot. The nRF DK sent the complete file with ~1200kbps in only 1 minute(let's say), but on the app side, the data is coming very slowly (taking more than 10 minutes to receive the whole file) as I am seeing/confirming in the log of Android studio and the CSV file(generated in the phone storage), and the nRF Toolbox app got hanged and remains hanged until the complete data has been received. 

I thought the problem was in my modified app, but when I install a fresh new App from the play store and test it with the same example it hung also which I supposed to work fine. I don't know the reason and I am stuck with this problem for the last 1 week. I think maybe the problem is in the nRF DK or maybe in the mobile phone, I don't know. Your help will be highly appreciated. 

Best Regards,

Sami

 

[ovrebekk]

Hi Sami

Have you tried with more than one phone? 

If the issue can be reproduced with the standard nRF Toolbox app I can try to reproduce it here. I guess just sending a large file at maximum speed should be sufficient to reproduce it? 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the help. 

The problem has been solved temporarily by switching Off and ON the nRF52 chip, I don't know the reason for this problem but somehow it has been solved. 

Now I have merged this program(ble_UART) with the main project based on ble_app_hrs_rscs_relay_pca10056_s140( which acts both as a central & peripheral) to which I have interfaced different sensors. I have added the ble_nus and other related libraries and do some necessary changes to the main project but the program is stuck at line 184 of the main project which is line 18 in the below code snippet:

static void scan_init(void)
{
    ret_code_t          err_code;
    nrf_ble_scan_init_t init_scan;

    memset(&init_scan, 0, sizeof(init_scan));

    init_scan.p_scan_param = &m_scan_param;

    err_code = nrf_ble_scan_init(&m_scan, &init_scan, scan_evt_handler);
    APP_ERROR_CHECK(err_code);

    if (strlen(m_target_periph_name) != 0)
    {
        err_code = nrf_ble_scan_filter_set(&m_scan, 
                                           SCAN_NAME_FILTER, 
                                           m_target_periph_name);
        APP_ERROR_CHECK(err_code);
    }

    err_code = nrf_ble_scan_filter_set(&m_scan, 
                                       SCAN_UUID_FILTER, 
                                       &m_adv_uuids[HART_RATE_SERVICE_UUID_IDX]);
    APP_ERROR_CHECK(err_code);


    err_code = nrf_ble_scan_filters_enable(&m_scan, 
                                           NRF_BLE_SCAN_NAME_FILTER, 
                                           false);
    APP_ERROR_CHECK(err_code);

}

Showing the below error:

<error> app: ERROR 12 [NRF_ERROR_DATA_SIZE] at F:\nRF5_SDK_17.0.2_d674dde Sports Tracker\examples\sport tracker main\experimental 9\ble_app_hrs_rscs_relay\main.c:184
PC at: 0x000415EF
<error> app: End of error report

And if I comment out the above error lines(line 18 of the above code snippet) then the program is stuck at another point. I don't know what and where I am doing wrong. Has there been any example/thread/blog etc in which they have implemented the ble_uart functionality in the ble_app_hrs_rscs_relay example? Your help will be highly appreciated. 

Best Regards,

Sami