How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[Samiulhaq]

Hi Torbjørn, I hope you are fine.

Now I can store the data received via UART(of the nRF Toolbox app) in a CSV file, but the problem is that the receiving data lags a lot in the nRF Toolbox app(as I am confirming from the log in the Android studio). It means that let's say the nRF52 has sent the data/file in around 1-2 minutes but the app is receiving it very slowly (taking more than 10 minutes to finish the receiving) as I am confirming from the log in the Android studio. Even if I erase or power down the nRF DK after sending the data, the app still receives the data due to lagging. As I am storing the log data in a CSV file, therefore, storing the data in a CSV file in the app is not synced with the nRF52.

This lagging problem occurs when the data rate of sending the data (by nRF DK) is high (~1200kbps).

One other problem, the nRF Toolbox app got hanged on the high data rate(~1200kbps) and then I am not able to do anything with the app, what will be the reason in your opinion? 

Any help regarding this will be highly appreciated. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

How do you buffer the data on the phone side? 
If you're not doing it already you could try to store all the incoming data to a RAM buffer temporarily, and write it to a file from a separate thread, to make sure the two operations can run in parallel. 

Our apps guys haven't really done any high speed testing to verify how much data the apps can process, but personally I would expect 1200kbps to be manageable. The question is how much changes you have done to the app in order to implement things such as CSV support. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the reply.

ovrebekk said:

How do you buffer the data on the phone side? 

I am simply saving the log data coming from UART of the nRF Toolbox app in a CSV file using pure Kotlin. Everything is perfect and the data is saved in a CSV file but the coming log data(via UART) is lagging a lot. The nRF DK sent the complete file with ~1200kbps in only 1 minute(let's say), but on the app side, the data is coming very slowly (taking more than 10 minutes to receive the whole file) as I am seeing/confirming in the log of Android studio and the CSV file(generated in the phone storage), and the nRF Toolbox app got hanged and remains hanged until the complete data has been received. 

I thought the problem was in my modified app, but when I install a fresh new App from the play store and test it with the same example it hung also which I supposed to work fine. I don't know the reason and I am stuck with this problem for the last 1 week. I think maybe the problem is in the nRF DK or maybe in the mobile phone, I don't know. Your help will be highly appreciated. 

Best Regards,

Sami

 

[ovrebekk]

Hi Sami

Have you tried with more than one phone? 

If the issue can be reproduced with the standard nRF Toolbox app I can try to reproduce it here. I guess just sending a large file at maximum speed should be sufficient to reproduce it? 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the help. 

The problem has been solved temporarily by switching Off and ON the nRF52 chip, I don't know the reason for this problem but somehow it has been solved. 

Now I have merged this program(ble_UART) with the main project based on ble_app_hrs_rscs_relay_pca10056_s140( which acts both as a central & peripheral) to which I have interfaced different sensors. I have added the ble_nus and other related libraries and do some necessary changes to the main project but the program is stuck at line 184 of the main project which is line 18 in the below code snippet:

static void scan_init(void)
{
    ret_code_t          err_code;
    nrf_ble_scan_init_t init_scan;

    memset(&init_scan, 0, sizeof(init_scan));

    init_scan.p_scan_param = &m_scan_param;

    err_code = nrf_ble_scan_init(&m_scan, &init_scan, scan_evt_handler);
    APP_ERROR_CHECK(err_code);

    if (strlen(m_target_periph_name) != 0)
    {
        err_code = nrf_ble_scan_filter_set(&m_scan, 
                                           SCAN_NAME_FILTER, 
                                           m_target_periph_name);
        APP_ERROR_CHECK(err_code);
    }

    err_code = nrf_ble_scan_filter_set(&m_scan, 
                                       SCAN_UUID_FILTER, 
                                       &m_adv_uuids[HART_RATE_SERVICE_UUID_IDX]);
    APP_ERROR_CHECK(err_code);


    err_code = nrf_ble_scan_filters_enable(&m_scan, 
                                           NRF_BLE_SCAN_NAME_FILTER, 
                                           false);
    APP_ERROR_CHECK(err_code);

}

Showing the below error:

<error> app: ERROR 12 [NRF_ERROR_DATA_SIZE] at F:\nRF5_SDK_17.0.2_d674dde Sports Tracker\examples\sport tracker main\experimental 9\ble_app_hrs_rscs_relay\main.c:184
PC at: 0x000415EF
<error> app: End of error report

And if I comment out the above error lines(line 18 of the above code snippet) then the program is stuck at another point. I don't know what and where I am doing wrong. Has there been any example/thread/blog etc in which they have implemented the ble_uart functionality in the ble_app_hrs_rscs_relay example? Your help will be highly appreciated. 

Best Regards,

Sami