How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[Samiulhaq]

Hello Torbjørn Sir, thanks for the reply.

ovrebekk said:

Are you doing both Android and iOS development, or just one of them? 

Currently, I am working on the android nRF Toolbox app once I implement it in Android then will maybe go for iOS. 

ovrebekk said:

We have various template applications available on Github to help you get started with app development for both of these platforms, as well as a BLE library that handles a lot of the basic Bluetooth setup for you. 

Yes Sir, I have downloaded a slightly lower version of the nRF Toolbox app from Github and successfully run it in the Android studio and it is working fine. Now I have to make little modifications(as I have mentioned earlier) in this app.

ovrebekk said:

The best place to start is probably the nRF Blinky application, available both for Android and iOS. It is much simpler than the Toolbox, allowing you more easily to build your own interface on top of it. 

Don't you think this will be a lengthy and tedious approach because? this app doesn't have a lot of functionality that I need for example ble-UART functionality through which I will receive the data sent by nRF DK and then I only have to store the data received via ble-uart in a CSV file. So in my opinion, the nRF Toolbox app is the best option to store with, and then I have to do a little modification of saving the UART data in a CSV file, What do you think?

Best Regards,

Sami

[ovrebekk]

Hi Sami

If what you are doing is very close to what the Toolbox app is already doing I agree it might be a better option to start there. 

Normally I find it easier to start with a simple example and build up, but in this case the more complex nRF Toolbox application seems to be a good fit

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, I hope you are fine.

Now I can store the data received via UART(of the nRF Toolbox app) in a CSV file, but the problem is that the receiving data lags a lot in the nRF Toolbox app(as I am confirming from the log in the Android studio). It means that let's say the nRF52 has sent the data/file in around 1-2 minutes but the app is receiving it very slowly (taking more than 10 minutes to finish the receiving) as I am confirming from the log in the Android studio. Even if I erase or power down the nRF DK after sending the data, the app still receives the data due to lagging. As I am storing the log data in a CSV file, therefore, storing the data in a CSV file in the app is not synced with the nRF52.

This lagging problem occurs when the data rate of sending the data (by nRF DK) is high (~1200kbps).

One other problem, the nRF Toolbox app got hanged on the high data rate(~1200kbps) and then I am not able to do anything with the app, what will be the reason in your opinion? 

Any help regarding this will be highly appreciated. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

How do you buffer the data on the phone side? 
If you're not doing it already you could try to store all the incoming data to a RAM buffer temporarily, and write it to a file from a separate thread, to make sure the two operations can run in parallel. 

Our apps guys haven't really done any high speed testing to verify how much data the apps can process, but personally I would expect 1200kbps to be manageable. The question is how much changes you have done to the app in order to implement things such as CSV support. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the reply.

ovrebekk said:

How do you buffer the data on the phone side? 

I am simply saving the log data coming from UART of the nRF Toolbox app in a CSV file using pure Kotlin. Everything is perfect and the data is saved in a CSV file but the coming log data(via UART) is lagging a lot. The nRF DK sent the complete file with ~1200kbps in only 1 minute(let's say), but on the app side, the data is coming very slowly (taking more than 10 minutes to receive the whole file) as I am seeing/confirming in the log of Android studio and the CSV file(generated in the phone storage), and the nRF Toolbox app got hanged and remains hanged until the complete data has been received. 

I thought the problem was in my modified app, but when I install a fresh new App from the play store and test it with the same example it hung also which I supposed to work fine. I don't know the reason and I am stuck with this problem for the last 1 week. I think maybe the problem is in the nRF DK or maybe in the mobile phone, I don't know. Your help will be highly appreciated. 

Best Regards,

Sami

 

[ovrebekk]

Hi Sami

Have you tried with more than one phone? 

If the issue can be reproduced with the standard nRF Toolbox app I can try to reproduce it here. I guess just sending a large file at maximum speed should be sufficient to reproduce it? 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the help. 

The problem has been solved temporarily by switching Off and ON the nRF52 chip, I don't know the reason for this problem but somehow it has been solved. 

Now I have merged this program(ble_UART) with the main project based on ble_app_hrs_rscs_relay_pca10056_s140( which acts both as a central & peripheral) to which I have interfaced different sensors. I have added the ble_nus and other related libraries and do some necessary changes to the main project but the program is stuck at line 184 of the main project which is line 18 in the below code snippet:

static void scan_init(void)
{
    ret_code_t          err_code;
    nrf_ble_scan_init_t init_scan;

    memset(&init_scan, 0, sizeof(init_scan));

    init_scan.p_scan_param = &m_scan_param;

    err_code = nrf_ble_scan_init(&m_scan, &init_scan, scan_evt_handler);
    APP_ERROR_CHECK(err_code);

    if (strlen(m_target_periph_name) != 0)
    {
        err_code = nrf_ble_scan_filter_set(&m_scan, 
                                           SCAN_NAME_FILTER, 
                                           m_target_periph_name);
        APP_ERROR_CHECK(err_code);
    }

    err_code = nrf_ble_scan_filter_set(&m_scan, 
                                       SCAN_UUID_FILTER, 
                                       &m_adv_uuids[HART_RATE_SERVICE_UUID_IDX]);
    APP_ERROR_CHECK(err_code);


    err_code = nrf_ble_scan_filters_enable(&m_scan, 
                                           NRF_BLE_SCAN_NAME_FILTER, 
                                           false);
    APP_ERROR_CHECK(err_code);

}

Showing the below error:

<error> app: ERROR 12 [NRF_ERROR_DATA_SIZE] at F:\nRF5_SDK_17.0.2_d674dde Sports Tracker\examples\sport tracker main\experimental 9\ble_app_hrs_rscs_relay\main.c:184
PC at: 0x000415EF
<error> app: End of error report

And if I comment out the above error lines(line 18 of the above code snippet) then the program is stuck at another point. I don't know what and where I am doing wrong. Has there been any example/thread/blog etc in which they have implemented the ble_uart functionality in the ble_app_hrs_rscs_relay example? Your help will be highly appreciated. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

If it reports the DATA_SIZE error when you try to set a scan filter on the name it could be that the name stored in m_target_periph_name is too long for the advertise packet. Could you let me know what the name is?

I made a relay example based on ble_app_uart and ble_app_uart_c a while back which you can find here, but I don't think there are any official or unofficial examples combining the hrs_rscs_relay and ble_app_uart examples, no. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the help. 

I have merged ble-nus to the hrs_rscs_relay(both central & peripheral), everything works fine, and I can see my device in the UART portion of the nRF Toolbox app(after adding ble-nus) and the nRF52 DK can connect to the nRF Toolbox or connect app. But the problem is that when I connect the Heart rate sensor to the nRF52 DK(both central & peripheral) then the nRF Toolbox or nRF Connect app is no longer able to connect to the nRF52 DK. 

In simple words when I am trying to connect nRF Toolbox or nRF Connect app to the nRF52 DK(both central & peripheral) before I connect the Heart Rate Sensor then the Apps are connecting very easily, but when I connect the Heart Rate Sensor first to the nRF52DK  then the Apps are no longer able to connect to the DK.

The heart rate sensor can connect at any time whether the DK is connected to App or not, but the App is not able to connect to the DK when there is a heart rate sensor already connected. Can you please help me with what I am doing wrong?  

I am facing this problem after adding ble-nus functionality in the hrs_rscs_relay project. Before adding the ble-nus, the device was advertising(and scanning of course)  but with another UUID which was not compatible with the UART functionality of the nRF Toolbox app, and the device name was only visible in the nRF Connect app and the nRF connect app was easily connectable to the DK irrespective of the heart rate sensor. But after adding the nus functionality and changing the advertising UUID of DK, the device is now visible in the nRF Toolbox app as well, but now I am facing a connectivity problem after connecting the heart rate sensor. 

Best Regards,

Sami

[Samiulhaq]

## UPDATE

Seems like I have found the reason for this behavior. When I Change the following setting in the config.h file then this problem is not occurring. 

change    NRF_SDH_BLE_GATT_MAX_MTU_SIZE           from 247 to 243

                NRF_SDH_BLE_GAP_EVENT_LENGTH            from 400 to 6

                NRF_SDH_BLE_GAP_DATA_LENGTH               from 251 to 27 

and          BLE_GATT_ATT_MTU_DEFAULT     from 247 to   23

After doing these changes in the Sdk_config.h file, I no longer face the connecting issue to both the Heart Rate sensor and Mobile App. 

Now the problem is that I want to use a larger MTU (247), an Event length of 400, and a Data Lenght of 251 to increase the throughput. But When I am trying with these settings for high throughput then I am facing the connectivity issue as described in the previous reply. I am totally confused about what to do, your help will be highly appreciated. 

Best Regards,

Sami