How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[Samiulhaq]

** updates:

seems like when I send something from the app then the nRF52 DK starts sending data. I want the nRF52 DK to start sending data to the app whenever it is connected to the app without waiting for the app to send a notification to it, is it possible? 

This is not a problem, the problem is where to read from the SD card and save it in temporary storage and then send it to the app. If you add these functionalities it will be very helpful for me.

Best Regards,

Sami

[ovrebekk]

Hi Sami

Essentially all you have to do is to replace the simulated_dummy_data_read(..) function that I implemented with a function that actually reads from the file using the f_read(..) function. 

I see in your code snippet that the test_queue_write() function is more or less doing this, the only difference is that you need to provide the buffer and length as an argument, and you should not write to a queue inside the function, but rather use the dummy_buffer_write(..) function like I do in the example. 

Samiulhaq said:

I want the nRF52 DK to start sending data to the app whenever it is connected to the app without waiting for the app to send a notification to it, is it possible? 

This is possible, yes. Just keep in mind that you should not try to send data right after connection, since it will take some time for the client to perform service discovery and enable notifications in the NUS service. 

I would recommend waiting for the BLE_NUS_EVT_COMM_STARTED event to be forwarded through the nus_data_handler(..) in main, before starting a data upload. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjorn, thanks for the reply. 

ovrebekk said:

Essentially all you have to do is to replace the simulated_dummy_data_read(..) function that I implemented with a function that actually reads from the file using the f_read(..) function. 

Yes, I have implemented this logic. Now I am able to read from the SD card and send it to the Mobile App.

Kindly check the below code snippet as I have  changed something in it:

// Create a FIFO structure
app_fifo_t dummy_fifo;

uint32_t dummy_fifo_bytes_used;

// Create a buffer for the FIFO
#define DUMMY_FIFO_SIZE 512
uint8_t dummy_buffer[DUMMY_FIFO_SIZE];

static int dummy_buffer_write(uint8_t *data, uint32_t len)
{
    app_fifo_write(&dummy_fifo, data, &len);
    dummy_fifo_bytes_used += len;
    return len;
}

static int dummy_buffer_bytes_free(void)
{
    return DUMMY_FIFO_SIZE - dummy_fifo_bytes_used;
}

static int dummy_buffer_read(uint8_t *data, uint32_t len)
{
    app_fifo_read(&dummy_fifo, data, &len);
    dummy_fifo_bytes_used -= len;
    return len;
}

static int dummy_buffer_bytes_used(void)
{
    return dummy_fifo_bytes_used;
}

static void init_dummy_buffer(void)
{
    // Initialize FIFO structure
    uint32_t err_code = app_fifo_init(&dummy_fifo, dummy_buffer, DUMMY_FIFO_SIZE);
    APP_ERROR_CHECK(err_code);

    dummy_fifo_bytes_used = 0;
}


#define TEST_DUMP_SIZE  size                       // Size of the file in the SD card
#define TEST_READ_SIZE  BLE_NUS_MAX_DATA_LEN       // BLE_NUS_MAX_DATA_LEN  =244
#define TEST_WRITE_SIZE BLE_NUS_MAX_DATA_LEN       //m_ble_nus_max_data_len
uint32_t dummy_test_remaining_bytes_read = 0;
uint32_t dummy_test_remaining_bytes_write = 0;

/**@brief Application main function.
 */
int main(void)
{
    bool erase_bonds;
    uint32_t err_code;

    // Initialize.
    uart_init();
    log_init();
    timers_init();
    buttons_leds_init(&erase_bonds);
    power_management_init();
     fatfs_init();
    ble_stack_init();
    gap_params_init();
    gatt_init();
    services_init();
    advertising_init();
    conn_params_init();

    init_dummy_buffer();

    // Start execution.
    //printf("\r\NRF_ERROR_RESOURCES test started.\r\n");
    NRF_LOG_INFO("Debug logging for UART over RTT started.");
    advertising_start();

    static uint8_t tmp_read_buffer[TEST_READ_SIZE];
    static uint8_t tmp_write_buffer[BLE_NUS_MAX_DATA_LEN];
    uint16_t write_length;
    int bt_written;

    // Enter main loop.
    for (;;)
    {
        // If the run_dummy_test flag is set, start a new data dump test
        if(run_dummy_test) 
        {
            run_dummy_test = false;
            bt_written = 0;
            dummy_test_remaining_bytes_read = TEST_DUMP_SIZE;
        }

        // As long as there are bytes left to read, and there is room in the dummy buffer, read out a single packet
        if(dummy_test_remaining_bytes_read && dummy_buffer_bytes_free() >= TEST_READ_SIZE)
        {
            // Set the read_length to the minimum of the remaining bytes to read and the test read size
            uint32_t read_length = (dummy_test_remaining_bytes_read > TEST_READ_SIZE) ? TEST_READ_SIZE : dummy_test_remaining_bytes_read;

            // Read SD card data into a temporary array
            ff_result = f_read(&file, tmp_read_buffer, sizeof(tmp_read_buffer), (UINT *) &bytes_read);

            // Move the SD card data from the temporary array into our FIFO buffer
            dummy_buffer_write(tmp_read_buffer, read_length);

            // Reduce the remaining number of bytes to read for the test
            dummy_test_remaining_bytes_read -= read_length;

            NRF_LOG_DEBUG("Dummy data produced %i, remaining %i", read_length, dummy_test_remaining_bytes_read);
        }

        // As long as there is data in the dummy buffer, and the Bluetooth stack has free buffers, upload a packet to the Bluetooth stack
        // If retransmit_previous_buffer is set it means the packet stored in tmp_write_buffer still hasn't been successfully
        // sent (because of NRF_ERROR_RESOURCES). In this case we try to send it again, without reading new data from the FIFO
        if(ble_buffers_available && (dummy_buffer_bytes_used() > 0 || retransmit_previous_buffer))
        {
            bsp_board_led_off(0);

            // Read new data from the FIFO unless retransmit_previous_buffer is set, in which case we have to retransmit the last packet
            if(!retransmit_previous_buffer)
            {
                // Move a new packet from the FIFO to the temporary write buffer
                write_length = dummy_buffer_read(tmp_write_buffer, TEST_WRITE_SIZE);

                NRF_LOG_DEBUG(" Fifo to tmp %i bytes", write_length);
            } 

            // Forward a single packet to the Bluetooth stack
            // NOTE: This code assumes that write_length will not be changed by ble_nus_data_send(..) (this could happen if you try to send a packet
            //       longer than the negotiated MTU size). If this could happen then the code needs to be changed to ensure that the remaining data gets
            //       written afterwards. 
            err_code = ble_nus_data_send(&m_nus, tmp_write_buffer, &write_length, m_conn_handle);
            if(err_code == NRF_SUCCESS)
            {
                // In case of success, update the bt_written parameter (please note the data will still take some time to reach the Bluetooth client)
                bt_written += write_length;

                NRF_LOG_INFO("   Tmp to BT total: %i (+%i)", bt_written, write_length);
            }
            else if(err_code == NRF_ERROR_RESOURCES) 
            {
                // In case of NRF_ERROR_RESOURCES, clear the ble_buffers_available flag to avoid trying to send more data until the Bluetooth buffers
                // clear up. This flag will be cleared by the BLE_GATTS_EVT_HVN_TX_COMPLETE event in the ble_evt_handler function in main.c
                ble_buffers_available = false;

                bsp_board_led_on(0);

                NRF_LOG_DEBUG("     ERROR RESOURCES");
            }
            else if(err_code != NRF_ERROR_INVALID_STATE && err_code != NRF_ERROR_NOT_FOUND)
            {
                APP_ERROR_CHECK(err_code);
            }
            
            // The retransmit_previous_buffer should be cleared for any upload. 
            // If NRF_ERROR_RESOURCES were to happen again it should be set again in the ble_evt_handler
            retransmit_previous_buffer = false;

            if(bt_written == size) 
            {
                // Any other code that should be added upon completion can be added here. 
                NRF_LOG_INFO("Test complete!");
            }
        }

        idle_state_handle();
    }
}

 

The problem is that the throughput is decreased actually, Maybe it's the mobile issue. Have you tested it? if yes,  how much throughput have you achieved? If not, Can you please test it for finding the throughput? I will be very thankful.

One more thing, where do you mention that using PHY= 2MBPS? See the following: 

        case BLE_GAP_EVT_PHY_UPDATE_REQUEST:
        {
            NRF_LOG_DEBUG("PHY update request.");
            ble_gap_phys_t const phys =
            {
                .rx_phys = BLE_GAP_PHY_AUTO,
                .tx_phys = BLE_GAP_PHY_AUTO,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);
        } break;

Here it is not mentioned to use PHY= 2MBPS. How to enable 2MBPS PHY?

Can I enable PHY=2MBPS in the case BLE_GAP_EVT_CONNECTED:? Like below:

        case BLE_GAP_EVT_CONNECTED:
            NRF_LOG_INFO("Connected");
            ble_gap_phys_t const phys =
            {
                 .rx_phys = BLE_GAP_PHY_2MBPS,
                 .tx_phys = BLE_GAP_PHY_2MBPS,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);

            err_code = bsp_indication_set(BSP_INDICATE_CONNECTED);
            APP_ERROR_CHECK(err_code);
            m_conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
            err_code = nrf_ble_qwr_conn_handle_assign(&m_qwr, m_conn_handle);
            APP_ERROR_CHECK(err_code);
            break;

Best Regards,

Sami 

[ovrebekk]

Hi Sami

What kind of data transfer speeds are you getting now?

Can you check what the packet size of the BLE packets is?

Do you know the connection parameters for the connection to the phone, and whether or not you have exchanged larger MTU?

The PHY_UPDATE_REQUEST code you link to is setting the RX and TX PHY to auto, which means it will accept any phy that the peer device requests. 

If you want to request a specific phy after connection establishment you can do it using the sd_ble_gap_phy_update(..) function, like shown here. 

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjørn, thanks for the reply.

ovrebekk said:

What kind of data transfer speeds are you getting now?

I am able to send 2284066 bytes in about 30 seconds. 

ovrebekk said:

Can you check what the packet size of the BLE packets is?

Yes, it is 244 bytes. 

ovrebekk said:

Do you know the connection parameters for the connection to the phone, and whether or not you have exchanged larger MTU?

How can I check the connection parameter? using nRF_Sniffer ? How can I check for a larger MTU? MTU size in the sdk_config.h is as follows:

          #define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

and event length is:

        #define NRF_SDH_BLE_GAP_EVENT_LENGTH 400

ovrebekk said:

If you want to request a specific phy after connection establishment you can do it using the sd_ble_gap_phy_update(..) function, like shown here. 

Where can I use these lines in my code? in the case BLE_GAP_EVT_CONNECTED:?

Can you please test the project on your side? if yes, I will send you the complete project and you can try it on your side and then let me know about the throughput. If the throughput is not good please do the necessary modification in the project to achieve the highest possible throughput. I will be very grateful to you. 

I have attached a sniffer file to this reply, kindly check that too. 

Best Regards,

Sami

New_Sniffer_file.pcapng

[ovrebekk]

Hi Sami

Samiulhaq said:

I am able to send 2284066 bytes in about 30 seconds.

Again, this is not bad. If my math is correct this is more than 600kbps, which is very close to the maximum limit when running at 1M phy (the maximum limit is somewhere around 760kbps). 

Samiulhaq said:

How can I check the connection parameter? using nRF_Sniffer ?

Yes, you can find the connection interval in the sniffer trace by looking for the LL_CONNECTION_UPDATE_IND packets. There are several of these in the trace, but it is the last one that is the important one. If you look at packets 874 and 877 in the trace you will see that the slave requests a connection interval of 50ms in packet 874, and the central accepts this request in packet 877. 

This implies that your project is configured to ask for a 50ms connection interval, which is not optimal for throughput. If you change this to 15ms you might be able to get even closer to the theoretical maximum. 

Samiulhaq said:

How can I check for a larger MTU? MTU size in the sdk_config.h is as follows:

Looking at the trace, and your throughput numbers, it is clear that the MTU is correctly configured. 

Samiulhaq said:

Where can I use these lines in my code? in the case BLE_GAP_EVT_CONNECTED:?

Correct. 

Samiulhaq said:

Can you please test the project on your side?

At the moment I don't think this is necessary. I believe you should be able to achieve better throughput by changing the connection interval and phy, I doubt there is much else you can do. 

By the way, did you confirm which phone you are using for the testing? 

As I mentioned earlier many phones will be slower than what you see with this phone, so it is important to not have too high expectations. BLE was never really designed for high throughput after all.

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thanks for the reply.

ovrebekk said:

Again, this is not bad. If my math is correct this is more than 600kbps, which is very close to the maximum limit when running at 1M phy (the maximum limit is somewhere around 760kbps). 

How can I check whether the communication is using 1M PHY or 2M? I want to use 2M PHY, how can I enable it, or maybe I have enabled it already but my phone is not supporting 2M PHY, how to check that? 

 

ovrebekk said:

This implies that your project is configured to ask for a 50ms connection interval, which is not optimal for throughput. If you change this to 15ms you might be able to get even closer to the theoretical maximum. 

When I am trying to reduce the connection interval then I am facing a lot of empty packets which reduce the throughput, what will be a problem in your opinion? 

ovrebekk said:

By the way, did you confirm which phone you are using for the testing? 

I have tested on 03 phones, Vivo, Samsung, and Poco. My personal phone Vivo Y51s (Android version 12) does not support 2M PHY I guess. The sniffer trace I have shared with you is from the Vivo phone. The Samsung I guess is supporting 2M PHY but the throughput is very low compared to the Vivo phone. And the POCO one is very fast, I have tested it on my own project( the one with serial operations of reading from SD card and sending to BLE) and I have achieved a decent throughput of about2284066 bytes in 23-25 seconds. But now it is not giving a good throughput (even worse than the Vivo with 1M PHY). I don't know the reason.  That's why I want you to test it on your side and do the necessary modifications ( like connection interval, 2M PHY, etc). 

Thanks & Best Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

How can I check whether the communication is using 1M PHY or 2M? I want to use 2M PHY, how can I enable it, or maybe I have enabled it already but my phone is not supporting 2M PHY, how to check that? 

To try and enable 2M phy you can run the following code after the connection is established:

ble_gap_phys_t gap_phys_settings; 
gap_phys_settings.tx_phys = BLE_GAP_PHY_2MBPS;  
gap_phys_settings.rx_phys = BLE_GAP_PHY_2MBPS;         
sd_ble_gap_phy_update(m_conn_handle, &gap_phys_settings);

In order to see which phy you actually get you have to wait for the BLE_GAP_EVT_PHY_UPDATE event to be forwarded to the application. If the phone supports this event will tell you. 

Just to show which phy was enabled you can log the status of this event like this:

case BLE_GAP_EVT_PHY_UPDATE:
    NRF_LOG_INFO("Phy update: TX Phy %i, RX Phy %i", \
          (int)p_ble_evt->evt.gap_evt.params.phy_update.tx_phy, \
          (int)p_ble_evt->evt.gap_evt.params.phy_update.rx_phy);
    break;

What the phy values mean can be found in ble_gap.h:

/**@defgroup BLE_GAP_PHYS GAP PHYs
 * @{ */
#define BLE_GAP_PHY_AUTO                         0x00    /**< Automatic PHY selection. Refer @ref sd_ble_gap_phy_update for more information.*/
#define BLE_GAP_PHY_1MBPS                        0x01    /**< 1 Mbps PHY. */
#define BLE_GAP_PHY_2MBPS                        0x02    /**< 2 Mbps PHY. */
#define BLE_GAP_PHY_CODED                        0x04    /**< Coded PHY. */
#define BLE_GAP_PHY_NOT_SET                      0xFF    /**< PHY is not configured. */

Another way to check if your phone supports 2M phy is to install the nRF Connect app, and open the Device Information tab. There you will get an overview of which optional BLE features are supported or not. 

Samiulhaq said:

When I am trying to reduce the connection interval then I am facing a lot of empty packets which reduce the throughput, what will be a problem in your opinion? 

This sounds very odd. Does this happen on all the phones, or only on some of them? 
How small is the connection interval in this case?

Samiulhaq said:

That's why I want you to test it on your side and do the necessary modifications ( like connection interval, 2M PHY, etc). 

I can do some testing later in the week, but I will be limited by the phones I have available. I can test it with one of the high end Samsung phones and ensure that there is no bottle neck on the nRF side at least, and make sure that 2M is enabled, but optimizing across many different phones I will have to leave with you ;)

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjørn, thanks for the thorough explanation. 

ovrebekk said:

To try and enable 2M phy you can run the following code after the connection is established:

Yes, I can enable 2M PHY in the case BLE_GAP_EVT_CONNECTED: of the ble_evt_handler() with the help of the following lines:

            ble_gap_phys_t const phys =
            {
                 .rx_phys = BLE_GAP_PHY_2MBPS,
                 .tx_phys = BLE_GAP_PHY_2MBPS,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);

I can confirm that I have switched to the 2M PHY from the sniffer trace and the log. Is this the correct way to enable 2M PHY?

When I am using 1M PHY I can send the 2284066 bytes in about 26-27 seconds which is quite good but the problem is that when I switched to 2M PHY then the rate of data transfer(throughput) decreased a lot although I assumed a high throughput in 2M PHY but it is just the opposite of my expectation. What do you think will be the reason for the small throughput in 2M PHY than 1M? 

ovrebekk said:

This sounds very odd. Does this happen on all the phones, or only on some of them? 
How small is the connection interval in this case?

I have tested on three different phones and almost every phone has this behavior(decreasing conn. interval leads to more empty packets). when the conn. interval is 40-50 then the number of empty packets is the least but when I decrease the conn. interval then I am facing empty packets which leads to small throughput and when I increase the conn. interval then the throughput decreases( I don't know why).

ovrebekk said:

I can do some testing later in the week, but I will be limited by the phones I have available. I can test it with one of the high end Samsung phones and ensure that there is no bottle neck on the nRF side at least, and make sure that 2M is enabled, but optimizing across many different phones I will have to leave with you ;)

Yes please, do a test on a high-speed phone on your side and let me know about the throughput. If you want me to send you my project which has the SD card functionality then let me know I will share it with you and you can test it on your side with a good phone. If the throughput is small then kindly do some necessary modifications ( like conn. interval, 2M PHY, MTU size, Event_length, etc) in the projects and sent me back. I will be very thank full to you. 

Best Regards,

Sami

[Samiulhaq]

Hi Torbjørn,

**UPDATE:

Samiulhaq said:

I have tested on three different phones and almost every phone has this behavior(decreasing conn. interval leads to more empty packets). when the conn. interval is 40-50 then the number of empty packets is the least but when I decrease the conn. interval then I am facing empty packets which leads to small throughput and when I increase the conn. interval then the throughput decreases( I don't know why).

I have reduced the connection interval and it increases the throughput(in the phone which supports 2M PHY).

Now I am the problem of "When I am using 1M PHY I can send the 2284066 bytes in about 26-27 seconds which is quite good but the problem is that when I switched to 2M PHY then the rate of data transfer(throughput) decreased a lot although I assumed a high throughput in 2M PHY but it is just the opposite of my expectation. What do you think will be the reason for the small throughput in 2M PHY than 1M"?  This happens only in this project (the one you have provided).

What do you think about this?

Best Regards,

Sami