How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[Samiulhaq]

Hey Torbjørn, thanks for the reply.

ovrebekk said:

What kind of data transfer speeds are you getting now?

I am able to send 2284066 bytes in about 30 seconds. 

ovrebekk said:

Can you check what the packet size of the BLE packets is?

Yes, it is 244 bytes. 

ovrebekk said:

Do you know the connection parameters for the connection to the phone, and whether or not you have exchanged larger MTU?

How can I check the connection parameter? using nRF_Sniffer ? How can I check for a larger MTU? MTU size in the sdk_config.h is as follows:

          #define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

and event length is:

        #define NRF_SDH_BLE_GAP_EVENT_LENGTH 400

ovrebekk said:

If you want to request a specific phy after connection establishment you can do it using the sd_ble_gap_phy_update(..) function, like shown here. 

Where can I use these lines in my code? in the case BLE_GAP_EVT_CONNECTED:?

Can you please test the project on your side? if yes, I will send you the complete project and you can try it on your side and then let me know about the throughput. If the throughput is not good please do the necessary modification in the project to achieve the highest possible throughput. I will be very grateful to you. 

I have attached a sniffer file to this reply, kindly check that too. 

Best Regards,

Sami

New_Sniffer_file.pcapng

[ovrebekk]

Hi Sami

Samiulhaq said:

I am able to send 2284066 bytes in about 30 seconds.

Again, this is not bad. If my math is correct this is more than 600kbps, which is very close to the maximum limit when running at 1M phy (the maximum limit is somewhere around 760kbps). 

Samiulhaq said:

How can I check the connection parameter? using nRF_Sniffer ?

Yes, you can find the connection interval in the sniffer trace by looking for the LL_CONNECTION_UPDATE_IND packets. There are several of these in the trace, but it is the last one that is the important one. If you look at packets 874 and 877 in the trace you will see that the slave requests a connection interval of 50ms in packet 874, and the central accepts this request in packet 877. 

This implies that your project is configured to ask for a 50ms connection interval, which is not optimal for throughput. If you change this to 15ms you might be able to get even closer to the theoretical maximum. 

Samiulhaq said:

How can I check for a larger MTU? MTU size in the sdk_config.h is as follows:

Looking at the trace, and your throughput numbers, it is clear that the MTU is correctly configured. 

Samiulhaq said:

Where can I use these lines in my code? in the case BLE_GAP_EVT_CONNECTED:?

Correct. 

Samiulhaq said:

Can you please test the project on your side?

At the moment I don't think this is necessary. I believe you should be able to achieve better throughput by changing the connection interval and phy, I doubt there is much else you can do. 

By the way, did you confirm which phone you are using for the testing? 

As I mentioned earlier many phones will be slower than what you see with this phone, so it is important to not have too high expectations. BLE was never really designed for high throughput after all.

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thanks for the reply.

ovrebekk said:

Again, this is not bad. If my math is correct this is more than 600kbps, which is very close to the maximum limit when running at 1M phy (the maximum limit is somewhere around 760kbps). 

How can I check whether the communication is using 1M PHY or 2M? I want to use 2M PHY, how can I enable it, or maybe I have enabled it already but my phone is not supporting 2M PHY, how to check that? 

 

ovrebekk said:

This implies that your project is configured to ask for a 50ms connection interval, which is not optimal for throughput. If you change this to 15ms you might be able to get even closer to the theoretical maximum. 

When I am trying to reduce the connection interval then I am facing a lot of empty packets which reduce the throughput, what will be a problem in your opinion? 

ovrebekk said:

By the way, did you confirm which phone you are using for the testing? 

I have tested on 03 phones, Vivo, Samsung, and Poco. My personal phone Vivo Y51s (Android version 12) does not support 2M PHY I guess. The sniffer trace I have shared with you is from the Vivo phone. The Samsung I guess is supporting 2M PHY but the throughput is very low compared to the Vivo phone. And the POCO one is very fast, I have tested it on my own project( the one with serial operations of reading from SD card and sending to BLE) and I have achieved a decent throughput of about2284066 bytes in 23-25 seconds. But now it is not giving a good throughput (even worse than the Vivo with 1M PHY). I don't know the reason.  That's why I want you to test it on your side and do the necessary modifications ( like connection interval, 2M PHY, etc). 

Thanks & Best Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

How can I check whether the communication is using 1M PHY or 2M? I want to use 2M PHY, how can I enable it, or maybe I have enabled it already but my phone is not supporting 2M PHY, how to check that? 

To try and enable 2M phy you can run the following code after the connection is established:

ble_gap_phys_t gap_phys_settings; 
gap_phys_settings.tx_phys = BLE_GAP_PHY_2MBPS;  
gap_phys_settings.rx_phys = BLE_GAP_PHY_2MBPS;         
sd_ble_gap_phy_update(m_conn_handle, &gap_phys_settings);

In order to see which phy you actually get you have to wait for the BLE_GAP_EVT_PHY_UPDATE event to be forwarded to the application. If the phone supports this event will tell you. 

Just to show which phy was enabled you can log the status of this event like this:

case BLE_GAP_EVT_PHY_UPDATE:
    NRF_LOG_INFO("Phy update: TX Phy %i, RX Phy %i", \
          (int)p_ble_evt->evt.gap_evt.params.phy_update.tx_phy, \
          (int)p_ble_evt->evt.gap_evt.params.phy_update.rx_phy);
    break;

What the phy values mean can be found in ble_gap.h:

/**@defgroup BLE_GAP_PHYS GAP PHYs
 * @{ */
#define BLE_GAP_PHY_AUTO                         0x00    /**< Automatic PHY selection. Refer @ref sd_ble_gap_phy_update for more information.*/
#define BLE_GAP_PHY_1MBPS                        0x01    /**< 1 Mbps PHY. */
#define BLE_GAP_PHY_2MBPS                        0x02    /**< 2 Mbps PHY. */
#define BLE_GAP_PHY_CODED                        0x04    /**< Coded PHY. */
#define BLE_GAP_PHY_NOT_SET                      0xFF    /**< PHY is not configured. */

Another way to check if your phone supports 2M phy is to install the nRF Connect app, and open the Device Information tab. There you will get an overview of which optional BLE features are supported or not. 

Samiulhaq said:

When I am trying to reduce the connection interval then I am facing a lot of empty packets which reduce the throughput, what will be a problem in your opinion? 

This sounds very odd. Does this happen on all the phones, or only on some of them? 
How small is the connection interval in this case?

Samiulhaq said:

That's why I want you to test it on your side and do the necessary modifications ( like connection interval, 2M PHY, etc). 

I can do some testing later in the week, but I will be limited by the phones I have available. I can test it with one of the high end Samsung phones and ensure that there is no bottle neck on the nRF side at least, and make sure that 2M is enabled, but optimizing across many different phones I will have to leave with you ;)

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjørn, thanks for the thorough explanation. 

ovrebekk said:

To try and enable 2M phy you can run the following code after the connection is established:

Yes, I can enable 2M PHY in the case BLE_GAP_EVT_CONNECTED: of the ble_evt_handler() with the help of the following lines:

            ble_gap_phys_t const phys =
            {
                 .rx_phys = BLE_GAP_PHY_2MBPS,
                 .tx_phys = BLE_GAP_PHY_2MBPS,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);

I can confirm that I have switched to the 2M PHY from the sniffer trace and the log. Is this the correct way to enable 2M PHY?

When I am using 1M PHY I can send the 2284066 bytes in about 26-27 seconds which is quite good but the problem is that when I switched to 2M PHY then the rate of data transfer(throughput) decreased a lot although I assumed a high throughput in 2M PHY but it is just the opposite of my expectation. What do you think will be the reason for the small throughput in 2M PHY than 1M? 

ovrebekk said:

This sounds very odd. Does this happen on all the phones, or only on some of them? 
How small is the connection interval in this case?

I have tested on three different phones and almost every phone has this behavior(decreasing conn. interval leads to more empty packets). when the conn. interval is 40-50 then the number of empty packets is the least but when I decrease the conn. interval then I am facing empty packets which leads to small throughput and when I increase the conn. interval then the throughput decreases( I don't know why).

ovrebekk said:

I can do some testing later in the week, but I will be limited by the phones I have available. I can test it with one of the high end Samsung phones and ensure that there is no bottle neck on the nRF side at least, and make sure that 2M is enabled, but optimizing across many different phones I will have to leave with you ;)

Yes please, do a test on a high-speed phone on your side and let me know about the throughput. If you want me to send you my project which has the SD card functionality then let me know I will share it with you and you can test it on your side with a good phone. If the throughput is small then kindly do some necessary modifications ( like conn. interval, 2M PHY, MTU size, Event_length, etc) in the projects and sent me back. I will be very thank full to you. 

Best Regards,

Sami