How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example.

The nrf52-ble-image-transfer-demo I mentioned earlier follows this approach. 

The TX COMPLETE event is handled here. 

The implementation is a bit complicated unfortunately, in hindsight I should have optimized the ble_image_transfer_service.c implementation a bit to make it easier to understand how the data is sent. 

Samiulhaq said:

currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

What phone are you using for this test?

This is around 760kbps, which is actually quite good throughput when connected to a phone. Many phones will be limited to slower rates than this, including the iPhone which doesn't allow you to send BLE packets for more than 50% of the time. 

The best phones for throughput in my experience is the Samsung Android phones, which can go as high as ~1200kbps in some cases. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, I hope you are doing well.

Sir, I have tried my best but can't make the two operations execute in parallel. Can you please help me with the code snippet based on my code? I will be very grateful to you. I have checked the TX_COMPLETE event of the ble_image_transfer_demo example but can't understand its logic. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

I spent some time today on an example for you. It is almost ready, but needs a bit of cleanup and testing before I can share it. 

I will do my best to get it to you by tomorrow. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjorn Sir, thank you for the reply.

ovrebekk said:

I spent some time today on an example for you. It is almost ready, but needs a bit of cleanup and testing before I can share it. 

Thank you so much for your effort. Okay no problem, take time and send me whenever it is ready.

Once again thank you so much.

Best Regrds,

Sami

[ovrebekk]

Hi Sami

I have shared the project here:
https://github.com/too1/nrf52-ble-app-uart-error-resources

I guess the main takeaway is to keep the state machine as simple as possible, and make sure to run as much as possible of the application logic from the main context, rather than directly in the event handlers. 

Most of the code you can find in the for loop in main()

In order to test the demo you just have to connect to the device (using the nRF Connect mobile app for instance), enable notifications on the TX characteristic, and send anything on the RX characteristic (what you send doesn't matter). 

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjørn, thanks for your effort and help.

I have tested the project, but it doesn't send anything to the mobile app and when I try to send something from the mobile app to the nRF52DK then I don't see anything in the log.

My main aim is reading data from a file in the SD card and sending it to the mobile but seems like you didn't add the SD card functionality. It is not a problem I will add the SD card functionality, but then reading from the SD card and saving it in temporary storage, and sending it to the mobile app will be a problem for me. Which temporary storage have you used in this project? queue or FIFO? Because according to you:

ovrebekk said:

I guess the point I was trying to make is that you should try to implement the readout from flash and the BLE transmissions in parallel, so that you don't limit the upload speed by having to wait for a flash read for every BLE packet. 

One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

I am totally confused about where to read data from the SD card and where to save it in the temporary storage and where to send it to the mobile app. 

Can you please make it something like this:

#define MIN_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Minimum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define MAX_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Maximum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define FILE_SIZE_MAX                   25* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */

#define QUEUE_SIZE     FILE_SIZE_MAX                               
NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);

uint16_t bytes_read;
static uint8_t file_buffer[FILE_SIZE_MAX];

ff_result = f_open(&file, FILE_NAME, FA_READ);
size = (f_size(&file)); 
uint16_t arrLen=0;
 uint32_t err_code;
static void test_queue_write()
{
    arrLen = 0;
    ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
    err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
    APP_ERROR_CHECK(err_code);
    arrLen = bytes_read ;
}

static uint8_t data_out[FILE_SIZE_MAX] = {0};
uint32_t j=0;
static void test_queue_read()
{
    ret_code_t err_code;
    err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); 

   if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_RESOURCES))
      {
        APP_ERROR_CHECK(err_code);
        }
}

for (;;)
    {

    if(file_send_to_peripheral)
      {
        if(size > 0)
          {
           uint32_t remaining_bytes = size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;
           while(remaining_bytes>0)
                {
               if(!nrf_queue_is_full(&m_queue))

                   {
                    nrf_gpio_pin_set(SD_CARD_LED);                // set gpio
                    test_queue_write();  // Writing to queue
                    nrf_gpio_pin_clear(SD_CARD_LED);             // clear gpio
                   }

               if(!nrf_queue_is_empty(&m_queue))
                    {
                      nrf_gpio_pin_set(PACKET_LED);            // set gpio
                      test_queue_read();                        
                     
                      // sending to app
                            uint16_t rem_bytes = arrLen;
                            while((rem_bytes > 0))
                                  {
                                   err_code= ble_nus_data_send(&m_nus, &data_out[arrLen-rem_bytes], &chunk_length, m_conn_handle);
                                   if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                                      {
                                         APP_ERROR_CHECK(err_code);
                                         remaining_bytes -= chunk_length;
                                         rem_bytes -= chunk_length;
                                         if(remaining_bytes < chunk_length)
                                            {
                                             chunk_length = remaining_bytes;
                                            }
                                          NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                                      }
                                  }
                     nrf_gpio_pin_clear(PACKET_LED);         // clear gpio
                    } 
                }
          }

            file_send_to_peripheral = false;
      }
        idle_state_handle();
    }
}

Best Regards,

Sami

[Samiulhaq]

** updates:

seems like when I send something from the app then the nRF52 DK starts sending data. I want the nRF52 DK to start sending data to the app whenever it is connected to the app without waiting for the app to send a notification to it, is it possible? 

This is not a problem, the problem is where to read from the SD card and save it in temporary storage and then send it to the app. If you add these functionalities it will be very helpful for me.

Best Regards,

Sami

[ovrebekk]

Hi Sami

Essentially all you have to do is to replace the simulated_dummy_data_read(..) function that I implemented with a function that actually reads from the file using the f_read(..) function. 

I see in your code snippet that the test_queue_write() function is more or less doing this, the only difference is that you need to provide the buffer and length as an argument, and you should not write to a queue inside the function, but rather use the dummy_buffer_write(..) function like I do in the example. 

Samiulhaq said:

I want the nRF52 DK to start sending data to the app whenever it is connected to the app without waiting for the app to send a notification to it, is it possible? 

This is possible, yes. Just keep in mind that you should not try to send data right after connection, since it will take some time for the client to perform service discovery and enable notifications in the NUS service. 

I would recommend waiting for the BLE_NUS_EVT_COMM_STARTED event to be forwarded through the nus_data_handler(..) in main, before starting a data upload. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjorn, thanks for the reply. 

ovrebekk said:

Essentially all you have to do is to replace the simulated_dummy_data_read(..) function that I implemented with a function that actually reads from the file using the f_read(..) function. 

Yes, I have implemented this logic. Now I am able to read from the SD card and send it to the Mobile App.

Kindly check the below code snippet as I have  changed something in it:

// Create a FIFO structure
app_fifo_t dummy_fifo;

uint32_t dummy_fifo_bytes_used;

// Create a buffer for the FIFO
#define DUMMY_FIFO_SIZE 512
uint8_t dummy_buffer[DUMMY_FIFO_SIZE];

static int dummy_buffer_write(uint8_t *data, uint32_t len)
{
    app_fifo_write(&dummy_fifo, data, &len);
    dummy_fifo_bytes_used += len;
    return len;
}

static int dummy_buffer_bytes_free(void)
{
    return DUMMY_FIFO_SIZE - dummy_fifo_bytes_used;
}

static int dummy_buffer_read(uint8_t *data, uint32_t len)
{
    app_fifo_read(&dummy_fifo, data, &len);
    dummy_fifo_bytes_used -= len;
    return len;
}

static int dummy_buffer_bytes_used(void)
{
    return dummy_fifo_bytes_used;
}

static void init_dummy_buffer(void)
{
    // Initialize FIFO structure
    uint32_t err_code = app_fifo_init(&dummy_fifo, dummy_buffer, DUMMY_FIFO_SIZE);
    APP_ERROR_CHECK(err_code);

    dummy_fifo_bytes_used = 0;
}


#define TEST_DUMP_SIZE  size                       // Size of the file in the SD card
#define TEST_READ_SIZE  BLE_NUS_MAX_DATA_LEN       // BLE_NUS_MAX_DATA_LEN  =244
#define TEST_WRITE_SIZE BLE_NUS_MAX_DATA_LEN       //m_ble_nus_max_data_len
uint32_t dummy_test_remaining_bytes_read = 0;
uint32_t dummy_test_remaining_bytes_write = 0;

/**@brief Application main function.
 */
int main(void)
{
    bool erase_bonds;
    uint32_t err_code;

    // Initialize.
    uart_init();
    log_init();
    timers_init();
    buttons_leds_init(&erase_bonds);
    power_management_init();
     fatfs_init();
    ble_stack_init();
    gap_params_init();
    gatt_init();
    services_init();
    advertising_init();
    conn_params_init();

    init_dummy_buffer();

    // Start execution.
    //printf("\r\NRF_ERROR_RESOURCES test started.\r\n");
    NRF_LOG_INFO("Debug logging for UART over RTT started.");
    advertising_start();

    static uint8_t tmp_read_buffer[TEST_READ_SIZE];
    static uint8_t tmp_write_buffer[BLE_NUS_MAX_DATA_LEN];
    uint16_t write_length;
    int bt_written;

    // Enter main loop.
    for (;;)
    {
        // If the run_dummy_test flag is set, start a new data dump test
        if(run_dummy_test) 
        {
            run_dummy_test = false;
            bt_written = 0;
            dummy_test_remaining_bytes_read = TEST_DUMP_SIZE;
        }

        // As long as there are bytes left to read, and there is room in the dummy buffer, read out a single packet
        if(dummy_test_remaining_bytes_read && dummy_buffer_bytes_free() >= TEST_READ_SIZE)
        {
            // Set the read_length to the minimum of the remaining bytes to read and the test read size
            uint32_t read_length = (dummy_test_remaining_bytes_read > TEST_READ_SIZE) ? TEST_READ_SIZE : dummy_test_remaining_bytes_read;

            // Read SD card data into a temporary array
            ff_result = f_read(&file, tmp_read_buffer, sizeof(tmp_read_buffer), (UINT *) &bytes_read);

            // Move the SD card data from the temporary array into our FIFO buffer
            dummy_buffer_write(tmp_read_buffer, read_length);

            // Reduce the remaining number of bytes to read for the test
            dummy_test_remaining_bytes_read -= read_length;

            NRF_LOG_DEBUG("Dummy data produced %i, remaining %i", read_length, dummy_test_remaining_bytes_read);
        }

        // As long as there is data in the dummy buffer, and the Bluetooth stack has free buffers, upload a packet to the Bluetooth stack
        // If retransmit_previous_buffer is set it means the packet stored in tmp_write_buffer still hasn't been successfully
        // sent (because of NRF_ERROR_RESOURCES). In this case we try to send it again, without reading new data from the FIFO
        if(ble_buffers_available && (dummy_buffer_bytes_used() > 0 || retransmit_previous_buffer))
        {
            bsp_board_led_off(0);

            // Read new data from the FIFO unless retransmit_previous_buffer is set, in which case we have to retransmit the last packet
            if(!retransmit_previous_buffer)
            {
                // Move a new packet from the FIFO to the temporary write buffer
                write_length = dummy_buffer_read(tmp_write_buffer, TEST_WRITE_SIZE);

                NRF_LOG_DEBUG(" Fifo to tmp %i bytes", write_length);
            } 

            // Forward a single packet to the Bluetooth stack
            // NOTE: This code assumes that write_length will not be changed by ble_nus_data_send(..) (this could happen if you try to send a packet
            //       longer than the negotiated MTU size). If this could happen then the code needs to be changed to ensure that the remaining data gets
            //       written afterwards. 
            err_code = ble_nus_data_send(&m_nus, tmp_write_buffer, &write_length, m_conn_handle);
            if(err_code == NRF_SUCCESS)
            {
                // In case of success, update the bt_written parameter (please note the data will still take some time to reach the Bluetooth client)
                bt_written += write_length;

                NRF_LOG_INFO("   Tmp to BT total: %i (+%i)", bt_written, write_length);
            }
            else if(err_code == NRF_ERROR_RESOURCES) 
            {
                // In case of NRF_ERROR_RESOURCES, clear the ble_buffers_available flag to avoid trying to send more data until the Bluetooth buffers
                // clear up. This flag will be cleared by the BLE_GATTS_EVT_HVN_TX_COMPLETE event in the ble_evt_handler function in main.c
                ble_buffers_available = false;

                bsp_board_led_on(0);

                NRF_LOG_DEBUG("     ERROR RESOURCES");
            }
            else if(err_code != NRF_ERROR_INVALID_STATE && err_code != NRF_ERROR_NOT_FOUND)
            {
                APP_ERROR_CHECK(err_code);
            }
            
            // The retransmit_previous_buffer should be cleared for any upload. 
            // If NRF_ERROR_RESOURCES were to happen again it should be set again in the ble_evt_handler
            retransmit_previous_buffer = false;

            if(bt_written == size) 
            {
                // Any other code that should be added upon completion can be added here. 
                NRF_LOG_INFO("Test complete!");
            }
        }

        idle_state_handle();
    }
}

 

The problem is that the throughput is decreased actually, Maybe it's the mobile issue. Have you tested it? if yes,  how much throughput have you achieved? If not, Can you please test it for finding the throughput? I will be very thankful.

One more thing, where do you mention that using PHY= 2MBPS? See the following: 

        case BLE_GAP_EVT_PHY_UPDATE_REQUEST:
        {
            NRF_LOG_DEBUG("PHY update request.");
            ble_gap_phys_t const phys =
            {
                .rx_phys = BLE_GAP_PHY_AUTO,
                .tx_phys = BLE_GAP_PHY_AUTO,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);
        } break;

Here it is not mentioned to use PHY= 2MBPS. How to enable 2MBPS PHY?

Can I enable PHY=2MBPS in the case BLE_GAP_EVT_CONNECTED:? Like below:

        case BLE_GAP_EVT_CONNECTED:
            NRF_LOG_INFO("Connected");
            ble_gap_phys_t const phys =
            {
                 .rx_phys = BLE_GAP_PHY_2MBPS,
                 .tx_phys = BLE_GAP_PHY_2MBPS,
            };
            err_code = sd_ble_gap_phy_update(p_ble_evt->evt.gap_evt.conn_handle, &phys);
            APP_ERROR_CHECK(err_code);

            err_code = bsp_indication_set(BSP_INDICATE_CONNECTED);
            APP_ERROR_CHECK(err_code);
            m_conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
            err_code = nrf_ble_qwr_conn_handle_assign(&m_qwr, m_conn_handle);
            APP_ERROR_CHECK(err_code);
            break;

Best Regards,

Sami 

[ovrebekk]

Hi Sami

What kind of data transfer speeds are you getting now?

Can you check what the packet size of the BLE packets is?

Do you know the connection parameters for the connection to the phone, and whether or not you have exchanged larger MTU?

The PHY_UPDATE_REQUEST code you link to is setting the RX and TX PHY to auto, which means it will accept any phy that the peer device requests. 

If you want to request a specific phy after connection establishment you can do it using the sd_ble_gap_phy_update(..) function, like shown here. 

Best regards
Torbjørn