How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[ovrebekk]

Hi 

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

In other words you will have to split up the transaction into smaller chunks, yes. 

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjørn, thanks for the quick help. 

ovrebekk said:

Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

Okay, I understand, and thanks for the thorough explanation. 

ovrebekk said:

The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

Thanks & kind Regards,

Sami

[Samiulhaq]

Hi Torbjorn

Updates;

I change the code a little bit and now the programming is not sticking,

My approach is:

1) Check if (! nrf_queue_is_full(&m_queue)), then do read from SD card & writing in queue until the queue get filled

2) check if (!nrf_queue_is_empty(&m_queue)), then start read from queue until

                queued = nrf_queue_utilization_get(&m_queue) >=244;

and do send the data to the mobile app (calling ble_nus_data_send()) until

                 err_code== NRF_ERROR_BUSY

But the ble_nus_data_send() is not called after every nrf_queue_read(). Can you please help me with that?

see the below code snippet:

    for (;;)
    {

        if(file_send_to_peripheral)
        {

           uint32_t remaining_bytes = size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;
           static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
           uint32_t j=1;

            if(size > 0)
            {

            while(remaining_bytes>0)
             {

            if(! nrf_queue_is_full(&m_queue))

               {
                while (!nrf_queue_is_full(&m_queue))
                  {
                    test_queue_write();  // Writing to queue
                  }
                }

            if(!nrf_queue_is_empty(&m_queue))
               {
                 while(queued >= 244)
                {
                   err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); // Reading from queue

                    if ( (err_code != NRF_SUCCESS) &&(err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
                        (err_code != NRF_ERROR_BUSY) &&
                        (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING) )
                      {
                        APP_ERROR_CHECK(err_code);
                      }
                      queued = nrf_queue_utilization_get(&m_queue);
                      NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                       j++;

                      // sending to app
                    do{
                    err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
                    if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                    {
                    APP_ERROR_CHECK(err_code);

                       remaining_bytes -= chunk_length;
                    if(remaining_bytes < chunk_length)
                    {
                     chunk_length = remaining_bytes;
                    }
                     NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                    }
                    }while(err_code== NRF_ERROR_BUSY);
                   
                    }
                    }
                  }
              }

            file_send_to_peripheral = false;
        }
        idle_state_handle();
    }
}

But the ble_nus_data_send() is not calling after every read from the queue(nrf_queue_read()) as you can see in the below attached log image. 

Best Regards,

Best Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

I don't know about that, how can I measure it?

The simple way (if you have an oscilloscope or logic analyzer available) is to set a GPIO high when you start reading the flash, and set it low after. Then you just measure the length of the pulse using your scope or logic analyzer. 

Samiulhaq said:

Should I call ble_nus_data_send() again after BLE_GATTS_EVT_HVN_TX_COMPLET?

Yes. The approach I recommend is to set a flag once the NRF_ERROR_RESOURCES occurs, and check this flag in the code before you try to call ble_nus_data_send() in your loop. If the flag is set you know that the Bluetooth stack is busy, and you should not read any more data from the buffer or try to send a new packet. 

When the TX_COMPLETE event occurs you can clear this flag, such that the next time you run your loop you will read out a new packet from the queue and call ble_nus_data_send() again. 

Reading through your code I think maybe the while(err_code ==  NRF_ERROR_BUSY) line should be changed to while(err_code == NRF_ERROR_RESOURCES) ?

The problem now is that you will run nrf_queue_read(..) over and over when the Bluetooth stack is busy transmitting data, which means all this data will be lost. If you make the change I mentioned above I believe your code makes more sense. 

Still, I have some comments on it:

1) Checking for the NRF_ERROR_BUSY, BLE_ERROR_GATTS_SYS_ATTR_MISSING, NRF_ERROR_INVALID_STATE errors and more after you run nrf_queue_read(..) doesn't make sense, as this function will not return these error codes. I am guessing you copied this if check from one of the Bluetooth calls, where these error codes might be returned. 

2) I would reconsider running two levels of while loops. It should be sufficient to run the one high level while loop where you check that remaining_bytes > 0. I don't think running the two other while loops helps with speed, since you will artificially serialize the flash read and Bluetooth transmit, rather than allow these two operations to occur in parallel. 

3) I still think a better handling of NRF_ERROR_RESOURCES would help. If I have some more time next week I will try to provide a more detailed flow chart, similar to how I handled this in the image transfer demo. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thanks for the thorough explanation. 

I have changed my code and added the GPIOs to track the time for reading from the SD card & writing to the queue, reading from the queue & sending to the mobile app. My approach;

1) check  if(!nrf_queue_is_full(&m_queue)) , read 25* BLE_NUS_MAX_DATA_LEN (25 * 244 = 6,100) bytes from SD card and write it to the queue. 

2) check if(!nrf_queue_is_empty(&m_queue)), read the whole bytes (6,100) from the queue at one flow, and then send it in chunks of 244 bytes to the mobile. 

These two operations will continue until the end of the file. Below is the code snippet, kindly go through it and let me know if something has to modify. 

#define MIN_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Minimum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define MAX_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Maximum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define FILE_SIZE_MAX                   25* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */

#define QUEUE_SIZE     FILE_SIZE_MAX                               
NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);

uint16_t bytes_read;
static uint8_t file_buffer[FILE_SIZE_MAX];

ff_result = f_open(&file, FILE_NAME, FA_READ);
size = (f_size(&file)); 
uint16_t arrLen=0;
 uint32_t err_code;
static void test_queue_write()
{
    arrLen = 0;
    ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
    err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
    APP_ERROR_CHECK(err_code);
    arrLen = bytes_read ;
}

static uint8_t data_out[FILE_SIZE_MAX] = {0};
uint32_t j=0;
static void test_queue_read()
{
    ret_code_t err_code;
    err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); 

   if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_RESOURCES))
      {
        APP_ERROR_CHECK(err_code);
        }
}

for (;;)
    {

    if(file_send_to_peripheral)
      {
        if(size > 0)
          {
           uint32_t remaining_bytes = size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;
           while(remaining_bytes>0)
                {
               if(!nrf_queue_is_full(&m_queue))

                   {
                    nrf_gpio_pin_set(SD_CARD_LED);                // set gpio
                    test_queue_write();  // Writing to queue
                    nrf_gpio_pin_clear(SD_CARD_LED);             // clear gpio
                   }

               if(!nrf_queue_is_empty(&m_queue))
                    {
                      nrf_gpio_pin_set(PACKET_LED);            // set gpio
                      test_queue_read();                        
                     
                      // sending to app
                            uint16_t rem_bytes = arrLen;
                            while((rem_bytes > 0))
                                  {
                                   err_code= ble_nus_data_send(&m_nus, &data_out[arrLen-rem_bytes], &chunk_length, m_conn_handle);
                                   if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                                      {
                                         APP_ERROR_CHECK(err_code);
                                         remaining_bytes -= chunk_length;
                                         rem_bytes -= chunk_length;
                                         if(remaining_bytes < chunk_length)
                                            {
                                             chunk_length = remaining_bytes;
                                            }
                                          NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                                      }
                                  }
                     nrf_gpio_pin_clear(PACKET_LED);         // clear gpio
                    } 
                }
          }

            file_send_to_peripheral = false;
      }
        idle_state_handle();
    }
}

I don't why both the GPIOs behave the same. I am attaching the logic analyzer and sniffer files to this reply, kindly check them and let me know whether the approach is correct or not. Also, give a brief about the behavior of GPIOs and sniffer file.

One other thing, when I change (Reduce) the Max. & Min. connection interval then there are more EMPTY PDU packets but the overall throughput is higher than with a large Min. & Max connection intervals with fewer EMPTY PDU packets. Your comment about that? How to decide and apply MIn. & Max. connection intervals?

Best Regards,

Sami

Sniffer_file.pcapng

Logic_Analyzer_file 

[ovrebekk]

Hi Sami

Thanks for sharing the traces, they show very clearly how much time you are spending on the data readout, and on the BLE transmission. 

The reason the two signals look like inverted versions of each other is the issue I mentioned earlier, where your program flow does not allow for the file readout and the BLE transmission to happen in parallel. 

Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

Looking at your traces I estimate that you would increase throughput by around 15-20% if you can get the two operations to run in parallel. 

To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

Still, it is probably not a dramatic change in throughput. Next week I might have more time to help out with a more optimized loop.

Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjorn, thanks for the thorough explanation. 

ovrebekk said:

Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

Yes, I understand that the while loop is waiting for 6100 bytes to send over ble, and then it goes back to the "read from SD card and save to the queue" operation, making the two operations run after one another. 

ovrebekk said:

To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example. It will be very beneficial if you provide me with a code snippet.

ovrebekk said:

Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

When I lower the connection intervals than 

#define MIN_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS) 
#define MAX_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS)

then I am facing more empty PDU packets, and when I increase connection intervals greater than 50 then the throughput decreases. 

ovrebekk said:

If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

 

Best Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example.

The nrf52-ble-image-transfer-demo I mentioned earlier follows this approach. 

The TX COMPLETE event is handled here. 

The implementation is a bit complicated unfortunately, in hindsight I should have optimized the ble_image_transfer_service.c implementation a bit to make it easier to understand how the data is sent. 

Samiulhaq said:

currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

What phone are you using for this test?

This is around 760kbps, which is actually quite good throughput when connected to a phone. Many phones will be limited to slower rates than this, including the iPhone which doesn't allow you to send BLE packets for more than 50% of the time. 

The best phones for throughput in my experience is the Samsung Android phones, which can go as high as ~1200kbps in some cases. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, I hope you are doing well.

Sir, I have tried my best but can't make the two operations execute in parallel. Can you please help me with the code snippet based on my code? I will be very grateful to you. I have checked the TX_COMPLETE event of the ble_image_transfer_demo example but can't understand its logic. 

Best Regards,

Sami

[ovrebekk]

Hi Sami

I spent some time today on an example for you. It is almost ready, but needs a bit of cleanup and testing before I can share it. 

I will do my best to get it to you by tomorrow. 

Best regards
Torbjørn

[Samiulhaq]

Hi Torbjorn Sir, thank you for the reply.

ovrebekk said:

I spent some time today on an example for you. It is almost ready, but needs a bit of cleanup and testing before I can share it. 

Thank you so much for your effort. Okay no problem, take time and send me whenever it is ready.

Once again thank you so much.

Best Regrds,

Sami

[ovrebekk]

Hi Sami

I have shared the project here:
https://github.com/too1/nrf52-ble-app-uart-error-resources

I guess the main takeaway is to keep the state machine as simple as possible, and make sure to run as much as possible of the application logic from the main context, rather than directly in the event handlers. 

Most of the code you can find in the for loop in main()

In order to test the demo you just have to connect to the device (using the nRF Connect mobile app for instance), enable notifications on the TX characteristic, and send anything on the RX characteristic (what you send doesn't matter). 

Best regards
Torbjørn