How to read a large file (About 240MB) from SD card

[Samiulhaq]

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

[Samiulhaq]

Hi Torbjorn

Updates;

I change the code a little bit and now the programming is not sticking,

My approach is:

1) Check if (! nrf_queue_is_full(&m_queue)), then do read from SD card & writing in queue until the queue get filled

2) check if (!nrf_queue_is_empty(&m_queue)), then start read from queue until

                queued = nrf_queue_utilization_get(&m_queue) >=244;

and do send the data to the mobile app (calling ble_nus_data_send()) until

                 err_code== NRF_ERROR_BUSY

But the ble_nus_data_send() is not called after every nrf_queue_read(). Can you please help me with that?

see the below code snippet:

    for (;;)
    {

        if(file_send_to_peripheral)
        {

           uint32_t remaining_bytes = size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;
           static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
           uint32_t j=1;

            if(size > 0)
            {

            while(remaining_bytes>0)
             {

            if(! nrf_queue_is_full(&m_queue))

               {
                while (!nrf_queue_is_full(&m_queue))
                  {
                    test_queue_write();  // Writing to queue
                  }
                }

            if(!nrf_queue_is_empty(&m_queue))
               {
                 while(queued >= 244)
                {
                   err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); // Reading from queue

                    if ( (err_code != NRF_SUCCESS) &&(err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
                        (err_code != NRF_ERROR_BUSY) &&
                        (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING) )
                      {
                        APP_ERROR_CHECK(err_code);
                      }
                      queued = nrf_queue_utilization_get(&m_queue);
                      NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                       j++;

                      // sending to app
                    do{
                    err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
                    if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                    {
                    APP_ERROR_CHECK(err_code);

                       remaining_bytes -= chunk_length;
                    if(remaining_bytes < chunk_length)
                    {
                     chunk_length = remaining_bytes;
                    }
                     NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                    }
                    }while(err_code== NRF_ERROR_BUSY);
                   
                    }
                    }
                  }
              }

            file_send_to_peripheral = false;
        }
        idle_state_handle();
    }
}

But the ble_nus_data_send() is not calling after every read from the queue(nrf_queue_read()) as you can see in the below attached log image. 

Best Regards,

Best Regards,

Sami

[ovrebekk]

Hi Sami

Samiulhaq said:

I don't know about that, how can I measure it?

The simple way (if you have an oscilloscope or logic analyzer available) is to set a GPIO high when you start reading the flash, and set it low after. Then you just measure the length of the pulse using your scope or logic analyzer. 

Samiulhaq said:

Should I call ble_nus_data_send() again after BLE_GATTS_EVT_HVN_TX_COMPLET?

Yes. The approach I recommend is to set a flag once the NRF_ERROR_RESOURCES occurs, and check this flag in the code before you try to call ble_nus_data_send() in your loop. If the flag is set you know that the Bluetooth stack is busy, and you should not read any more data from the buffer or try to send a new packet. 

When the TX_COMPLETE event occurs you can clear this flag, such that the next time you run your loop you will read out a new packet from the queue and call ble_nus_data_send() again. 

Reading through your code I think maybe the while(err_code ==  NRF_ERROR_BUSY) line should be changed to while(err_code == NRF_ERROR_RESOURCES) ?

The problem now is that you will run nrf_queue_read(..) over and over when the Bluetooth stack is busy transmitting data, which means all this data will be lost. If you make the change I mentioned above I believe your code makes more sense. 

Still, I have some comments on it:

1) Checking for the NRF_ERROR_BUSY, BLE_ERROR_GATTS_SYS_ATTR_MISSING, NRF_ERROR_INVALID_STATE errors and more after you run nrf_queue_read(..) doesn't make sense, as this function will not return these error codes. I am guessing you copied this if check from one of the Bluetooth calls, where these error codes might be returned. 

2) I would reconsider running two levels of while loops. It should be sufficient to run the one high level while loop where you check that remaining_bytes > 0. I don't think running the two other while loops helps with speed, since you will artificially serialize the flash read and Bluetooth transmit, rather than allow these two operations to occur in parallel. 

3) I still think a better handling of NRF_ERROR_RESOURCES would help. If I have some more time next week I will try to provide a more detailed flow chart, similar to how I handled this in the image transfer demo. 

Best regards
Torbjørn

[Samiulhaq]

Hello Torbjørn, thanks for the thorough explanation. 

I have changed my code and added the GPIOs to track the time for reading from the SD card & writing to the queue, reading from the queue & sending to the mobile app. My approach;

1) check  if(!nrf_queue_is_full(&m_queue)) , read 25* BLE_NUS_MAX_DATA_LEN (25 * 244 = 6,100) bytes from SD card and write it to the queue. 

2) check if(!nrf_queue_is_empty(&m_queue)), read the whole bytes (6,100) from the queue at one flow, and then send it in chunks of 244 bytes to the mobile. 

These two operations will continue until the end of the file. Below is the code snippet, kindly go through it and let me know if something has to modify. 

#define MIN_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Minimum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define MAX_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Maximum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
#define FILE_SIZE_MAX                   25* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */

#define QUEUE_SIZE     FILE_SIZE_MAX                               
NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);

uint16_t bytes_read;
static uint8_t file_buffer[FILE_SIZE_MAX];

ff_result = f_open(&file, FILE_NAME, FA_READ);
size = (f_size(&file)); 
uint16_t arrLen=0;
 uint32_t err_code;
static void test_queue_write()
{
    arrLen = 0;
    ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
    err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
    APP_ERROR_CHECK(err_code);
    arrLen = bytes_read ;
}

static uint8_t data_out[FILE_SIZE_MAX] = {0};
uint32_t j=0;
static void test_queue_read()
{
    ret_code_t err_code;
    err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); 

   if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_RESOURCES))
      {
        APP_ERROR_CHECK(err_code);
        }
}

for (;;)
    {

    if(file_send_to_peripheral)
      {
        if(size > 0)
          {
           uint32_t remaining_bytes = size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;
           while(remaining_bytes>0)
                {
               if(!nrf_queue_is_full(&m_queue))

                   {
                    nrf_gpio_pin_set(SD_CARD_LED);                // set gpio
                    test_queue_write();  // Writing to queue
                    nrf_gpio_pin_clear(SD_CARD_LED);             // clear gpio
                   }

               if(!nrf_queue_is_empty(&m_queue))
                    {
                      nrf_gpio_pin_set(PACKET_LED);            // set gpio
                      test_queue_read();                        
                     
                      // sending to app
                            uint16_t rem_bytes = arrLen;
                            while((rem_bytes > 0))
                                  {
                                   err_code= ble_nus_data_send(&m_nus, &data_out[arrLen-rem_bytes], &chunk_length, m_conn_handle);
                                   if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                                      {
                                         APP_ERROR_CHECK(err_code);
                                         remaining_bytes -= chunk_length;
                                         rem_bytes -= chunk_length;
                                         if(remaining_bytes < chunk_length)
                                            {
                                             chunk_length = remaining_bytes;
                                            }
                                          NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                                      }
                                  }
                     nrf_gpio_pin_clear(PACKET_LED);         // clear gpio
                    } 
                }
          }

            file_send_to_peripheral = false;
      }
        idle_state_handle();
    }
}

I don't why both the GPIOs behave the same. I am attaching the logic analyzer and sniffer files to this reply, kindly check them and let me know whether the approach is correct or not. Also, give a brief about the behavior of GPIOs and sniffer file.

One other thing, when I change (Reduce) the Max. & Min. connection interval then there are more EMPTY PDU packets but the overall throughput is higher than with a large Min. & Max connection intervals with fewer EMPTY PDU packets. Your comment about that? How to decide and apply MIn. & Max. connection intervals?

Best Regards,

Sami

Sniffer_file.pcapng

Logic_Analyzer_file 

[ovrebekk]

Hi Sami

Thanks for sharing the traces, they show very clearly how much time you are spending on the data readout, and on the BLE transmission. 

The reason the two signals look like inverted versions of each other is the issue I mentioned earlier, where your program flow does not allow for the file readout and the BLE transmission to happen in parallel. 

Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

Looking at your traces I estimate that you would increase throughput by around 15-20% if you can get the two operations to run in parallel. 

To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

Still, it is probably not a dramatic change in throughput. Next week I might have more time to help out with a more optimized loop.

Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

Best regards
Torbjørn

[Samiulhaq]

Hey Torbjorn, thanks for the thorough explanation. 

ovrebekk said:

Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

Yes, I understand that the while loop is waiting for 6100 bytes to send over ble, and then it goes back to the "read from SD card and save to the queue" operation, making the two operations run after one another. 

ovrebekk said:

To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example. It will be very beneficial if you provide me with a code snippet.

ovrebekk said:

Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

When I lower the connection intervals than 

#define MIN_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS) 
#define MAX_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS)

then I am facing more empty PDU packets, and when I increase connection intervals greater than 50 then the throughput decreases. 

ovrebekk said:

If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

 

Best Regards,

Sami