How to read a large file (About 240MB) from SD card

Hello all, I hope you are doing well. 

I Have a project on NRF52840 in which I have multiple sensors connected to the NRF52 and the sensor data are storing in the SD card in the form of a CSV file of size 240 MB. I want to read this large file from the SD card and then send it to the mobile app using ble_periperhial. I have used This example and modified it, but the problem is that I can only read 220000 bytes from the SD card in the buffer at once and when I am trying to increase 

   #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARDgreater than 220000, while the following;

                                            static uint8_t file_buffer[FILE_SIZE_MAX]; 

                                           ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

                                      .bss is too large to fit in RAM1 memory segment
                                     .heap is too large to fit in RAM1 memory segment
                                      section .heap overlaps absolute placed section .stack
                                      section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA [0000000020002c64,0000000020224b08]   

                                               

I don't know about this error and how to solve it. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Any help regarding this will be highly appreciated.

Parents
  • Hi 

    Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

    In other words you will have to split up the transaction into smaller chunks, yes. 

    The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

    Best regards
    Torbjørn

  • Hi Torbjørn, thanks for the quick help. 

    Unfortunately you only have 256kB of RAM available in the nRF52840 device, and parts of this RAM will be needed by the rest of your application, which is probably why the code won't build if you make your buffer larger than 220000 bytes. 

    Okay, I understand, and thanks for the thorough explanation. 

    The most efficient solution would be to match the reads from your external memory to the size of your Bluetooth packets, and make sure that you buffer multiple BLE packets at once to make sure the BLE communication doesn't have to wait for the reads from external memory. 

    Can you please refer me to a ticket/blog etc where they have read from the SD card in chunks(my Bluetooth packet is 244 bytes)? I have tried different approaches but I am unable to read from an SD card in chunks. I don't know how to change the pointer to the next byte/line of the SD card data. When I read the first 220000 bytes( #define FILE_SIZE_MAX    220000) of data from a file on the SD card then I am unable to change the pointer to the next index( 220000+1th  byte). Can you please help me with that? 

    Thanks & kind Regards,

    Sami

  • Hi Torbjørn, thanks for the reply. 

    Yes, ff.c is an implementation of the FatFS file system provided by Zephyr. If you are using this then you can easily read the file in smaller chunks, to avoid running out of memory. 

    The third argument to the f_read(..) function allows you to set the maximum number of bytes that you want to read, and you can scale this after your buffers. 

    Yes, I have successfully implemented that, I can read data from an SD card and send it to the mobile app continuously. Below is my approach, kindly check if it is efficient or not:

    #define FILE_SIZE_MAX                   BLE_NUS_MAX_DATA_LEN  // file size is same as the packet size
    static uint8_t file_buffer[FILE_SIZE_MAX];
    ff_result = f_open(&file, FILE_NAME, FA_READ);
    
    for (;;)
        {
            if(file_send_to_peripheral)
            {
    
               file_actual_read_size = (f_size(&file));
               uint32_t remaining_bytes = file_actual_read_size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
    
                if(file_actual_read_size > 0)
                {
    
                while(remaining_bytes > 0)
                 {
    
                 ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
                 err_code= ble_nus_data_send(&m_nus, file_buffer, &chunk_length, m_conn_handle);
    
                    if (err_code != NRF_ERROR_RESOURCES)
                    {
                        APP_ERROR_CHECK(err_code);
    
                       remaining_bytes -= chunk_length;
                       if(remaining_bytes < chunk_length)
                        {
                         chunk_length = remaining_bytes;
                        }
                        NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
    
                    }
    
                    }
                    }
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }

    One way to do this is to use some kind of FIFO or queue in the application to store the data temporarily. Then you can have one thread that is constantly reading data from the flash and storing it in the FIFO, while you have another thread that will empty the FIFO and pass the data to the Bluetooth stack. As long as the FIFO is not full you can start a new flash read operation, and as long as the FIFO is not empty you can start a new Bluetooth transaction. 

     

    Thanks for this best suggestion, but I haven't used any FIFO or queue before if you refer me to an example/thread/blog, this will be beneficial for me. I will be very grateful to you. 

    Thanks & Best Regards,

    Sami

  • Hello Torbjørn, I hope you will be fine. 

    Updates: I have implemented the nrf_queue as per your suggestion, below is my approach: 

    #define FILE_SIZE_MAX                   10* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */
    static uint8_t file_buffer[FILE_SIZE_MAX];
    
    ff_result = f_open(&file, FILE_NAME, FA_READ);
    size = (f_size(&file)); 
    
    NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);
    size_t queued;
    uint8_t i=1;
    static void test_queue_write()
    {
        uint32_t err_code;
    
        ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
        NRF_LOG_INFO("%d bytes read.", bytes_read);
        err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
        APP_ERROR_CHECK(err_code);
        queued = nrf_queue_utilization_get(&m_queue);
        NRF_LOG_INFO("Queued after %d write: %i",i, queued);
        i++;
    }
    
      for (;;)
        {
            if(file_send_to_peripheral)
            {
    
    
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
               uint8_t j=1;
    
                if(size > 0)
                {
    
                 while(! nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();
                      }
    
                while(remaining_bytes>0)
                 {
    
                 //ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
                 if(!nrf_queue_is_empty(&m_queue))
                 {
      
                    err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out));
    
                       if ( (err_code != NRF_SUCCESS) &&
                            (err_code != NRF_ERROR_INVALID_STATE) &&
                            (err_code != NRF_ERROR_RESOURCES) &&
                            (err_code != NRF_ERROR_BUSY) &&
                            (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING)
                           )
                       {
                         APP_ERROR_CHECK(err_code);
    
                         }
    
                     err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
    
                    if ((err_code != NRF_ERROR_INVALID_STATE) &&
                        (err_code != NRF_ERROR_RESOURCES) &&
                        (err_code != NRF_ERROR_NOT_FOUND))
                       {
                        APP_ERROR_CHECK(err_code);
    
                          remaining_bytes -= chunk_length;
                       if(remaining_bytes < chunk_length)
                         {
                         chunk_length = remaining_bytes;
                         }
                         NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                         //queued = nrf_queue_utilization_get(&m_queue);
                         //NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                         //j++;
    
                      }      
                      }
    
                    else
                     {
                     while(! nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();
                      }
                   
                       }
               }
             
               }
    
    
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }
     

    But it doesn't increase the throughput. The old approach(reading data from an SD card and sending it to App directly without using a Queue) and the new approach(queue involvement)  has almost the same throughput.

    Can you please check what I am doing wrong?

    Best Regards,

    Sami

  • Hi Sami

    What is the throughput that you are achieving, and what kind of phone are you connecting to?

    Do you know how long it takes to read a single packet from the flash? 

    I can't spot an issue in your flow that would explain why the throughput is no better. The critical thing is that you read more data from the flash in parallel while the Bluetooth stack is busy, but just looking at your code it seems like you should be able to do this. 

    Still I would recommend simplifying the flow a bit. Ideally you want a main loop that works something like this:

    while(more_bytes_to_send)
    {
        if(queue not full)
        {
            read_buffer_from_flash_and_store_in_queue();
        }
        if(queue not empty)
        {
            read_buffer_from_queue_and_write_to_nus();
        }
    }

    What complicates matters a bit is how to handle the RESOURCES error when you try to send a packet over BLE.
    When you get this error the last packet you tried to send has not been forwarded to the Bluetooth stack, which means the data stored in the data_out buffer (to reference your code) is not stored anywhere else. You need to call ble_nus_data_send(..) again, and make sure not to overwrite the data_out buffer unti the ble_nus_data_send(..) function returns NRF_SUCCESS, indicating that the data has been successfully forwarded to the stack. 

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    To assist you during the development of this mechanism I suggest setting up some GPIO toggling to measure when the MCU is busy reading data from the flash, and toggle a separate pin whenever a packet is forwarded to the SoftDevice. Then you can check these pins on a scope and try to get an overview of the scheduling and timing of these activities. Then it should be easier to understand why things are running too slow. 

    Best regards
    Torbjørn

  • Hi Torbjorn, 

    What is the throughput that you are achieving, and what kind of phone are you connecting to?

    I am able to send 2284066 bytes in about 40-45 seconds. I have tested on 2 phones, one is VIVO Y51s(Android version 12) and the other is Samsung Galaxy A30 (Android version 11).

    Do you know how long it takes to read a single packet from the flash? 

    I don't know about that, how can I measure it?

    Still I would recommend simplifying the flow a bit. Ideally you want a main loop that works something like this:

    I have implement this approach, but now the program got stuck after 1 write to the queue and 1 read from the queue as you can see in the attached log:

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    To assist you during the development of this mechanism I suggest setting up some GPIO toggling to measure when the MCU is busy reading data from the flash, and toggle a separate pin whenever a packet is forwarded to the SoftDevice. Then you can check these pins on a scope and try to get an overview of the scheduling and timing of these activities. Then it should be easier to understand why things are running too slow. 

    I am trying to implement your valuable suggestions and will let you know if I face any problems. 

    To avoid calling ble_nus_data_send(..) over and over while the Bluetooth buffers are full you should wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event to be returned by the SoftDevice, which signals that there are more buffers available for data. 

    Should I call ble_nus_data_send() again after BLE_GATTS_EVT_HVN_TX_COMPLET?

    Best Regards,

    Sami

  • Hi Torbjorn

    Updates;

    I change the code a little bit and now the programming is not sticking,

    My approach is:

    1) Check if (! nrf_queue_is_full(&m_queue)), then do read from SD card & writing in queue until the queue get filled

    2) check if (!nrf_queue_is_empty(&m_queue)), then start read from queue until

                    queued = nrf_queue_utilization_get(&m_queue) >=244;

    and do send the data to the mobile app (calling ble_nus_data_send()) until

                     err_code== NRF_ERROR_BUSY

    But the ble_nus_data_send() is not called after every nrf_queue_read(). Can you please help me with that?

    see the below code snippet:

        for (;;)
        {
    
            if(file_send_to_peripheral)
            {
    
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
               uint32_t j=1;
    
                if(size > 0)
                {
    
                while(remaining_bytes>0)
                 {
    
                if(! nrf_queue_is_full(&m_queue))
    
                   {
                    while (!nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();  // Writing to queue
                      }
                    }
    
                if(!nrf_queue_is_empty(&m_queue))
                   {
                     while(queued >= 244)
                    {
                       err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); // Reading from queue
    
                        if ( (err_code != NRF_SUCCESS) &&(err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
                            (err_code != NRF_ERROR_BUSY) &&
                            (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING) )
                          {
                            APP_ERROR_CHECK(err_code);
                          }
                          queued = nrf_queue_utilization_get(&m_queue);
                          NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                           j++;
    
                          // sending to app
                        do{
                        err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
                        if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                        {
                        APP_ERROR_CHECK(err_code);
    
                           remaining_bytes -= chunk_length;
                        if(remaining_bytes < chunk_length)
                        {
                         chunk_length = remaining_bytes;
                        }
                         NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                        }
                        }while(err_code== NRF_ERROR_BUSY);
                       
                        }
                        }
                      }
                  }
    
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }

    But the ble_nus_data_send() is not calling after every read from the queue(nrf_queue_read()) as you can see in the below attached log image. 

    Best Regards,

    Best Regards,

    Sami

Reply
  • Hi Torbjorn

    Updates;

    I change the code a little bit and now the programming is not sticking,

    My approach is:

    1) Check if (! nrf_queue_is_full(&m_queue)), then do read from SD card & writing in queue until the queue get filled

    2) check if (!nrf_queue_is_empty(&m_queue)), then start read from queue until

                    queued = nrf_queue_utilization_get(&m_queue) >=244;

    and do send the data to the mobile app (calling ble_nus_data_send()) until

                     err_code== NRF_ERROR_BUSY

    But the ble_nus_data_send() is not called after every nrf_queue_read(). Can you please help me with that?

    see the below code snippet:

        for (;;)
        {
    
            if(file_send_to_peripheral)
            {
    
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               static uint8_t data_out[BLE_NUS_MAX_DATA_LEN] = {0};
               uint32_t j=1;
    
                if(size > 0)
                {
    
                while(remaining_bytes>0)
                 {
    
                if(! nrf_queue_is_full(&m_queue))
    
                   {
                    while (!nrf_queue_is_full(&m_queue))
                      {
                        test_queue_write();  // Writing to queue
                      }
                    }
    
                if(!nrf_queue_is_empty(&m_queue))
                   {
                     while(queued >= 244)
                    {
                       err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); // Reading from queue
    
                        if ( (err_code != NRF_SUCCESS) &&(err_code != NRF_ERROR_INVALID_STATE) && (err_code != NRF_ERROR_RESOURCES) &&
                            (err_code != NRF_ERROR_BUSY) &&
                            (err_code != BLE_ERROR_GATTS_SYS_ATTR_MISSING) )
                          {
                            APP_ERROR_CHECK(err_code);
                          }
                          queued = nrf_queue_utilization_get(&m_queue);
                          NRF_LOG_INFO("Queued after %d read: %i", j, queued);
                           j++;
    
                          // sending to app
                        do{
                        err_code= ble_nus_data_send(&m_nus, data_out, &chunk_length, m_conn_handle);
                        if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                        {
                        APP_ERROR_CHECK(err_code);
    
                           remaining_bytes -= chunk_length;
                        if(remaining_bytes < chunk_length)
                        {
                         chunk_length = remaining_bytes;
                        }
                         NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                        }
                        }while(err_code== NRF_ERROR_BUSY);
                       
                        }
                        }
                      }
                  }
    
                file_send_to_peripheral = false;
            }
            idle_state_handle();
        }
    }

    But the ble_nus_data_send() is not calling after every read from the queue(nrf_queue_read()) as you can see in the below attached log image. 

    Best Regards,

    Best Regards,

    Sami

Children
  • Hi Sami

    Samiulhaq said:
    I don't know about that, how can I measure it?

    The simple way (if you have an oscilloscope or logic analyzer available) is to set a GPIO high when you start reading the flash, and set it low after. Then you just measure the length of the pulse using your scope or logic analyzer. 

    Samiulhaq said:
    Should I call ble_nus_data_send() again after BLE_GATTS_EVT_HVN_TX_COMPLET?

    Yes. The approach I recommend is to set a flag once the NRF_ERROR_RESOURCES occurs, and check this flag in the code before you try to call ble_nus_data_send() in your loop. If the flag is set you know that the Bluetooth stack is busy, and you should not read any more data from the buffer or try to send a new packet. 

    When the TX_COMPLETE event occurs you can clear this flag, such that the next time you run your loop you will read out a new packet from the queue and call ble_nus_data_send() again. 

    Reading through your code I think maybe the while(err_code ==  NRF_ERROR_BUSY) line should be changed to while(err_code == NRF_ERROR_RESOURCES) ?

    The problem now is that you will run nrf_queue_read(..) over and over when the Bluetooth stack is busy transmitting data, which means all this data will be lost. If you make the change I mentioned above I believe your code makes more sense. 

    Still, I have some comments on it:

    1) Checking for the NRF_ERROR_BUSY, BLE_ERROR_GATTS_SYS_ATTR_MISSING, NRF_ERROR_INVALID_STATE errors and more after you run nrf_queue_read(..) doesn't make sense, as this function will not return these error codes. I am guessing you copied this if check from one of the Bluetooth calls, where these error codes might be returned. 

    2) I would reconsider running two levels of while loops. It should be sufficient to run the one high level while loop where you check that remaining_bytes > 0. I don't think running the two other while loops helps with speed, since you will artificially serialize the flash read and Bluetooth transmit, rather than allow these two operations to occur in parallel. 

    3) I still think a better handling of NRF_ERROR_RESOURCES would help. If I have some more time next week I will try to provide a more detailed flow chart, similar to how I handled this in the image transfer demo

    Best regards
    Torbjørn

  • Hello Torbjørn, thanks for the thorough explanation. 

    I have changed my code and added the GPIOs to track the time for reading from the SD card & writing to the queue, reading from the queue & sending to the mobile app. My approach;

    1) check  if(!nrf_queue_is_full(&m_queue)) , read 25* BLE_NUS_MAX_DATA_LEN (25 * 244 = 6,100) bytes from SD card and write it to the queue. 

    2) check if(!nrf_queue_is_empty(&m_queue)), read the whole bytes (6,100) from the queue at one flow, and then send it in chunks of 244 bytes to the mobile. 

    These two operations will continue until the end of the file. Below is the code snippet, kindly go through it and let me know if something has to modify. 

    #define MIN_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Minimum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
    #define MAX_CONN_INTERVAL               MSEC_TO_UNITS(7.5, UNIT_1_25_MS)             /**< Maximum acceptable connection interval (7.5 ms), Connection interval uses 1.25 ms units. */
    #define FILE_SIZE_MAX                   25* BLE_NUS_MAX_DATA_LEN                      /**< Maximum size in bytes of file to be read from SDCARD */
    
    #define QUEUE_SIZE     FILE_SIZE_MAX                               
    NRF_QUEUE_DEF(uint8_t, m_queue, QUEUE_SIZE, NRF_QUEUE_MODE_NO_OVERFLOW);
    
    uint16_t bytes_read;
    static uint8_t file_buffer[FILE_SIZE_MAX];
    
    ff_result = f_open(&file, FILE_NAME, FA_READ);
    size = (f_size(&file)); 
    uint16_t arrLen=0;
     uint32_t err_code;
    static void test_queue_write()
    {
        arrLen = 0;
        ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
        err_code = nrf_queue_write(&m_queue, &file_buffer, sizeof(file_buffer));
        APP_ERROR_CHECK(err_code);
        arrLen = bytes_read ;
    }
    
    static uint8_t data_out[FILE_SIZE_MAX] = {0};
    uint32_t j=0;
    static void test_queue_read()
    {
        ret_code_t err_code;
        err_code = nrf_queue_read(&m_queue, &data_out, sizeof(data_out)); 
    
       if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_RESOURCES))
          {
            APP_ERROR_CHECK(err_code);
            }
    }
    
    for (;;)
        {
    
        if(file_send_to_peripheral)
          {
            if(size > 0)
              {
               uint32_t remaining_bytes = size;  
               uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
               ret_code_t err_code;
               while(remaining_bytes>0)
                    {
                   if(!nrf_queue_is_full(&m_queue))
    
                       {
                        nrf_gpio_pin_set(SD_CARD_LED);                // set gpio
                        test_queue_write();  // Writing to queue
                        nrf_gpio_pin_clear(SD_CARD_LED);             // clear gpio
                       }
    
                   if(!nrf_queue_is_empty(&m_queue))
                        {
                          nrf_gpio_pin_set(PACKET_LED);            // set gpio
                          test_queue_read();                        
                         
                          // sending to app
                                uint16_t rem_bytes = arrLen;
                                while((rem_bytes > 0))
                                      {
                                       err_code= ble_nus_data_send(&m_nus, &data_out[arrLen-rem_bytes], &chunk_length, m_conn_handle);
                                       if ((err_code != NRF_ERROR_INVALID_STATE) &&( err_code != NRF_ERROR_RESOURCES) &&(err_code != NRF_ERROR_NOT_FOUND))    
                                          {
                                             APP_ERROR_CHECK(err_code);
                                             remaining_bytes -= chunk_length;
                                             rem_bytes -= chunk_length;
                                             if(remaining_bytes < chunk_length)
                                                {
                                                 chunk_length = remaining_bytes;
                                                }
                                              NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);
                                          }
                                      }
                         nrf_gpio_pin_clear(PACKET_LED);         // clear gpio
                        } 
                    }
              }
    
                file_send_to_peripheral = false;
          }
            idle_state_handle();
        }
    }

    I don't why both the GPIOs behave the same. I am attaching the logic analyzer and sniffer files to this reply, kindly check them and let me know whether the approach is correct or not. Also, give a brief about the behavior of GPIOs and sniffer file.

    One other thing, when I change (Reduce) the Max. & Min. connection interval then there are more EMPTY PDU packets but the overall throughput is higher than with a large Min. & Max connection intervals with fewer EMPTY PDU packets. Your comment about that? How to decide and apply MIn. & Max. connection intervals?

    Best Regards,

    Sami

    Sniffer_file.pcapng

    Logic_Analyzer_file 

  • Hi Sami

    Thanks for sharing the traces, they show very clearly how much time you are spending on the data readout, and on the BLE transmission. 

    The reason the two signals look like inverted versions of each other is the issue I mentioned earlier, where your program flow does not allow for the file readout and the BLE transmission to happen in parallel. 

    Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

    Looking at your traces I estimate that you would increase throughput by around 15-20% if you can get the two operations to run in parallel. 

    To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

    Still, it is probably not a dramatic change in throughput. Next week I might have more time to help out with a more optimized loop.

    Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

    If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

    Best regards
    Torbjørn

  • Hey Torbjorn, thanks for the thorough explanation. 

    Since you have a while loop around the ble transmission logic you will hang here until the entire 6100 byte buffer is transmitted over BLE, and only then do you exit the inner loop so that you can read more from the file. This means the two operations will run one after the other, slowing down the overall communication. 

    Yes, I understand that the while loop is waiting for 6100 bytes to send over ble, and then it goes back to the "read from SD card and save to the queue" operation, making the two operations run after one another. 

    To do this you would have to do something like I mentioned earlier, where you stop trying to push packets to the BLE stack as soon as the NRF_ERROR_RESOURCES error is returned, and go back to reading out data from the file. Once the TX COMPLETE interrupt occurs you can upload more packets again. 

    Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example. It will be very beneficial if you provide me with a code snippet.

    Regarding high throughput on mobile phones we have had good results in the past using a 15ms connection interval. Combined with a 2M PHY and long packets this will allow you to maximize the throughput when sending data to a phone. 

    When I lower the connection intervals than 

    #define MIN_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS) 
    #define MAX_CONN_INTERVAL MSEC_TO_UNITS(50, UNIT_1_25_MS)

    then I am facing more empty PDU packets, and when I increase connection intervals greater than 50 then the throughput decreases. 

    If you are able to measure the amount of data sent, and the time spent, you will get a measure of the overall throughput. 

    currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

     

    Best Regards,

    Sami

  • Hi Sami

    Samiulhaq said:
    Can you please direct me to a thread/example where they have implemented this logic? I am unable to find TX COMPLETE interrupt in my example.

    The nrf52-ble-image-transfer-demo I mentioned earlier follows this approach. 

    The TX COMPLETE event is handled here

    The implementation is a bit complicated unfortunately, in hindsight I should have optimized the ble_image_transfer_service.c implementation a bit to make it easier to understand how the data is sent. 

    Samiulhaq said:
    currently, I am able to send 2284066 bytes in 23-25 seconds (using 2M PHY). 

    What phone are you using for this test?

    This is around 760kbps, which is actually quite good throughput when connected to a phone. Many phones will be limited to slower rates than this, including the iPhone which doesn't allow you to send BLE packets for more than 50% of the time. 

    The best phones for throughput in my experience is the Samsung Android phones, which can go as high as ~1200kbps in some cases. 

    Best regards
    Torbjørn

Related