Sending a complete file at once from SD card to Mobile App using NRF52840.

[Samiulhaq]

Hello Nordic Family. I hope this post finds you well. 

I am working on a project where I have connected multiple sensors to the NRF52840. I have collected all the sensor's data and saved it on the SD card as a CSV file. The file size is about 240MB which has 60 Hours of data from all the sensors(client requirements). Now I want to send this complete file at once from the SD card to the Mobile App in the fastest possible way. I have run the below program on my board

/cfs-file/__key/communityserver-discussions-components-files/4/8204.ble_5F00_app_5F00_uart_5F00_fatfs_5F00_send_5F00_sdk_5F00_16.0.0.zip.

but this program reads data one by one byte from the SD card and sends it byte by byte to the mobile app which I don't want. I want to send the whole file at once in the fastest possible way not byte by byte. Any help regarding this will be highly appreciated. 

Best Regards,

Sami 

[Håkon Alseth]

Hi,

 

sends it byte by byte to the mobile app

Buffer up the read out data from the SD card and send them in size of "BLE_NUS_MAX_DATA_LEN" instead?

 

Kind regards,

Håkon

[Samiulhaq]

Hi Hakon, thank you for your kind suggestion. Yes, I am sending the data in the size of "BLE_NUS_MAX_DATA_LEN"( 243byte) but it can't solve my problem because the data is sent in chunks of 243bytes per chunk and I want to send the complete file at once, not in chunks. My file size is about 240MB, will it be fully sent to the mobile App at once? If yes Can you please recommend to me the fastest possible way?

Best Regards,

Sami

[Samiulhaq]

Hi Hakan, thanks for the reply. 

Håkon Alseth said:

The PHY is changed successfully to 2MBit in frame #867.

Yes sir, it is. My mobile phone responded unknown to the PHY update request, so I tried with another mobile and it worked. 

Håkon Alseth said:

What is the throughput you're seeing now?

Unfortunately, the throughput is still the same(220KB per 3-4 seconds).  

Håkon Alseth said:

From the traces it seems that the central (phone) does not poll the peripheral (nrf), which can indicate that it is doing something else. The problem with using a phone is that most bluetooth chipsets there are combo-chipsets, and also uses the same radio to server multiple services.

What do you mean by "the central doesn't poll the peripheral"? If you scroll down the Sniffer file (I shared with you), you will see the data has been exchanged correctly( the green lines in the Sniffer file).

Seems like sending the data file chunk by chunk won't be the best solution to send a large file (about 240MB). Have you watched this video where they are sending images from nRF52 to the mobile app in a single capture and in stream mode like a video? I want to achieve exactly the same of sending the file in bursts, not in chunks. Is it possible with the CSV file or only for the images? 

Best Regards,

Sami

[Håkon Alseth]

Hi Sami,

Samiulhaq said:

What do you mean by "the central doesn't poll the peripheral"? If you scroll down the Sniffer file (I shared with you), you will see the data has been exchanged correctly( the green lines in the Sniffer file).

The data is exchanged properly, but the overall throughput it limited by the central.

 

The phone has "radio silence" for smaller periods (< 50 ms), which indicates that the phone itself is servicing other bluetooth devices or similar on its end.

Even though the peripheral has more data to send, and is signalling this to the central, the central ends the current connection interval earlier than usual.

 

Samiulhaq said:

Seems like sending the data file chunk by chunk won't be the best solution to send a large file (about 240MB). Have you watched this video where they are sending images from nRF52 to the mobile app in a single capture and in stream mode like a video? I want to achieve exactly the same of sending the file in bursts, not in chunks. Is it possible with the CSV file or only for the images? 

The transport both your case, and this video is similar. Data is streamed from a source and over the bluetooth layer, in packets of >200 bytes per payload.

If you're using the same phone as previous test, it will highly likely give you similar throughput.

 

Kind regards,

Håkon

[Samiulhaq]

Thanks Hakan for the quick reply. 

Håkon Alseth said:

The phone has "radio silence" for smaller periods (< 50 ms), which indicates that the phone itself is servicing other bluetooth devices or similar on its end.

Okay got it, what will be the solution for this? Is it depend on the phone?

Håkon Alseth said:

The phone has "radio silence" for smaller periods (< 50 ms), which indicates that the phone itself is servicing other bluetooth devices or similar on its end.

But my phone is only connected to the UART inside the nRF toolbox app and it is only receiving the data from the nRF52. How can I check whether the phone is servicing other Bluetooth devices?

Håkon Alseth said:

The transport both your case, and this video is similar. Data is streamed from a source and over the bluetooth layer, in packets of >200 bytes per payload.

If you're using the same phone as previous test, it will highly likely give you similar throughput.

The sniffer file I have shared with you is the result of a Samsung mobile. It doesn't change(increase) the throughput. 

Another thing I have noticed, if you look at the actual data in the Sniffer file you will see that the data is some like:

UART Rx [truncated]: 6152,15433,36722,45008,23274,41643,855,475,15158,6906,13015,19129,40270,38062,17184,20220111121250\r\n23784,38793,36320,49942,19511,34148,1653,45854,26109,10248,6625,7790,46092,40673,18648,4132,31523,20220111121250\r\

What does [truncated] mean?

How to send 2 lines (about 243Bytes) of the CSV file per chunk and when the \r\n of every 2nd line comes then send the chunk (in short I want to send 2 lines per chunk).

Best Regards,

Sami

[Håkon Alseth]

Hi,

 

Samiulhaq said:

Okay got it, what will be the solution for this? Is it depend on the phone?

Yes, it will depend on the phone.

 

Samiulhaq said:

But my phone is only connected to the UART inside the nRF toolbox app and it is only receiving the data from the nRF52. How can I check whether the phone is servicing other Bluetooth devices?

It could be as simple as the phone needs time to scan for new devices in the background, or that it has a wifi/BLE combo-chipset, where wifi takes priority. It is hard to say exactly what is the issue, unfortunately.

Samiulhaq said:

The sniffer file I have shared with you is the result of a Samsung mobile. It doesn't change(increase) the throughput. 

Another thing I have noticed, if you look at the actual data in the Sniffer file you will see that the data is some like:

UART Rx [truncated]: 6152,15433,36722,45008,23274,41643,855,475,15158,6906,13015,19129,40270,38062,17184,20220111121250\r\n23784,38793,36320,49942,19511,34148,1653,45854,26109,10248,6625,7790,46092,40673,18648,4132,31523,20220111121250\r\

What does [truncated] mean?

How to send 2 lines (about 243Bytes) of the CSV file per chunk and when the \r\n of every 2nd line comes then send the chunk (in short I want to send 2 lines per chunk).

From the log, there's a field called "more data" - this indicates to the central that the peripheral has more data to send.

I have highlighted it here:

 

From the trace, this field will always be set to true, meaning that the peripheral always tries to send more data to the central.

If you look at frame #1226 for instance, where the central uses 27 ms to ask for new data:

There's several of these timing gaps within the trace, which limits your overall throughput, and these are driven by the phone side.

Samiulhaq said:

What does [truncated] mean?

How to send 2 lines (about 243Bytes) of the CSV file per chunk and when the \r\n of every 2nd line comes then send the chunk (in short I want to send 2 lines per chunk).

Truncated means that the wireshark view basically does not show all your data, ie. the graphical view stops listing the data. The data is being sent, it is just a graphical feature of wireshark.

The "\r\n" are line-endings in a file. You will need to parse the datastream before sending it through the bluetooth link to remove or detect these.

 

Kind regards,

Håkon

[Samiulhaq]

Thank you Hakan, for the thorough explanation. 

It concluded that the problem is with the mobile app side(central ) not the peripheral, am I right? 

One last question and sorry for taking your precious time. I want to read a large file (about 240MB) from the SD card but the problem is that when I increase the #define FILE_SIZE_MAX (Maximum size in bytes of the file to be read from SDCARD) greater than 220000, while the following;

                static uint8_t file_buffer[FILE_SIZE_MAX]; 

              ff_result = f_read(&file, file_buffer, FILE_SIZE_MAX, (UINT *) &bytes_read);

 Then it gives the following error:

              .bss is too large to fit in RAM1 memory segment
              .heap is too large to fit in RAM1 memory segment
               section .heap overlaps absolute placed section .stack
               section .stack VMA [000000002003e000,000000002003ffff] overlaps section .bss VMA 

                 [0000000020002c64,0000000020224b08]         

I don't know how to solve this. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

Kind Regards,

Sami 

[Håkon Alseth]

Hi Sami,

Samiulhaq said:

It concluded that the problem is with the mobile app side(central ) not the peripheral, am I right? 

Yes, it is the central that drives the communication and essentially sets all the rules. A peripheral can ask for certain parameters, but it is the central that eventually grants permissions.

Samiulhaq said:

I don't know how to solve this. Simply I want to read the whole file (234MB) from the SD card at once and then send it in chunks to the mobile app, Is this possible or do I have to read in chunks from the SD card too? 

You are running out of RAM by trying to buffer ~220kB of data. You must handle this in a smaller buffer.

 

Kind regards,

Håkon

[Samiulhaq]

Thanks Hakan for helping.

Håkon Alseth said:

Yes, it is the central that drives the communication and essentially sets all the rules. A peripheral can ask for certain parameters, but it is the central that eventually grants permissions.

Okay Sir, I understand, and thank you for the thorough explanation. 

Håkon Alseth said:

You are running out of RAM by trying to buffer ~220kB of data. You must handle this in a smaller buffer.

Can you please refer me to a thread/blog where they have accessed and read large data from SD in small chunks/buffer? I have tried but I am unable to read the large file in a small buffer. I will be very thankful to you. 

Best Regards,

Sami

[Håkon Alseth]

Hi,

 

Samiulhaq said:

Can you please refer me to a thread/blog where they have accessed and read large data from SD in small chunks/buffer? I have tried but I am unable to read the large file in a small buffer. I will be very thankful to you. 

I do not think we have a blog/thread for this scenario.

What you need to handle is a arbitrary byte stream going into a buffer, which again is passed through your bluetooth link. You should not need to buffer many kilobytes of data from the SD card, you need to read SD-data out into your buffer and pass the data through the bluetooth link.

 

Kind regards,

Håkon

[Samiulhaq]

Hi Hakan, thanks for the suggestion.

Yes exactly I have done the same and now I am able to read data from the SD card and send it to the mobile app continuously.  Below code snippet is my approach;

#define FILE_SIZE_MAX                   BLE_NUS_MAX_DATA_LEN  // file size is same as the packet size
static uint8_t file_buffer[FILE_SIZE_MAX];
ff_result = f_open(&file, FILE_NAME, FA_READ);

for (;;)
    {
        if(file_send_to_peripheral)
        {

           file_actual_read_size = (f_size(&file));
           uint32_t remaining_bytes = file_actual_read_size;  
           uint16_t chunk_length = BLE_NUS_MAX_DATA_LEN;
           ret_code_t err_code;

            if(file_actual_read_size > 0)
            {

            while(remaining_bytes > 0)
             {

             ff_result = f_read(&file, file_buffer, sizeof(file_buffer), (UINT *) &bytes_read);
             err_code= ble_nus_data_send(&m_nus, file_buffer, &chunk_length, m_conn_handle);

                if (err_code != NRF_ERROR_RESOURCES)
                {
                    APP_ERROR_CHECK(err_code);

                   remaining_bytes -= chunk_length;
                   if(remaining_bytes < chunk_length)
                    {
                     chunk_length = remaining_bytes;
                    }
                    NRF_LOG_INFO("Remaining bytes to send: %d", remaining_bytes);

                }

                }
                }
            file_send_to_peripheral = false;
        }
        idle_state_handle();
    }
}

Is this the efficient way of reading from the SD card and sending it to the Mobile App?

Best Regards,

Sami

[Håkon Alseth]

Hi Sami,

 

That looks good in terms of algorithm, but this line:

if (err_code != NRF_ERROR_RESOURCES)

Should be expanded to ensure that you do not lose track in cases where the link is disconnected. You should check similar to what is done in the original ble_app_uart sample:

if ((err_code != NRF_ERROR_INVALID_STATE) &&
    (err_code != NRF_ERROR_RESOURCES) &&
    (err_code != NRF_ERROR_NOT_FOUND))

 

Kind regards,

Håkon