Sending a CSV File over ble Gatt connection

RE: Sending a CSV File over ble Gatt connection

Edvin — Thu, 23 Apr 2020 11:03:52 GMT

[quote user="ThomasMorris"]When I try to send 1800 it gets stuck and when I cancel the connection it crashes. I am not sure why.[/quote]

I assume it is an APP_ERROR_CHECK(err_code) that receives an err_code != 0. Which one is it? Does your log say something like "fatal error"? If you define DEBUG in your preprocessor defines, the log should say where it is. So where does it point to?

When you find out where it points to, see if you can figure out why the function call that returned the err_code returned that err_code.

You need to check this yourself. If I just figure out each and every case for you, I basically have to develop the entire application for you. It is better (and faster) if you learn how to deal with the error handler yourself.

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 23 Apr 2020 10:32:54 GMT

Thanks for the psudo code. I have adapted it and added to my SD card reading code so I read a block of data and then send 60 lines of data before re reading the data. This code does not seem to crash when processing upto 1000 data.

However, it crashes when I try to send more data I've tried to test 1800. I am not sure why this is.

For transferring 1000 data (CSV lines) it takes about 30seconds. When I try to send 1800 it gets stuck and when I cancel the connection it crashes. I am not sure why.

void NEW_SEND_CSV()//Send the CSV file chop off when it gets to \r\n
{
    ret_code_t err_code;
   
    uint16_t data_length = 0;
    uint16_t offset_length = 11;
    data_length =(sizeof(ble_csv_data)-offset_length);//Only send 40 bytes

    if(End_of_file == 1)//End of the file
    {
        uint8_t End_of_file_data[1]={0x78}; /*End of file*/
        uint16_t end_of_file_length = sizeof(End_of_file_data);
        err_code = ble_cus_send_csv(&m_cus, End_of_file_data,&end_of_file_length, m_conn_handle);
        APP_ERROR_CHECK(err_code);
    }
    else
    {
        while(End_of_file != 1)//Loop this till end of file is reached
        {
            {
              SD_CARD_READ_DATA();//Read 1 min of data
              #define  LINE_SIZE 51//Single line of data
              #define  DATA_SIZE 3060//1 min of data
              for(int index = 0; index < DATA_SIZE; index += LINE_SIZE)
              {
                    uint8_t data_to_send[LINE_SIZE];//Create an array of 51 bytes to store a single line
                    memcpy(data_to_send, &ble_csv_data_minute[index],LINE_SIZE);//Copy 51 bytes at a time
                    //Send code
                    err_code = ble_cus_send_csv(&m_cus, data_to_send,&data_length, m_conn_handle);
                    if(err_code == NRF_ERROR_RESOURCES)
                    {
                        //Wait
                        while(Ble_gatts_evt_hvn_tx_complete_flag == false)//Gets set to true from interrupt
                        {
                            //Wait here
                        }
                        Ble_gatts_evt_hvn_tx_complete_flag = false;
                        err_code = ble_cus_send_csv(&m_cus, data_to_send,&data_length, m_conn_handle);//Error code has passed re send this data
                        if(err_code == NRF_ERROR_RESOURCES)
                        {
                            while(Ble_gatts_evt_hvn_tx_complete_flag == false)//Gets set to true from interrupt
                            {
                                //Wait here
                            }
                            Ble_gatts_evt_hvn_tx_complete_flag = false;
                            err_code = ble_cus_send_csv(&m_cus, data_to_send,&data_length, m_conn_handle);//Error code has passed re send this data
                        }
                        APP_ERROR_CHECK(err_code);
                    }
                }
            }
        }
    }    
}

RE: Sending a CSV File over ble Gatt connection

Edvin — Fri, 17 Apr 2020 15:06:04 GMT

If it returns NRF_ERROR_RESOURCES, the packet isn't lost. It is never queued. Only packets that are queued with the return value NRF_SUCCESS will be sent. The last snippet you sent will not work, because if it returns NRF_ERROR_RESOURCES then this packet is lost in your case, because I assume your function SD_CARD_Read_Line() will read the next line each time. Do you see where I'm going with this?

Why don't you try to start over with this state machine. Set a variable saying whether or not the last line you read using SD_CARD_Read_Line() was successfully sent or not, and only read a new line from your file when the previous packet was sent successfully.

volatile bool tx_complete_flag = true;
volatile bool last_line_queued = true;

event TX_COMPLETE:
tx_complete_flag = true;

...

send_data(void)
{
    ret_code_t err_code = NRF_SUCCESS;
    
    while (err_code = NRF_SUCCESS)
    {
        err_code = ble_cus_data_send(...);
        if (last_line_queued == true)
        {
            last_line_queued = false;
            SD_CARD_Read_Line();
        }
        if (err_code == NRF_SUCCESS)
        {
            last_line_queued = true;
        }
    }
    if (err_code != NRF_ERROR_RESOURCES)
    {
        APP_ERROR_CHECK(err_code);
    }
    else
    {
        tx_complete_flag = false;
    }
}

This is pseudo code, but hopefully you get the idea.

Best regards,

Edvin

RE: Sending a CSV File over ble Gatt connection

Thomas — Fri, 17 Apr 2020 10:07:44 GMT

Ok So i need to wait for BLE_GATTS_EVT_HVN_TX_COMPLETE twice?

When I get 533 out of 600. I does have the NRF_ERROR_RESOURCES after I have waited for the flag from my BLE_GATTS_EVT_HVN_TX_COMPLETE flag.

case BLE_GATTS_EVT_HVN_TX_COMPLETE:
              Ble_gatts_evt_hvn_tx_complete_flag = true;//Set the flag
              break;

RE: Sending a CSV File over ble Gatt connection

Edvin — Fri, 17 Apr 2020 10:02:57 GMT

If the new packet is longer than the one that was ACKed (in the TX_COMPLETE event), then this still may return NRF_ERROR_RESOURCES. The queue isn't a number of packets, but number of bytes. Wait for the next TX_COMPLETE event if it still returns NRF_ERROR_RESOURCES.

[quote user="ThomasMorris"]When I store this data to my phone I receive 533 data out of 600 in my CSV file. [/quote]

Do you have any idea of why you are missing some files? When you called ble_cus_send_csv with these lines, did it return NRF_SUCCESS, or something else?

RE: Sending a CSV File over ble Gatt connection

Thomas — Fri, 17 Apr 2020 09:32:49 GMT

I have tried adding the parts for BLE_GATTS_EVT_HVN_TX_COMPLETE and setting a volatile bool to true when this event runs. However, I then try to queue up another packet and it still returns NRF_ERROR_RESOURCES.

When I store this data to my phone I receive 533 data out of 600 in my CSV file. Below is my loop code. What is incorrect here. Not my notifyflag runs once every 10ms. Without it I store 524 data.

Note I terminate the reading with an end of file packet.(Just an X)

void SEND_CSV()//Send the CSV file chop off when it gets to \r\n
{
    ret_code_t err_code;
    uint16_t data_length = 0;
    uint16_t offset_length = 11;
    data_length =(sizeof(ble_csv_data)-offset_length);//Only send 40 bytes

    if(End_of_file == 1)//End of the file
    {
        uint8_t End_of_file_data[1]={0x78}; /*End of file*/
        uint16_t end_of_file_length = sizeof(End_of_file_data);
        err_code = ble_cus_send_csv(&m_cus, End_of_file_data,&end_of_file_length, m_conn_handle);
        APP_ERROR_CHECK(err_code);
        End_of_file =0;//Allows it to be re-read
    }
    else
    {
        while(End_of_file != 1)//Loop this till end of file is reached
        {
        if(notify_flag == true)
            {
                SD_CARD_Read_Line();//Gets a line of 51bytes of data
                err_code = ble_cus_send_csv(&m_cus, ble_csv_data,&data_length, m_conn_handle);
                if(err_code == NRF_ERROR_RESOURCES)
                {
                    //Wait
                    while(Ble_gatts_evt_hvn_tx_complete_flag == false)//Gets set to true from interrupt
                    {
                        //Wait here
                    }
                    Ble_gatts_evt_hvn_tx_complete_flag = false;
                    err_code = ble_cus_send_csv(&m_cus, ble_csv_data,&data_length, m_conn_handle);//Error code has passed re send this data
                    //APP_ERROR_CHECK(err_code);//This still returns nrf_error_resources
                }
                
               
              notify_flag = false;
            }
        }
    }
 
     
}

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 16 Apr 2020 12:10:40 GMT

So therefore, My bullet point code shown above works via queuing?

This means then I don't need a flag to make it loop 10 times a second.

Therefore, I just need to queue up all my data?

RE: Sending a CSV File over ble Gatt connection

Edvin — Thu, 16 Apr 2020 12:06:05 GMT

Your function ble_cus_send() is calling sd_ble_gatts_hvx(). This is what I mean by queuing. ble_cus_send() = queuing.

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 16 Apr 2020 11:52:12 GMT

I am not quite sure what you mean by queue. Presently I am sending all my data (10 CSV lines for testing like below)

Receive notification command
Loop
- If flag = true(set by notification timer 10 times a second)
- Read sd card line
- send data (ble_cus_send)
- Check error
- reset flag
- Exit condition is all CSV read

This works but is this queuing? If so how can I speed it up atm its 510bytes per second.

RE: Sending a CSV File over ble Gatt connection

Edvin — Thu, 16 Apr 2020 11:02:02 GMT

If you want this to happen fast, you can just queue up as many packets as you like. Queue one using the sd_ble_gatts_hvx function, which you already do, and if that returns NRF_SUCCESS, try to queue another one immediately. Do this as long as it returns NRF_SUCCESS. Eventually, it will return NRF_ERROR_RESOURCES, which means that the buffer is full. When this happens, you must wait for the BLE_GATTS_EVT_HVN_TX_COMPLETE event (add this to your ble_evt_handler()). When you receive this event, you can continue to queue packets until it returns NRF_ERROR_RESOURCES.

Remember that the packet you queue that returns NRF_ERROR_RESOURCES isn't queued, so this would be the first packet you try to queue after the BLE_GATTS_EVT_HVN_TX_COMPLETE event.

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 16 Apr 2020 07:58:33 GMT

Thanks, that did seem to fix it. I have another question. I am now reading my CSV file by manually each time calling my function send CSV. This is working however, I want it to be automatic.

Therefore I presume I can loop this until all my data is sent. My question is: Is there a way to confirm each time data is sent it is received before sending the next data?

I need this as I want to send my full log of data. This will be done using the notifications and, checking for a notification value change.

RE: Sending a CSV File over ble Gatt connection

Edvin — Tue, 14 Apr 2020 09:37:58 GMT

Don't pass it into the error handler by commenting out the line saying APP_ERROR_HANDLER(...):

        case APP_UART_COMMUNICATION_ERROR:
            //APP_ERROR_HANDLER(p_event->data.error_communication);
            break;

RE: Sending a CSV File over ble Gatt connection

Thomas — Tue, 14 Apr 2020 07:56:14 GMT

How would I go about ignoring it. Or better yet how do I prevent the overrun? This crash occurs seemingly around a couple of minutes after i disconnect.

RE: Sending a CSV File over ble Gatt connection

Edvin — Wed, 08 Apr 2020 09:23:53 GMT

I tried to point to the app_uart_evt_t->data.error_communication, which is an uint32_t, but I wanted you to check the description of this event in app_uart.h line 117.

Does it say anything about the value of the error_communication?

From app_uart.h:

uint32_t error_communication; /**< Field used if evt_type is: APP_UART_COMMUNICATION_ERROR. This field contains the value in the ERRORSRC register for the UART peripheral. The UART_ERRORSRC_x defines from nrf5x_bitfields.h can be used to parse the error code. See also the \nRFXX Series Reference Manual for specification. */

You are using the nRF52840, so I suggest you check the ERRORSRC register for the UART on the nRF52840:
https://infocenter.nordicsemi.com/topic/ps_nrf52840/uart.html?cp=4_0_0_5_32_9#topic

A value of 1 means OVERRUN. Perhaps this is to be expected when you disable and/or enable the UART when there is UART data traffic ongoing? Have you tried to ignore the APP_UART_COMMUNICATION_ERROR when you are enabling and disabling the UART?

RE: Sending a CSV File over ble Gatt connection

Thomas — Wed, 08 Apr 2020 08:57:36 GMT

The type is app_uart_evt_t* is is an APP_UART_COMMUNICATION_ERROR.

I disable the uart in my gatt connection when a connection is made. Then I restart it when I disconnect. This error comes some time after i disconnect.

RE: Sending a CSV File over ble Gatt connection

Edvin — Tue, 07 Apr 2020 11:59:30 GMT

For the UART event handler, the suggestion that the error handler comes with (in this case NRF_ERROR_SVC_HANDLER_MISSING) is not correct. The err_code is 1. You must look at the event description. What sort of type is p_event->data.error_communication? See if you can find the declaration of the error_communication in a header file, and see if you can see what it means when this is set to 1.

Hint: What type is p_event? Right click and click go to definition.

RE: Sending a CSV File over ble Gatt connection

Thomas — Tue, 07 Apr 2020 11:25:10 GMT

Just a quick one:

I left my system running and got this,

ERROR 1 [NRF_ERROR_SVC_HANDLER_MISSING] at D:\Segger projects\Segger Projects\SDKFOLDER\examples\ble_peripheral\gatt_issue\custom_files\Uart_comms.c:27
00> PC at: 0x0003F671.

This relates to this line of code:

APP_ERROR_HANDLER(p_event->data.error_communication);

Why does this error occur?

RE: Sending a CSV File over ble Gatt connection

Edvin — Tue, 07 Apr 2020 10:11:12 GMT

[quote user="ThomasMorris"]I presume I need to index it similar to the app_uart example and if the characters \n or \r are found stop adding it to the new data to send. Something like that?[/quote]

Yes, indeed, and then set the p_length to a pointer to a uint16_t containing the length, just like in ble_app_uart.

RE: Sending a CSV File over ble Gatt connection

Thomas — Tue, 07 Apr 2020 09:59:18 GMT

Ok I think I want to send a single line at a time which in memory is 40 bytes. How do I only send 40 bytes as opposed to the whole 51? I presume I need to index it similar to the app_uart example and if the characters \n or \r are found stop adding it to the new data to send. Something like that?

Then send the next line of data.

RE: Sending a CSV File over ble Gatt connection

Edvin — Tue, 07 Apr 2020 09:10:45 GMT

Do you mean if one line in the CSV file is shorter than 51 bytes? Doesn't that already have the linefeed termination on each line?

You can find this before you send it as well. If each line is terminated by a linefeed you could say that each linefeed is the end, and set the length to that position, instead of 51. Another way is to just send all the data, 51 bytes at the time (or even more), and reassemble the file on the other side of the connection.

RE: Sending a CSV File over ble Gatt connection

Thomas — Tue, 07 Apr 2020 07:28:21 GMT

Sorry for my late response.

It does print out 51 bytes via the gatt (setting the characteristic value) However for me to read the data correctly via my NRF log it needs 51 bytes. But, when I inspect the data itself its 40 bytes long. So how can I chop off the excess?

When I read this data on my mobile application I am developing it sees all the data and the stuff I dont want aka the full 51 I need a way of chopping this off. I am thinking of adding a termination character to the CSV file such as X as it wont be used in the CSV. Then when I see X it is the end of a new piece of data. Is this a good idea?

RE: Sending a CSV File over ble Gatt connection

Edvin — Thu, 02 Apr 2020 12:22:05 GMT

[quote user="ThomasMorris"]When you state I have included my headers wrong. Where was this?[/quote]

It seems you have included the header files like you included your .c files. You must include header files by adding their location to this list:

[quote user="ThomasMorris"]I have merged my code back together and I seem to have the same problem it is reading 51 bytes now however.[/quote]

What problem would that be?

[quote user="ThomasMorris"][/quote]

In order to get my SD card line to read the line for the csv I need to read 51 bytes otherwise:

D1: 1584969092,1881,1843,1862,1862,-0.010000

Does not output the right length of data.

Ok.

[quote user="ThomasMorris"]But when I output this via my gatt connection it only takes 30 bytes.[/quote]

What do you mean by "output this via my gatt"?

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 02 Apr 2020 08:25:57 GMT

I have merged my code back together and I seem to have the same problem it is reading 51 bytes now however.

In order to get my SD card line to read the line for the csv I need to read 51 bytes otherwise:

D1: 1584969092,1881,1843,1862,1862,-0.010000

Does not output the right length of data.

But when I output this via my gatt connection it only takes 30 bytes.

Why could this be?

RE: Sending a CSV File over ble Gatt connection

Thomas — Thu, 02 Apr 2020 07:24:36 GMT

Thanks Edvin, I have just tested your code changes and it did output 51 bytes. But, I do not know what the issue was. I will now try re-add my code back in to restore my functionality and test with the changes that you have made.

When you state I have included my headers wrong. Where was this?

RE: Sending a CSV File over ble Gatt connection

Edvin — Wed, 01 Apr 2020 15:06:49 GMT

Ok, a couple of general hints:

Don't include header files the way that you have done. These should only be included in the project settings' preprocessor setting: "Include directories".

I see that you have added the files nrf_nvic.c and nrf_soc.c with some modified options, including some header files that are not intended for use when you use the softdevice. I couldn't compile the project before removing these.

Then there was a warning, which you should always check. NB: When you compile a file but you haven't done any changes, SES will ignore the previous warnings for these files. You didn't include app_fifo.h in your main.c (or Setup.h). When I did, I saw that you had the wrong input parameters (none) on app_fifo_flush(); so I removed that call.

I didn't manage to trigger the notification by your command sets. I just added the SEND_CSV function on a button press (button 4).

Then I sent one notification, and that would send one packet of 51 bytes.

I don't know what the issue was, but I did a bit of cleaning in your project.

Can you try the attached modified project, and see if you get more than 51 bytes if you press button 4 after enabling notifications?

devzone.nordicsemi.com/.../gatt_5F00_issue.zip

Best regards,

Edvin