Low transfer rate between nRF52840-DK (BLE) and nRF Connect (android app)


I'm trying to transfer data between an nRF52840-DK and an Android phone which has nRF Connect installed on it and I can only reach about 40kbps (that is kilo bytes per second) where I need about 68kbps.

Here is what I do:

1. on the Android phone (Android 8, Samsung Galaxy 10) I open "nRF Connect" and connect to my custom device. Then I request connection priority high and set preferred PHY to LE 2M for both TX and RX.
I get success for switching PHY TX & RX to LE 2M. and connection is updated to interval: 15ms, latency: 0, timeout: 20000ms.
Then I turn on notifications to my custom characteristic.

2. on the nRF52840-DK I have PDM driver running that collects data into a buffer of type: int16_t pdm_buffer[254]; // biggest buffer that doesnt return an error
When the PDM driver handler is called with p_evt->buffer_released != NULL I send this buffer as a notification over a custom service + characteristics I created that handles the size of this buffer + 1 byte at the beginning for a running packet index (to see if I lose packets).

Overall PDM produces 16kHz * 2 (2 microphones) * 2 (16 bit) = 64000 bytes per second of data which I try to pass to the nRF Connect app in packets of size 254*2 + 1 = 509 bytes (254 samples + 1 index byte at the beginning).

Now, my problem is that I lose about 1/3 of the packets somehow (I can tell by looking at the first byte of the packets I do get in the nRF Connect log) which means a transfer rate of about 40kbps.

My question is - how can I increase the transfer rate ? how can I make it reliable ?