This behavior only occurs when connecting to the phone using the native connection procedure, it does not disconnect when using the nRF Connect app on the phone (Android Pixel 3a). Wireshark and the packet sniffer didn't shed much light, I've played with the connection interval settings, slave latency, update count etc. when I play with the max connection interval and the MTU settings I get a hardware fault.