Using the Softdevice Controller without RTC Crystal

Question

I want to use the Softdevice Controller without a crystal on the RTC.
I configured my project with

# use internal RC for low-frequency clock
CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL=n
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y

# enable RC oscillator calibration to prevent drift
CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=y
CONFIG_CLOCK_CONTROL_NRF_CALIBRATION_LF_ALWAYS_ON=y

# measured latency is 1.839 ms - add some margin to be safe
CONFIG_MPSL_HFCLK_LATENCY=1950

Advertiements look fine, but interaction with a BLE central either throw an assertion (MPSL ASSERT: 67, 129) or the communication fails:

[00:41:33.617,274] <wrn> bt_conn: conn 0x20004560 failed to establish. RF noise?
[00:41:33.617,399] <wrn> bt_gatt: conn 0x20004560 err 0x0e

I guess that my clock is misconfigured.
Note that I measured the HFCLK latency - adding more margin did not change the observed behaviour.
Also, I added the following tio the device tree:

/{
    clocks {
        /delete-node/ lfxo;
    };
};

&grtc {
      /delete-property/ clocks;
      /delete-property/ clock-names;
      clocks = <&pclk>;
      clock-names = "hfclock";
};