UART frame errors


I'm using a NRF52832 QFAA connected to a Quectel modem through UART. We're using UARTSerial from Mbed OS 5.

We've produced 12 custom boards where on 2 of them the Quectel modem mostly shows UART frame errors from the incoming serial.
We also have 2 other boards that show the same UART frame errors but only when the temperature is below -8°C.
When I connect a logic analyser on the serial lines everything seems fine. All characters are received and decoded as normal. I've also checked timing on each character but couldn't see anything strange.
Also tried disconnecting the NRF and sending the characters through a usb-ttl adapter to the modem and that works fine.
Also reading the NRF tx line with a usb-ttl works fine. No malformed characters. Somehow the Quectel modem still shows UART frame errors (it might be less tolerant?).

Here is a part of the NRF circuitry. I thought that maybe the antenna was interfering with the crystal. After disabling BLE in code entirely the issue remained.
The crystal I'm using: MCSJK-7F-32.00-8-20-60-B-20
The 12pF capacitors I'm using: GJM1555C1H120GB01D
I just see that the crystal has an operating temperature range of just -20 instead of -40. Still this doesn't clarify why the other 2 always won't work right.
When the board is in the freezer and it is showing UART frame errors I start heating the MCU and the frame errors immediately disappear. This shows that the issue is definitely related to the MCU/crystal. I cannot heat only the crystal because it is way to close to the MCU.

Does anyone know what might be the issue here?