nRF53 802.15.4 unit timeout when attached to SWD debugger

Dear nRF support,

nRF5340 custom board

nRF NCS 2.5.x

Running into this very weird issue:

Program runs fine if I don't attach debugger to it 

When I do "attach to running program" using Ozone debugger, while it's past unit, program runs and debugger works fine.

however, if I attach to it and reset it using the debugger or reset using some other way while debugger is attached (e.g. kernel reboot cold via shell) - then it fails to boot, and I can catch it with a breakpoint in sys_utils.c asser_post_action. Call stack points to a failure in comms with the 802.15.4 core , sterilization fault with error -5. I believe it is timing out.

Any idea how SWD debugger could affect this stage of init?