Debugger often hangs on FIH_PANIC

Often when I try to add a breakpoint while debugging the asset tracker example for the Thingy 91, I end up stuck in "FIH_PANIC" in the debugger.
Sometimes breakpoints do work tough, and if I remove the breakpoint that seemingly caused me to end up in FIH_PANIC, the debugger also works again. I also can't find any logic in when adding breakpoints causes it to hang and when it doesn't.
I tried both SDK and toolchain 2.1.1 and the latest 2.2.0 version.
This is where the debugger ends up in when it hangs:
    if (fih_not_eq(fih_rc, FIH_SUCCESS)) {
        BOOT_LOG_ERR("Unable to find bootable image");

        mcuboot_status_change(MCUBOOT_STATUS_NO_BOOTABLE_IMAGE_FOUND);

        FIH_PANIC;
    }

    BOOT_LOG_INF("Bootloader chainload address offset: 0x%x",
                 rsp.br_image_off);
Related