nrf_modem_lib_init() gets stuck and never finishes for some boards

Our nrf9151 boards are getting stuck when nrf_modem_lib_init() is called.

We just see this line. Usually, the device gets past it in less than a second.

[00:00:00.376,983] <dbg> mqtt_helper: mqtt_helper_poll_loop: Waiting for connection_poll_sem

We're using mwf_nrf91x1_2.0.2

With SDK v2.6.2

We have ordered a couple batches of boards.

In one of our batches 2 of 5 boards got stuck at this step, in our most recent batch 3 of 5 boards got stuck on this.

I'm trying to take modem stack trace but I can't get it to build.

I added snippets for nrf91-modem-trace-uart and tf-enable-share-uart

I added CONFIG_NRF_MODEM_LIB_TRACE=y

and in my dts:

    chosen {
        zephyr,console = &uart0;
        zephyr,shell-uart = &uart0;
        zephyr,uart-mcumgr = &uart0;
        nordic,modem-trace-uart = &uart0;
    };

but during build I get:

-- Found devicetree overlay: /opt/nordic/ncs/v2.6.2/nrf/snippets/nrf91-modem-trace-uart/modem-trace-uart.overlay
devicetree error: 'pinctrl-0' is marked as required in 'properties:' in /opt/nordic/ncs/v2.6.2/zephyr/dts/bindings/serial/nordic,nrf-uarte.yaml, but does not appear in <Node /soc/peripheral@40000000/uart@9000 in '/opt/nordic/ncs/v2.6.2/zephyr/misc/empty_file.c'>

Related