BLE hci_lpuart hangs and crashes app on nrf sdk 2+ (works with 1.9.1)

[CRD]

We tested our project a while ago with nrf sdk v1.9.1 using the lte_ble_gateway sample (app on nrf9160, hci_lpuart on nrf52840 using a nrf9160dk board). Now we have developed a good portion of our production firmware using the v2.1.2 SDK. We recently started adding the BLE interface features and the project builds fine (no errors) but when running the app on a dev board it hangs for a few seconds then crashes while initializing BLE with the following error:

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:327
        k_sem_take failed with err -11

We tried rebuilding our test project with nrf SDK v1.9.1 (we didn't change the hci_lpuart app or make any other changes) and found that the v1.9.1 app still worked as expected without issues. Unfortunately, our new firmware makes use of many v2+ sdk features like the event manager so it's difficult to revert back to 1.9.1.

Beside the SDK version there is one other change that I believe could be a culprit. In SDK 2.1.2 SPM is deprecated and we are building with TF-M now. The old project has a child image folder with this config:

# Set UARTE2 as non-secure, used for HCI
CONFIG_SPM_NRF_UARTE2_NS=y

I'm not sure if there is an equivalent config we are missing for TF-M? Should we revert to SPM?

Configs below (They are all the same as in the lte_ble_gateway sample):

main_app/proj.conf

CONFIG_BT=y
CONFIG_BT_H4=y
CONFIG_BT_WAIT_NOP=y
CONFIG_BT_CENTRAL=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DM=y
CONFIG_BT_SCAN=y
CONFIG_BT_SCAN_FILTER_ENABLE=y
CONFIG_BT_SCAN_UUID_CNT=1

CONFIG_UART_INTERRUPT_DRIVEN=y

boards/nrf9160dk_nrf9160_ns.conf

# BLE LPUART HCI
CONFIG_NRF_SW_LPUART=y
CONFIG_NRF_SW_LPUART_INT_DRIVEN=y

CONFIG_UART_2_ASYNC=y
CONFIG_UART_2_INTERRUPT_DRIVEN=n
CONFIG_UART_2_NRF_HW_ASYNC=y
CONFIG_UART_2_NRF_HW_ASYNC_TIMER=2

hci_lpuart/proj.conf

CONFIG_SERIAL=y
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_BT=y
CONFIG_BT_HCI_RAW=y
CONFIG_BT_HCI_RAW_H4=y
CONFIG_BT_HCI_RAW_H4_ENABLE=y
CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_BUF_EVT_DISCARDABLE_SIZE=255

I can share board files too, but they are also the same as in the sample

POST EDITS/UPDATES:

1. I saw the changes listed in this similar thread https://devzone.nordicsemi.com/f/nordic-q-a/69603/lte_ble_gateway-sample-k_sem_take-failed-with-err--11 but sdk v2.1.2 seems to already implement both.

2. This can be reproduced by building the lte_ble_sample on sdk v2.1.2 and flashing an nrf9160dk (with hci_lpuart flashed to the boards nrf52840)

[CRD]

UPDATE:

I have two nrf9160dks and only one of them has this issue, the other board boots and starts BLE with no issue using the same firmware on both boards.

Board 1 (works as expected):

PCA10090

1.0.1

2021.37

Board 2 (Ble does not work):

PCA10090

1.0.0

2021.15

Now it seems I will need to figure out what is different between these two boards. Hopefully this is just a quick solder update?

Update to the update: After flashing the board that was working a few times it no works with BLE either (same k_sem_take error at hci_core.c:327). Even if I load the firmware that was working for bit. So now I'm feeling like this is a software issue and not a hardware issue. 

[Øivind]

Hi, we are aware of some issues with the lpuart in newer SDK releases, I will look into whether this is the same issue and get back to you.

It is interesting that you are only seeing issues on one of your boards, thank you for that update.

[CRD]

Thanks for taking a look. I updated the previous post just now. I flashed a few times on my board that was working and all of a sudden it has the same issue, and I can't seem to revert it even if I flash the app firmware that was working before.

If there is a patch or something that can be tested to hot fix this, please let me know and I can test it. We were pushing to release prototypes soon so now my only other consideration is to reformat our entire project to build with sdk 1.9.1.

For some extra info:
- I have built hci_lpuart for the nrf52840 with 1.9.1 and with 2.1.2 and it has worked with both of those builds.

- Reproduced on multiple boards and with multiple apps including the default lte_ble_gateway app

- Seems to occasionally work, then after flashing (no idea what changed in configs or app) stops working and will not work anymore even after re-flashing with different firmware. 

[CRD]

SUCCESS!!

Okay so after writing the last update I wanted to double check that hci_lpuart actually did work when built with both 1.9.1 and 2.1.2. 

I rebuilt hci_lpuart with sdk v1.9.1 and flashed it to the nrf52840 and now BLE works, also I can still build my main app with the 2.1.2 sdk and it works fine.

Solution for now: Build hci_lpuart app for the nrf52840 chip with SDK v1.9.1. The main app flashed to the nrf9160 can be built with newer sdk versions.