Problem on nRF9160 making a modified lte_ble_gateway sample

[Christopher Smith]

Our application on a nRF9160 to act as intermediary between an AWS cloud and another nRF devicee over bluetooth.  So I attempted to merge the bluetooth IOT sample with the bluetooth portion lte_ble_gateway sample

When I tried running this I got this crash in the early portion of the bluetooth intialization, before any IOT code is running. 

[00:00:00.251,251] <inf> health_hub: The AWS IoT sample started, version: v1.0.0
[00:00:00.251,281] <inf> hp_ble: Initializing Bluetooth..
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.251,617] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.251,647] <err> os: r3/a4: 0x20011498 r12/ip: 0x00000010 r14/lr: 0x00020a35
[00:00:10.251,647] <err> os: xpsr: 0x41000000
[00:00:10.251,678] <err> os: Faulting instruction address (r15/pc): 0x00020a40
[00:00:10.251,708] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.251,739] <err> os: Current thread: 0x20014010 (unknown)
*** Booting nRF Connect SDK v2.5.1 ***Resetting system

    net_buf_put(&bt_dev.cmd_tx_queue, net_buf_ref(buf));

    err = k_sem_take(&sync_sem, HCI_CMD_TIMEOUT);

    BT_ASSERT_MSG(err == 0, "command opcode 0x%04x timeout with err %d", opcode, err);

Somewhere in the k_sem_take()  we get that error 11 but it isn't clear what it is looking for at that point and stepping in doesn't work well.

Any ideas of what sort of problem can give this error?

Here is the prj.conf in case this is a missing configuration problem or something

#
# Copyright (c) 2020 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#
#CONFIG_DEBUG=y

# General
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_HW_ID_LIBRARY=y
CONFIG_ASSERT=y
CONFIG_JSON_LIBRARY=y

# Heap and stacks
#CONFIG_HEAP_MEM_POOL_SIZE=8192
#CONFIG_MAIN_STACK_SIZE=4096
#CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

#CONFIG_BOOTLOADER_MCUBOOT=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_IPV4=y
CONFIG_NET_CONNECTION_MANAGER=y

# AWS IoT library
CONFIG_AWS_IOT=y
CONFIG_AWS_IOT_CLIENT_ID_STATIC="my-thing"
CONFIG_AWS_IOT_BROKER_HOST_NAME="xxx.amazonaws.com"
CONFIG_AWS_IOT_SEC_TAG=201
CONFIG_AWS_IOT_APP_SUBSCRIPTION_LIST_COUNT=2
CONFIG_AWS_IOT_TOPIC_UPDATE_DELTA_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_ACCEPTED_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_REJECTED_SUBSCRIBE=y
CONFIG_AWS_IOT_LAST_WILL=y

# MQTT - Maximum MQTT keepalive timeout specified by AWS IoT Core
CONFIG_MQTT_KEEPALIVE=1200
CONFIG_MQTT_CLEAN_SESSION=y

# Console for user association
CONFIG_CONSOLE_SUBSYS=y
CONFIG_CONSOLE_GETCHAR=y

# Enable Bluetooth stack and libraries
CONFIG_BT=y
CONFIG_BT_H4=y
CONFIG_BT_WAIT_NOP=y
CONFIG_BT_CENTRAL=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DM=y
CONFIG_BT_SCAN=y
CONFIG_BT_SCAN_FILTER_ENABLE=y
CONFIG_BT_SCAN_UUID_CNT=1

CONFIG_UART_INTERRUPT_DRIVEN=y

# Heap and stacks
CONFIG_HEAP_MEM_POOL_SIZE=16384
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

# Update Data Length and MTU (BT_L2CAP_RX_MTU = CONFIG_BT_BUF_ACL_RX_SIZE - BT_L2CAP_HDR_SIZE)
CONFIG_BT_USER_DATA_LEN_UPDATE=y
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_L2CAP_TX_MTU=247

#================================================
# Little FS
#
# Optionally force the file system to be recreated
# CONFIG_APP_WIPE_STORAGE=y

# Let __ASSERT do its job
CONFIG_DEBUG=n

#CONFIG_FLASH=n
#CONFIG_FLASH_MAP=n
#CONFIG_FLASH_PAGE_LAYOUT=n

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

# CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y

CONFIG_NVS=y
CONFIG_SETTINGS=y

[anneterry]

Hello @ minecraftle game, I think the crash occurred during the initialization of the Bluetooth stack in your modified nRF9160 application. The specific error message "ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338" indicates that an assertion failed, causing the program to terminate.

The error code -11 corresponds to a timeout error. It is possible that there is an issue with the Bluetooth initialization or a timeout occurred while waiting for a response from the Bluetooth stack.

I think you can review the code modifications. Verify that the merging of the Bluetooth IoT sample with the LTE BLE gateway sample was done correctly. Check for any potential conflicts or issues that might have arisen during the merge process.

[Christopher Smith]

I ran the lte_gateway sample and got the same error, so it isn't part of the merge.  It is too early for it to be part of the bluetooth scanning or connection itself.

We have used this dev board to do IOT and bluetooth on other samples using pre v2.5 versions.  Why is this sample not working?

Is there another sample or app that I can model the code off of instead

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.261,932] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.261,962] <err> os: r3/a4: 0x2000d218 r12/ip: 0x00000010 r14/lr: 0x000102b5
[00:00:10.261,962] <err> os: xpsr: 0x41000000
[00:00:10.261,962] <err> os: Faulting instruction address (r15/pc): 0x000102c0
[00:00:10.262,023] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.262,054] <err> os: Current thread: 0x2000f7e0 (unknown)
*** Booting nRF Connect SDK v2.5.0 ***Resetting system
[00:00:00.251,129] <inf> lte_ble_gw: LTE Sensor Gateway sample started
[00:00:00.251,312] <inf> lte_ble_gw: Initializing Bluetooth..
[00:00:00.499,786] <inf> lte_ble_gw: Establishing LTE link (this may take some time) ...
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11

[Christopher Smith]

device tree differences

in the lte_ble_gateway program

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
};

uart1: arduino_serial: uart@9000 {
compatible = "nordic,nrf-uarte";
reg = < 0x9000 0x1000 >;
interrupts = < 0x9 0x1 >;
status = "disabled";
current-speed = < 0x1c200 >;
pinctrl-0 = < &uart1_default >;
pinctrl-1 = < &uart1_sleep >;
pinctrl-names = "default", "sleep";
};
uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

nrf52840_reset: gpio-reset {
compatible = "nordic,nrf9160dk-nrf52840-reset";
status = "okay";
gpios = < &interface_to_nrf52840 0x9 0x1 >;
};

-----------------------------------------------------------------

in the merged iot program

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &uart2;
nordic,modem-trace-uart = &uart1;
};

uart1: arduino_serial: uart@9000 {
compatible = "nordic,nrf-uarte";
reg = < 0x9000 0x1000 >;
interrupts = < 0x9 0x1 >;
status = "okay";
current-speed = < 0xf4240 >;
pinctrl-0 = < &uart1_default >;
pinctrl-1 = < &uart1_sleep >;
pinctrl-names = "default", "sleep";
hw-flow-control;
};
uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

nrf52840_reset: gpio-reset {
compatible = "nordic,nrf9160dk-nrf52840-reset";
status = "disabled";
gpios = < &interface_to_nrf52840 0x9 0x1 >;
};

[Kenneth]

Can you try to copy the content in the .overlay file for lte_ble_gateway to merged iot program? Ref:

Kenneth

[Christopher Smith]

I could not see how to configure the enable/disable state for the nrf52840_reset.  Building the other set, though, can't find the name __device_dts_ord_97.  It seems to be a uart name but I am not sure which one.

b/CMakeFiles/..__nrf__lib__nrf_modem_lib.dir/trace_backends/uart/uart.c.obj -MF modules\nrf\lib\nrf_modem_lib\CMakeFiles\..__nrf__lib__nrf_modem_lib.dir\trace_backends\uart\uart.c.obj.d -o modules/nrf/lib/nrf_modem_lib/CMakeFiles/..__nrf__lib__nrf_modem_lib.dir/trace_backends/uart/uart.c.obj -c C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c
In file included from C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain/gcc.h:92,
from C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain.h:50,
from C:/ncs/v2.5.1/zephyr/include/zephyr/sys/time_units.h:10,
from C:/ncs/v2.5.1/zephyr/include/zephyr/sys/util.h:615,
from C:/ncs/v2.5.1/zephyr/include/zephyr/devicetree.h:25,
from C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:12,
from C:/ncs/v2.5.1/zephyr/include/zephyr/drivers/uart.h:26,
from C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c:7:
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:85:41: error: '__device_dts_ord_97' undeclared here (not in a function); did you mean '__device_dts_ord_17'?
85 | #define DEVICE_NAME_GET(dev_id) _CONCAT(__device_, dev_id)
| ^~~~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain/common.h:132:26: note: in definition of macro '_DO_CONCAT'
132 | #define _DO_CONCAT(x, y) x ## y
| ^
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:85:33: note: in expansion of macro '_CONCAT'
85 | #define DEVICE_NAME_GET(dev_id) _CONCAT(__device_, dev_id)
| ^~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:211:37: note: in expansion of macro 'DEVICE_NAME_GET'
211 | #define DEVICE_DT_NAME_GET(node_id) DEVICE_NAME_GET(Z_DEVICE_DT_DEV_ID(node_id))
| ^~~~~~~~~~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:228:34: note: in expansion of macro 'DEVICE_DT_NAME_GET'
228 | #define DEVICE_DT_GET(node_id) (&DEVICE_DT_NAME_GET(node_id))
| ^~~~~~~~~~~~~~~~~~
C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c:18:46: note: in expansion of macro 'DEVICE_DT_GET'
18 | static const struct device *const uart_dev = DEVICE_DT_GET(UART_DEVICE_NODE);
| ^~~~~~~~~~~~~
[9/52] Building C object modules/nrf/subsys/net/lib/download_client/CMakeFiles/..__nrf__subsys__net__lib__download_client.dir/src/sanity.c.obj

[Christopher Smith]

In trying to clean up these issues I keep getting undefined device tree like __device_dts_ord_97.  Is there a good way to tie these to the original symbol?

[Kenneth]

Hello,

There are some suggestions here to debug devicetree errors:
https://blog.golioth.io/zephyr-devicetree-troubleshooting-generated-files/  https://docs.zephyrproject.org/latest/build/dts/troubleshooting.html 

Kenneth

[Christopher Smith]

If I made the source for these projects available on GitHub would be be able to look at it yourself as well?  This is taking more time and we seem to have even gone backwards a bit, as the device tree wasn't complaining before.  I still don't see why the sample of aws_iot and dev_gateway seem to be so fundamentally incompatible.

[Christopher Smith]

../../../arm-zephyr-eabi/bin/ld.bfd.exe: zephyr/drivers/bluetooth/libdrivers__bluetooth.a(h4.c.obj):C:/ncs/v2.5.1/zephyr/drivers/bluetooth/hci/h4.c:560: more undefined references to `__device_dts_ord_118' follow
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: 'C:\ncs\toolchains\c57af46cb7\opt\bin\cmake.EXE' --build 'c:\PhaseMargin\HealthPatch\HealthHub_Dev\HealthHub_NordicDev\HealthHub\build_1'

------------- Device Tree Includes --------------

* 116 /soc/peripheral@40000000/spi@b000/mx25r6435f@1
* 117 /soc/peripheral@40000000/uart@a000
* 118 /soc/peripheral@40000000/uart@a000/nrf-sw-lpuart
*
* Definitions derived from these nodes in dependency order are next,
* followed by /chosen nodes.

------------  project zephyr.dts

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
nordic,modem-trace-uart = &uart1;
};

uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

------------  gateway project zephyr.dts

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
};

uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

Why can't it find the device tree reference in the other project?