Problem on nRF9160 making a modified lte_ble_gateway sample

[Christopher Smith]

Our application on a nRF9160 to act as intermediary between an AWS cloud and another nRF devicee over bluetooth.  So I attempted to merge the bluetooth IOT sample with the bluetooth portion lte_ble_gateway sample

When I tried running this I got this crash in the early portion of the bluetooth intialization, before any IOT code is running. 

[00:00:00.251,251] <inf> health_hub: The AWS IoT sample started, version: v1.0.0
[00:00:00.251,281] <inf> hp_ble: Initializing Bluetooth..
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.251,617] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.251,647] <err> os: r3/a4: 0x20011498 r12/ip: 0x00000010 r14/lr: 0x00020a35
[00:00:10.251,647] <err> os: xpsr: 0x41000000
[00:00:10.251,678] <err> os: Faulting instruction address (r15/pc): 0x00020a40
[00:00:10.251,708] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.251,739] <err> os: Current thread: 0x20014010 (unknown)
*** Booting nRF Connect SDK v2.5.1 ***Resetting system

    net_buf_put(&bt_dev.cmd_tx_queue, net_buf_ref(buf));

    err = k_sem_take(&sync_sem, HCI_CMD_TIMEOUT);

    BT_ASSERT_MSG(err == 0, "command opcode 0x%04x timeout with err %d", opcode, err);

Somewhere in the k_sem_take()  we get that error 11 but it isn't clear what it is looking for at that point and stepping in doesn't work well.

Any ideas of what sort of problem can give this error?

Here is the prj.conf in case this is a missing configuration problem or something

#
# Copyright (c) 2020 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#
#CONFIG_DEBUG=y

# General
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_HW_ID_LIBRARY=y
CONFIG_ASSERT=y
CONFIG_JSON_LIBRARY=y

# Heap and stacks
#CONFIG_HEAP_MEM_POOL_SIZE=8192
#CONFIG_MAIN_STACK_SIZE=4096
#CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

#CONFIG_BOOTLOADER_MCUBOOT=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_IPV4=y
CONFIG_NET_CONNECTION_MANAGER=y

# AWS IoT library
CONFIG_AWS_IOT=y
CONFIG_AWS_IOT_CLIENT_ID_STATIC="my-thing"
CONFIG_AWS_IOT_BROKER_HOST_NAME="xxx.amazonaws.com"
CONFIG_AWS_IOT_SEC_TAG=201
CONFIG_AWS_IOT_APP_SUBSCRIPTION_LIST_COUNT=2
CONFIG_AWS_IOT_TOPIC_UPDATE_DELTA_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_ACCEPTED_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_REJECTED_SUBSCRIBE=y
CONFIG_AWS_IOT_LAST_WILL=y

# MQTT - Maximum MQTT keepalive timeout specified by AWS IoT Core
CONFIG_MQTT_KEEPALIVE=1200
CONFIG_MQTT_CLEAN_SESSION=y

# Console for user association
CONFIG_CONSOLE_SUBSYS=y
CONFIG_CONSOLE_GETCHAR=y

# Enable Bluetooth stack and libraries
CONFIG_BT=y
CONFIG_BT_H4=y
CONFIG_BT_WAIT_NOP=y
CONFIG_BT_CENTRAL=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DM=y
CONFIG_BT_SCAN=y
CONFIG_BT_SCAN_FILTER_ENABLE=y
CONFIG_BT_SCAN_UUID_CNT=1

CONFIG_UART_INTERRUPT_DRIVEN=y

# Heap and stacks
CONFIG_HEAP_MEM_POOL_SIZE=16384
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

# Update Data Length and MTU (BT_L2CAP_RX_MTU = CONFIG_BT_BUF_ACL_RX_SIZE - BT_L2CAP_HDR_SIZE)
CONFIG_BT_USER_DATA_LEN_UPDATE=y
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_L2CAP_TX_MTU=247

#================================================
# Little FS
#
# Optionally force the file system to be recreated
# CONFIG_APP_WIPE_STORAGE=y

# Let __ASSERT do its job
CONFIG_DEBUG=n

#CONFIG_FLASH=n
#CONFIG_FLASH_MAP=n
#CONFIG_FLASH_PAGE_LAYOUT=n

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

# CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y

CONFIG_NVS=y
CONFIG_SETTINGS=y

[anneterry]

Hello @ minecraftle game, I think the crash occurred during the initialization of the Bluetooth stack in your modified nRF9160 application. The specific error message "ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338" indicates that an assertion failed, causing the program to terminate.

The error code -11 corresponds to a timeout error. It is possible that there is an issue with the Bluetooth initialization or a timeout occurred while waiting for a response from the Bluetooth stack.

I think you can review the code modifications. Verify that the merging of the Bluetooth IoT sample with the LTE BLE gateway sample was done correctly. Check for any potential conflicts or issues that might have arisen during the merge process.

[Christopher Smith]

I ran the lte_gateway sample and got the same error, so it isn't part of the merge.  It is too early for it to be part of the bluetooth scanning or connection itself.

We have used this dev board to do IOT and bluetooth on other samples using pre v2.5 versions.  Why is this sample not working?

Is there another sample or app that I can model the code off of instead

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.261,932] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.261,962] <err> os: r3/a4: 0x2000d218 r12/ip: 0x00000010 r14/lr: 0x000102b5
[00:00:10.261,962] <err> os: xpsr: 0x41000000
[00:00:10.261,962] <err> os: Faulting instruction address (r15/pc): 0x000102c0
[00:00:10.262,023] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.262,054] <err> os: Current thread: 0x2000f7e0 (unknown)
*** Booting nRF Connect SDK v2.5.0 ***Resetting system
[00:00:00.251,129] <inf> lte_ble_gw: LTE Sensor Gateway sample started
[00:00:00.251,312] <inf> lte_ble_gw: Initializing Bluetooth..
[00:00:00.499,786] <inf> lte_ble_gw: Establishing LTE link (this may take some time) ...
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11

[Kenneth]

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11

If you are thinking of this error, this one simply means that the host wasn't able to communicate with the controller over the serial interface, could be some electrical connection problem, hardware flow control, baudrate or wrong pin configuration etc. Maybe some of the pins used have been reconfigured to something else. If you have a logic analyzer you might potentially check working and non-working in case you can see if something is clearly missing.

Kenneth

[Christopher Smith]

OK, but I took the gateway example and loaded the version of the controller you suggested and this error did not occur.  So that should eliminate those sorts of errors unless the two examples have different pins or something.

The merged aws_iot example with the gateway examples bluetooth code still shows that error.  Why would this be if the new lp executable fixed it for the other sample?

does the aws_iot example require the low power controller program loaded?  

[Christopher Smith]

I  ran the base IOT and did not get the error, it attempted connection.  What in the set up of the bluetooth gateway bluetooth code am I missing in my combined example?

[Kenneth]

I don't know, but you may compare the output files for the two projects to try to compare, ref:

\build\zephyr\zephyr.dts
\build\zephyr\.config

If you can't find anything you can try to share the output files for the two projects also, and I can try to take a look.

Kenneth

[Christopher Smith]

in .config in the lte_ble_gateway program there is the section

#
# Serial Drivers
#
CONFIG_UART_NRFX=y
CONFIG_UART_NRFX_UARTE=y
CONFIG_UART_ASYNC_TX_CACHE_SIZE=8
CONFIG_UART_0_ENHANCED_POLL_OUT=y
CONFIG_UART_0_INTERRUPT_DRIVEN=y
# CONFIG_UART_0_NRF_PARITY_BIT is not set
CONFIG_UART_0_NRF_TX_BUFFER_SIZE=32
# CONFIG_UART_0_NRF_HW_ASYNC is not set
# CONFIG_UART_0_NRF_ASYNC_LOW_POWER is not set
# CONFIG_UART_2_INTERRUPT_DRIVEN is not set
CONFIG_UART_2_ASYNC=y
CONFIG_UART_2_ENHANCED_POLL_OUT=y
# CONFIG_UART_2_NRF_PARITY_BIT is not set
CONFIG_UART_2_NRF_TX_BUFFER_SIZE=32
CONFIG_UART_2_NRF_HW_ASYNC=y
# CONFIG_UART_2_NRF_ASYNC_LOW_POWER is not set
CONFIG_UART_2_NRF_HW_ASYNC_TIMER=2
CONFIG_UARTE_NRF_HW_ASYNC=y
CONFIG_UART_ENHANCED_POLL_OUT=y
# CONFIG_SYSCON is not set

-----------------------------------------------------------------------------

in the merged in aws_iot derived sample

#
# Serial Drivers
#
CONFIG_UART_NRFX=y
CONFIG_UART_NRFX_UARTE=y
CONFIG_UART_ASYNC_TX_CACHE_SIZE=8
CONFIG_UART_0_ENHANCED_POLL_OUT=y
CONFIG_UART_0_INTERRUPT_DRIVEN=y
# CONFIG_UART_0_NRF_PARITY_BIT is not set
CONFIG_UART_0_NRF_TX_BUFFER_SIZE=32
# CONFIG_UART_0_NRF_HW_ASYNC is not set
# CONFIG_UART_0_NRF_ASYNC_LOW_POWER is not set
# CONFIG_UART_1_INTERRUPT_DRIVEN is not set
CONFIG_UART_1_ASYNC=y
CONFIG_UART_1_ENHANCED_POLL_OUT=y
# CONFIG_UART_1_NRF_PARITY_BIT is not set
CONFIG_UART_1_NRF_TX_BUFFER_SIZE=32
# CONFIG_UART_1_NRF_HW_ASYNC is not set
# CONFIG_UART_1_NRF_ASYNC_LOW_POWER is not set
CONFIG_UART_2_INTERRUPT_DRIVEN=y
CONFIG_UART_2_ENHANCED_POLL_OUT=y
# CONFIG_UART_2_NRF_PARITY_BIT is not set
CONFIG_UART_2_NRF_TX_BUFFER_SIZE=32
# CONFIG_UART_2_NRF_ASYNC_LOW_POWER is not set
CONFIG_UART_ENHANCED_POLL_OUT=y
# CONFIG_SYSCON is not set

Why does the UART1 not get used in one sample?  Could this be an issue somehow?  what prof setting triggers this?

for completeness the basic aws_iot sample has:

#
# Serial Drivers
#
CONFIG_UART_NRFX=y
CONFIG_UART_NRFX_UARTE=y
CONFIG_UART_ASYNC_TX_CACHE_SIZE=8
CONFIG_UART_0_ENHANCED_POLL_OUT=y
CONFIG_UART_0_ASYNC=y
# CONFIG_UART_0_NRF_PARITY_BIT is not set
CONFIG_UART_0_NRF_TX_BUFFER_SIZE=32
# CONFIG_UART_0_NRF_HW_ASYNC is not set
# CONFIG_UART_0_NRF_ASYNC_LOW_POWER is not set
CONFIG_UART_1_ASYNC=y
CONFIG_UART_1_ENHANCED_POLL_OUT=y
# CONFIG_UART_1_NRF_PARITY_BIT is not set
# CONFIG_UART_1_NRF_HW_ASYNC is not set
# CONFIG_UART_1_NRF_ASYNC_LOW_POWER is not set
CONFIG_UART_ENHANCED_POLL_OUT=y
# CONFIG_SYSCON is not set

no information about UART2.

[Christopher Smith]

device tree differences

in the lte_ble_gateway program

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
};

uart1: arduino_serial: uart@9000 {
compatible = "nordic,nrf-uarte";
reg = < 0x9000 0x1000 >;
interrupts = < 0x9 0x1 >;
status = "disabled";
current-speed = < 0x1c200 >;
pinctrl-0 = < &uart1_default >;
pinctrl-1 = < &uart1_sleep >;
pinctrl-names = "default", "sleep";
};
uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

nrf52840_reset: gpio-reset {
compatible = "nordic,nrf9160dk-nrf52840-reset";
status = "okay";
gpios = < &interface_to_nrf52840 0x9 0x1 >;
};

-----------------------------------------------------------------

in the merged iot program

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &uart2;
nordic,modem-trace-uart = &uart1;
};

uart1: arduino_serial: uart@9000 {
compatible = "nordic,nrf-uarte";
reg = < 0x9000 0x1000 >;
interrupts = < 0x9 0x1 >;
status = "okay";
current-speed = < 0xf4240 >;
pinctrl-0 = < &uart1_default >;
pinctrl-1 = < &uart1_sleep >;
pinctrl-names = "default", "sleep";
hw-flow-control;
};
uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

nrf52840_reset: gpio-reset {
compatible = "nordic,nrf9160dk-nrf52840-reset";
status = "disabled";
gpios = < &interface_to_nrf52840 0x9 0x1 >;
};

[Kenneth]

Can you try to copy the content in the .overlay file for lte_ble_gateway to merged iot program? Ref:

Kenneth

[Christopher Smith]

I could not see how to configure the enable/disable state for the nrf52840_reset.  Building the other set, though, can't find the name __device_dts_ord_97.  It seems to be a uart name but I am not sure which one.

b/CMakeFiles/..__nrf__lib__nrf_modem_lib.dir/trace_backends/uart/uart.c.obj -MF modules\nrf\lib\nrf_modem_lib\CMakeFiles\..__nrf__lib__nrf_modem_lib.dir\trace_backends\uart\uart.c.obj.d -o modules/nrf/lib/nrf_modem_lib/CMakeFiles/..__nrf__lib__nrf_modem_lib.dir/trace_backends/uart/uart.c.obj -c C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c
In file included from C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain/gcc.h:92,
from C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain.h:50,
from C:/ncs/v2.5.1/zephyr/include/zephyr/sys/time_units.h:10,
from C:/ncs/v2.5.1/zephyr/include/zephyr/sys/util.h:615,
from C:/ncs/v2.5.1/zephyr/include/zephyr/devicetree.h:25,
from C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:12,
from C:/ncs/v2.5.1/zephyr/include/zephyr/drivers/uart.h:26,
from C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c:7:
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:85:41: error: '__device_dts_ord_97' undeclared here (not in a function); did you mean '__device_dts_ord_17'?
85 | #define DEVICE_NAME_GET(dev_id) _CONCAT(__device_, dev_id)
| ^~~~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/toolchain/common.h:132:26: note: in definition of macro '_DO_CONCAT'
132 | #define _DO_CONCAT(x, y) x ## y
| ^
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:85:33: note: in expansion of macro '_CONCAT'
85 | #define DEVICE_NAME_GET(dev_id) _CONCAT(__device_, dev_id)
| ^~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:211:37: note: in expansion of macro 'DEVICE_NAME_GET'
211 | #define DEVICE_DT_NAME_GET(node_id) DEVICE_NAME_GET(Z_DEVICE_DT_DEV_ID(node_id))
| ^~~~~~~~~~~~~~~
C:/ncs/v2.5.1/zephyr/include/zephyr/device.h:228:34: note: in expansion of macro 'DEVICE_DT_NAME_GET'
228 | #define DEVICE_DT_GET(node_id) (&DEVICE_DT_NAME_GET(node_id))
| ^~~~~~~~~~~~~~~~~~
C:/ncs/v2.5.1/nrf/lib/nrf_modem_lib/trace_backends/uart/uart.c:18:46: note: in expansion of macro 'DEVICE_DT_GET'
18 | static const struct device *const uart_dev = DEVICE_DT_GET(UART_DEVICE_NODE);
| ^~~~~~~~~~~~~
[9/52] Building C object modules/nrf/subsys/net/lib/download_client/CMakeFiles/..__nrf__subsys__net__lib__download_client.dir/src/sanity.c.obj

[Christopher Smith]

In trying to clean up these issues I keep getting undefined device tree like __device_dts_ord_97.  Is there a good way to tie these to the original symbol?

[Kenneth]

Hello,

There are some suggestions here to debug devicetree errors:
https://blog.golioth.io/zephyr-devicetree-troubleshooting-generated-files/  https://docs.zephyrproject.org/latest/build/dts/troubleshooting.html 

Kenneth