Problem on nRF9160 making a modified lte_ble_gateway sample

[Christopher Smith]

Our application on a nRF9160 to act as intermediary between an AWS cloud and another nRF devicee over bluetooth.  So I attempted to merge the bluetooth IOT sample with the bluetooth portion lte_ble_gateway sample

When I tried running this I got this crash in the early portion of the bluetooth intialization, before any IOT code is running. 

[00:00:00.251,251] <inf> health_hub: The AWS IoT sample started, version: v1.0.0
[00:00:00.251,281] <inf> hp_ble: Initializing Bluetooth..
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.251,617] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.251,647] <err> os: r3/a4: 0x20011498 r12/ip: 0x00000010 r14/lr: 0x00020a35
[00:00:10.251,647] <err> os: xpsr: 0x41000000
[00:00:10.251,678] <err> os: Faulting instruction address (r15/pc): 0x00020a40
[00:00:10.251,708] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.251,739] <err> os: Current thread: 0x20014010 (unknown)
*** Booting nRF Connect SDK v2.5.1 ***Resetting system

    net_buf_put(&bt_dev.cmd_tx_queue, net_buf_ref(buf));

    err = k_sem_take(&sync_sem, HCI_CMD_TIMEOUT);

    BT_ASSERT_MSG(err == 0, "command opcode 0x%04x timeout with err %d", opcode, err);

Somewhere in the k_sem_take()  we get that error 11 but it isn't clear what it is looking for at that point and stepping in doesn't work well.

Any ideas of what sort of problem can give this error?

Here is the prj.conf in case this is a missing configuration problem or something

#
# Copyright (c) 2020 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#
#CONFIG_DEBUG=y

# General
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_HW_ID_LIBRARY=y
CONFIG_ASSERT=y
CONFIG_JSON_LIBRARY=y

# Heap and stacks
#CONFIG_HEAP_MEM_POOL_SIZE=8192
#CONFIG_MAIN_STACK_SIZE=4096
#CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

#CONFIG_BOOTLOADER_MCUBOOT=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=y
CONFIG_NET_IPV4=y
CONFIG_NET_CONNECTION_MANAGER=y

# AWS IoT library
CONFIG_AWS_IOT=y
CONFIG_AWS_IOT_CLIENT_ID_STATIC="my-thing"
CONFIG_AWS_IOT_BROKER_HOST_NAME="xxx.amazonaws.com"
CONFIG_AWS_IOT_SEC_TAG=201
CONFIG_AWS_IOT_APP_SUBSCRIPTION_LIST_COUNT=2
CONFIG_AWS_IOT_TOPIC_UPDATE_DELTA_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_ACCEPTED_SUBSCRIBE=y
CONFIG_AWS_IOT_TOPIC_GET_REJECTED_SUBSCRIBE=y
CONFIG_AWS_IOT_LAST_WILL=y

# MQTT - Maximum MQTT keepalive timeout specified by AWS IoT Core
CONFIG_MQTT_KEEPALIVE=1200
CONFIG_MQTT_CLEAN_SESSION=y

# Console for user association
CONFIG_CONSOLE_SUBSYS=y
CONFIG_CONSOLE_GETCHAR=y

# Enable Bluetooth stack and libraries
CONFIG_BT=y
CONFIG_BT_H4=y
CONFIG_BT_WAIT_NOP=y
CONFIG_BT_CENTRAL=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DM=y
CONFIG_BT_SCAN=y
CONFIG_BT_SCAN_FILTER_ENABLE=y
CONFIG_BT_SCAN_UUID_CNT=1

CONFIG_UART_INTERRUPT_DRIVEN=y

# Heap and stacks
CONFIG_HEAP_MEM_POOL_SIZE=16384
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

# Update Data Length and MTU (BT_L2CAP_RX_MTU = CONFIG_BT_BUF_ACL_RX_SIZE - BT_L2CAP_HDR_SIZE)
CONFIG_BT_USER_DATA_LEN_UPDATE=y
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_BUF_ACL_RX_SIZE=251
CONFIG_BT_BUF_ACL_TX_SIZE=251
CONFIG_BT_L2CAP_TX_MTU=247

#================================================
# Little FS
#
# Optionally force the file system to be recreated
# CONFIG_APP_WIPE_STORAGE=y

# Let __ASSERT do its job
CONFIG_DEBUG=n

#CONFIG_FLASH=n
#CONFIG_FLASH_MAP=n
#CONFIG_FLASH_PAGE_LAYOUT=n

CONFIG_FILE_SYSTEM=y
CONFIG_FILE_SYSTEM_LITTLEFS=y

# CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y

CONFIG_NVS=y
CONFIG_SETTINGS=y

[anneterry]

Hello @ minecraftle game, I think the crash occurred during the initialization of the Bluetooth stack in your modified nRF9160 application. The specific error message "ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338" indicates that an assertion failed, causing the program to terminate.

The error code -11 corresponds to a timeout error. It is possible that there is an issue with the Bluetooth initialization or a timeout occurred while waiting for a response from the Bluetooth stack.

I think you can review the code modifications. Verify that the merging of the Bluetooth IoT sample with the LTE BLE gateway sample was done correctly. Check for any potential conflicts or issues that might have arisen during the merge process.

[Christopher Smith]

I ran the lte_gateway sample and got the same error, so it isn't part of the merge.  It is too early for it to be part of the bluetooth scanning or connection itself.

We have used this dev board to do IOT and bluetooth on other samples using pre v2.5 versions.  Why is this sample not working?

Is there another sample or app that I can model the code off of instead

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11
[00:00:10.261,932] <err> os: r0/a1: 0x00000003 r1/a2: 0x00000000 r2/a3: 0x00000002
[00:00:10.261,962] <err> os: r3/a4: 0x2000d218 r12/ip: 0x00000010 r14/lr: 0x000102b5
[00:00:10.261,962] <err> os: xpsr: 0x41000000
[00:00:10.261,962] <err> os: Faulting instruction address (r15/pc): 0x000102c0
[00:00:10.262,023] <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
[00:00:10.262,054] <err> os: Current thread: 0x2000f7e0 (unknown)
*** Booting nRF Connect SDK v2.5.0 ***Resetting system
[00:00:00.251,129] <inf> lte_ble_gw: LTE Sensor Gateway sample started
[00:00:00.251,312] <inf> lte_ble_gw: Initializing Bluetooth..
[00:00:00.499,786] <inf> lte_ble_gw: Establishing LTE link (this may take some time) ...
ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:338
command opcode 0x0c03 timeout with err -11

[Kenneth]

Hello,

There are some suggestions here to debug devicetree errors:
https://blog.golioth.io/zephyr-devicetree-troubleshooting-generated-files/  https://docs.zephyrproject.org/latest/build/dts/troubleshooting.html 

Kenneth

[Christopher Smith]

If I made the source for these projects available on GitHub would be be able to look at it yourself as well?  This is taking more time and we seem to have even gone backwards a bit, as the device tree wasn't complaining before.  I still don't see why the sample of aws_iot and dev_gateway seem to be so fundamentally incompatible.

[Christopher Smith]

../../../arm-zephyr-eabi/bin/ld.bfd.exe: zephyr/drivers/bluetooth/libdrivers__bluetooth.a(h4.c.obj):C:/ncs/v2.5.1/zephyr/drivers/bluetooth/hci/h4.c:560: more undefined references to `__device_dts_ord_118' follow
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: 'C:\ncs\toolchains\c57af46cb7\opt\bin\cmake.EXE' --build 'c:\PhaseMargin\HealthPatch\HealthHub_Dev\HealthHub_NordicDev\HealthHub\build_1'

------------- Device Tree Includes --------------

* 116 /soc/peripheral@40000000/spi@b000/mx25r6435f@1
* 117 /soc/peripheral@40000000/uart@a000
* 118 /soc/peripheral@40000000/uart@a000/nrf-sw-lpuart
*
* Definitions derived from these nodes in dependency order are next,
* followed by /chosen nodes.

------------  project zephyr.dts

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
nordic,modem-trace-uart = &uart1;
};

uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

------------  gateway project zephyr.dts

chosen {
zephyr,entropy = &psa_rng;
zephyr,flash-controller = &flash_controller;
zephyr,console = &uart0;
zephyr,shell-uart = &uart0;
zephyr,uart-mcumgr = &uart0;
zephyr,flash = &flash0;
zephyr,sram = &sram0_ns;
zephyr,code-partition = &slot0_ns_partition;
zephyr,bt-uart = &lpuart;
};

uart2: uart@a000 {
compatible = "nordic,nrf-uarte";
reg = < 0xa000 0x1000 >;
interrupts = < 0xa 0x1 >;
status = "okay";
pinctrl-0 = < &uart2_default_alt >;
pinctrl-1 = < &uart2_sleep_alt >;
pinctrl-names = "default", "sleep";
current-speed = < 0xf4240 >;
lpuart: nrf-sw-lpuart {
compatible = "nordic,nrf-sw-lpuart";
status = "okay";
req-pin = < 0x15 >;
rdy-pin = < 0x13 >;
};
};

Why can't it find the device tree reference in the other project?  

[Christopher Smith]

Why has there been no response to this thread?  

[Kenneth]

Hello again,

I was under the impression you were doing steady progress, and that you were getting further. But if that is not the case I can try also, not entirely sure if I am able to do so before the weekend since I am out of office on Friday.

Kenneth

[Christopher Smith]

So do you want to see the project sources on GITHub?

[Kenneth]

Christopher Smith said:

../../../arm-zephyr-eabi/bin/ld.bfd.exe: zephyr/drivers/bluetooth/libdrivers__bluetooth.a(h4.c.obj):C:/ncs/v2.5.1/zephyr/drivers/bluetooth/hci/h4.c:560: more undefined references to `__device_dts_ord_118' follow
collect2.exe: error: ld returned 1 exit status

To build at least I think you may be missing the content of nrf\samples\cellular\lte_ble_gateway\boards\nrf9160dk_nrf9160_ns.conf

Kenneth

[Kenneth]

At least it seems to compile and run here with that change after I tried following your description, see attachment.

Kenneth

aws_iot.rar

[Kenneth]

If that doesn't work with you, try to replace the following in nrf9160dk_nrf9160_ns.overlay:
#include <nrf9160dk_nrf52840_reset_on_if5.dtsi>
With:
#include <nrf9160dk_nrf52840_reset_on_if9.dtsi>

Because the interface between the nRF9160 and the nRF52840 depend on the DK version you have.. Maybe that was your problem all along.
 
Kenneth

 

[Christopher Smith]

What appears to rebuild?  you don't have the source for the project I need to get right.  I already mentioned that the lte_gateway built.

It looks to me in my project that the build.ninja has the .conf file you mentioned in it.  I don't see any files specifying how to include it in the gateway project.

And now I have the problem with a DeviceTree error after the changes you suggested with the 

118 /soc/peripheral@40000000/uart@a000/nrf-sw-lpuart not being found despite it being in the zephyr.dts file.

I have several posts above detailing this