Configure openthread for mqtt - tcp connection times out?

[Unitware]

Hi, I am in need of some guidance.

The overall goal is to set up MQTT/TLS communication from end devices in a thread network, via a border router to the AWS cloud.

I have my nrf52840DK up and running in the thread network, it has got a IPv6 address from the border router (a KiBRA KTBRN1) via DHCPv6 and I am able to ping in both directions.

My problem is that the TCP connection is not setup correctly and I suspect that I have missed something in the configuration.

on my dev pc:

ubuntu:~$ ifconfig | grep inet
inet 192.168.1.49 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fd47:d3c:923f:0:20c:29ff:fe88:7ce1 prefixlen 64 scopeid 0x0<global>

ubuntu:~$ ping -6 -c2 fd7e:b200:ced:0:0:0:a2e:24
PING fd7e:b200:ced:0:0:0:a2e:24(fd7e:b200:ced::a2e:24) 56 data bytes
64 bytes from fd7e:b200:ced::a2e:24: icmp_seq=1 ttl=63 time=42.5 ms
64 bytes from fd7e:b200:ced::a2e:24: icmp_seq=2 ttl=63 time=39.0 ms

--- fd7e:b200:ced:0:0:0:a2e:24 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 39.054/40.793/42.532/1.739 ms

ubuntu:~$ nc -l 1234

Then on the thread device:

uart:~$ ot ipaddr
fd7e:b200:ced:0:0:0:a2e:24
fdd9:20b7:f073:0:0:ff:fe00:1800
fdd9:20b7:f073:0:597f:71e6:d5f5:3775
fe80:0:0:0:b849:ed15:1dba:12ab
Done

uart:~$ ot ping 64:ff9b::c0a8:131
Done
16 bytes from 64:ff9b:0:0:0:0:c0a8:131: icmp_seq=1 hlim=63 time=57ms

uart:~$ ot ping fd47:d3c:923f:0:20c:29ff:fe88:7ce1
Done
16 bytes from fd47:d3c:923f:0:20c:29ff:fe88:7ce1: icmp_seq=2 hlim=63 time=58ms
uart:~$

uart:~$ net ping 64:ff9b::c0a8:131
PING 64:ff9b::c0a8:131
8 bytes from 64:ff9b::c0a8:131 to fd7e:b200:ced::a2e:24: icmp_seq=0 ttl=63 rssi=0 time=88.50 ms
8 bytes from 64:ff9b::c0a8:131 to fd7e:b200:ced::a2e:24: icmp_seq=1 ttl=63 rssi=0 time=87.65 ms
8 bytes from 64:ff9b::c0a8:131 to fd7e:b200:ced::a2e:24: icmp_seq=2 ttl=63 rssi=0 time=87.40 ms

uart:~$ net ping fd47:d3c:923f:0:20c:29ff:fe88:7ce1
PING fd47:d3c:923f:0:20c:29ff:fe88:7ce1
8 bytes from fd47:d3c:923f:0:20c:29ff:fe88:7ce1 to fd7e:b200:ced::a2e:24: icmp_seq=0 ttl=63 rssi=0 time=88.17 ms
8 bytes from fd47:d3c:923f:0:20c:29ff:fe88:7ce1 to fd7e:b200:ced::a2e:24: icmp_seq=1 ttl=63 rssi=0 time=114.07 ms
8 bytes from fd47:d3c:923f:0:20c:29ff:fe88:7ce1 to fd7e:b200:ced::a2e:24: icmp_seq=2 ttl=63 rssi=0 time=88.01 ms

So far it looks good, but when I try to open a tcp connection it fails.

uart:~$ net tcp connect fd47:d3c:923f:0:20c:29ff:fe88:7ce1 1234
Connecting from [fd7e:b200:ced::a2e:24]:0 to [fd47:d3c:923f:0:20c:29ff:fe88:7ce1]:1234
uart:~$

Here is the debuglog:

D: (shell_uart): conn: 0x2001b780, ref_count: 1
D: (shell_uart): conn: 0x2001b780
D: (shell_uart): Context 0x2000c3e0 binding to TCP [fd7e:b200:ced::a2e:24]:35896 iface 0x200008e4
D: (shell_uart): context: 0x2000c3e0, local: fd7e:b200:ced::a2e:24, remote: fd47:d3c:923f:0:20c:29ff:fe88:7ce1
D: (shell_uart): conn: 0x2001b780 src: fd7e:b200:ced::a2e:24, dst: fd47:d3c:923f:0:20c:29ff:fe88:7ce1
D: (shell_uart): [0x2000c8d8/6/2/0x7f] remote fd47:d3c:923f:0:20c:29ff:fe88:7ce1/1234 
D: (shell_uart):   local fd7e:b200:ced::a2e:24/35896 cb 0x19ea9 ud 0x2000c3e0
D: (shell_uart):  [LISTEN Seq=3781784924 Ack=0]
D: (shell_uart): On iface 0x200008e4 size 20
D: (shell_uart): HDRs length estimation 68
D: (shell_uart): Data allocation maximum size 88 (requested 20)
D: (shell_uart): TDATA (?) [0] frag 0x20019d58 ref 1 (tcp_out_ext():776)
D: (shell_uart): pkt 0x20018ad0 data 0x20019fc0 length 40
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): pkt 0x20018ad0 data 0x20019fe8 length 20
D: (shell_uart): pkt 0x20018ad0 skip 20
D: (shell_uart): pkt 0x20018ad0 data 0x20019fc0 length 40
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): pkt 0x20018ad0 skip 8
D: (shell_uart): pkt 0x20018ad0 skip 32
D: (shell_uart): pkt 0x20018ad0 data 0x20019fe8 length 20
D: (shell_uart): pkt 0x20018ad0 skip 20
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): SYN Seq=3781784924 Len=0
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): SYN Seq=3781784924 Len=0 
D: (shell_uart): pkt 0x20018ad0 skip 40
D: (shell_uart): On iface 0x200008e4 size 60
D: (shell_uart): Data allocation maximum size 60 (requested 60)
D: (shell_uart): TDATA (?) [0] frag 0x20019d70 ref 1 (net_pkt_clone():1765)
D: (shell_uart): pkt 0x20018a80 skip 40
D: (shell_uart): Cloned 0x20018ad0 to 0x20018a80
D: (shell_uart): pkt 0x20018a80 skip 40
D: (shell_uart): pkt 0x20018a80 skip 40
D: (shell_uart): SYN Seq=3781784924 Len=0
D: (shell_uart): TX [12] pkt 0x20018a80 ref 2 (tcp_send():284)
D: (tx_q[0]): Processing (pkt 0x20018a80, prio 1) network packet iface 0x200008e4/1
D: (openthread): TX [12] pkt 0x20018a80 ref 1 frags 0x20019d70 (openthread_handle_frame_to_send():362)
D: (shell_uart): TX [12] pkt 0x20018a80 ref 0 frags 0x20019d70 (tcp_send():325)
D: (shell_uart): TDATA (?) [0] frag 0x20019d70 ref 0 frags (nil) (tcp_send():325)
D: (shell_uartD: (ot_radio_workq): On iface 0x200008e4 size 3
D: (ot_radio_workq): Data allocation maximum size 3 (requested 3)
D: (ot_radio_workq): TDATA (?) [0] frag 0x20019d88 ref 1 (handle_ack():384)
D: (ot_radio_workq): pkt 0x20018a30 data 0x2001088d length 3
D: (ot_radio_workq): pkt 0x20018a30 data 0x2000cbe4 length 3
D: (ot_radio_workq): TX [11] pkt 0x20018a30 ref 0 frags 0x20019d88 (handle_ack():412)
D: (ot_radio_workq): TDATA (?) [0] frag 0x20019d88 ref 0 frags (nil) (handle_ack():412)
D: (ot_radio_workq): TDATA (?) [0] frag 0x20019d88 ref 0 (net_pkt_unref_debug():586)
): TDATA (?) [0] frag 0x20019d70 ref 0 (net_pkt_unref_debug():586)
D: (shell_uart): LISTEN->SYN_SENT
D: (shell_uart): conn: 0x2001b780, ref_count=1
D: (shell_uart): Connection handler 0x2000c8d8 removed
D: (shell_uart): context: 0x2000c3e0, conn: (nil)
D: (shell_uart): Context 0x2000c3e0 released
D: (shell_uart): TX [13] pkt 0x20018ad0 ref 0 frags 0x20019d58 (tcp_send_queue_flush():336)
D: (shell_uart): TDATA (?) [0] frag 0x20019d58 ref 0 frags (nil) (tcp_send_queue_flush():336)
D: (shell_uart): TDATA (?) [0] frag 0x20019d58 ref 0 (net_pkt_unref_debug():586)
D: (shell_uart): TX [14] pkt 0x20018b20 ref 0 frags (nil) (tcp_conn_unref():389)
D: (shell_uart): RX [9] pkt 0x20018670 ref 0 frags (nil) (tcp_conn_unref():392)
D: (shell_uart): conn: 0x2001b780, ret=-116

My configuration

CONFIG_GPIO=y
CONFIG_FPU=y
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_NCS_SAMPLES_DEFAULTS=y
CONFIG_LOG_DEFAULT_LEVEL=4
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MBEDTLS_HEAP_SIZE=10240
CONFIG_OPENTHREAD_RADIO_WORKQUEUE_STACK_SIZE=1024
# CONFIG_MBEDTLS_CIPHER_MODE_CBC is not set
# CONFIG_MBEDTLS_CIPHER_MODE_CTR is not set
# CONFIG_MBEDTLS_CHACHA20_C is not set
# CONFIG_MBEDTLS_POLY1305_C is not set
# CONFIG_MBEDTLS_DHM_C is not set
# CONFIG_MBEDTLS_RSA_C is not set
# CONFIG_MBEDTLS_SHA1_C is not set
# CONFIG_MBEDTLS_SHA512_C is not set
CONFIG_OPENTHREAD_LIBRARY_1_1=y
CONFIG_OPENTHREAD_NORDIC_LIBRARY_MASTER=y
CONFIG_SOC_SERIES_NRF52X=y
CONFIG_SOC_NRF52840_QIAA=y
CONFIG_ARM_MPU=y
CONFIG_NUM_METAIRQ_PRIORITIES=1
CONFIG_CONSOLE=y
CONFIG_RTT_CONSOLE=y
CONFIG_USE_SEGGER_RTT=y
CONFIG_UART_LINE_CTRL=y
CONFIG_ENTROPY_NRF5_THR_THRESHOLD=4
CONFIG_ENTROPY_NRF5_ISR_THRESHOLD=12
CONFIG_USB=y
CONFIG_NEWLIB_LIBC=y
# CONFIG_STDOUT_CONSOLE is not set
CONFIG_STACK_SENTINEL=y
CONFIG_NETWORKING=y
CONFIG_NET_L2_OPENTHREAD=y
CONFIG_OPENTHREAD_MANUAL_START=y
CONFIG_OPENTHREAD_FULL_LOGS=y
CONFIG_OPENTHREAD_JOINER_AUTOSTART=y
CONFIG_OPENTHREAD_JOINER_PSKD="1LABAB"
CONFIG_OPENTHREAD_DEBUG=y
CONFIG_OPENTHREAD_L2_DEBUG=y
CONFIG_OPENTHREAD_L2_LOG_LEVEL_INF=y
# CONFIG_OPENTHREAD_MBEDTLS is not set
CONFIG_NET_IF_MCAST_IPV6_ADDR_COUNT=4
# CONFIG_NET_IPV6_MLD is not set
# CONFIG_NET_IPV6_NBR_CACHE is not set
CONFIG_NET_SHELL=y
CONFIG_NET_TCP=y
CONFIG_NET_PKT_RX_COUNT=10
CONFIG_NET_PKT_TX_COUNT=16
CONFIG_NET_BUF_DATA_SIZE=256
CONFIG_NET_STATISTICS=y
CONFIG_NET_LOG=y
CONFIG_NET_CONN_LOG_LEVEL_DBG=y
CONFIG_MQTT_LIB=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_NET_SOCKETS_POLL_MAX=4
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_UART_SHELL_ON_DEV_NAME="CDC_ACM_0"
CONFIG_USB_DEVICE_VID=0x1915
CONFIG_USB_DEVICE_PID=0x520F
CONFIG_USB_DEVICE_MANUFACTURER="Unitware"
CONFIG_USB_DEVICE_PRODUCT="Zigbee NCP"
CONFIG_USB_CDC_ACM=y
CONFIG_DEBUG_OPTIMIZATIONS=y

[Unitware]

I have now investigated further and finally managed to get the radio sniffer working and decrypt the link level. At this time it seems like the error occurs on the border router, but it is also possible that I have a configuration in the end device that is in error.

The packet is corrupted slightly between the radio and the internal network interface in the router.

wireshark sniffer on 802.15.4 radio
88.854845 fdff:cafe:cafe:cafe::a2e:24 64:ff9b::c0a8:131 TCP 65470 → 1234 [SYN] Seq=0 Win=1280 Len=0
0000 69 98 26 ff b7 00 90 00 18 0d 8e 3e 00 00 01 1d
0010 2a d2 09 43 22 34 49 b4 67 10 2e 1c 46 8b 59 a1
0020 81 b5 de 72 25 1c 1f 68 71 e6 21 01 7d ef 72 61
0030 ca 95 dc 2d 9c 64 7d 33 64 56 ec 3f 43 66 49 a7
0040 b8 01 fa

The decrypted payload:

 

tcpdump on the internal interface:

root@KTBRN1:~# tcpdump -i enx8404d2000933 -vv -x
20:03:45.452525 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 20) fd7e:b200:ced::a2e:24 > 64:ff9b::c0a8:131: [icmp6 sum ok] ICMP6, unknown icmp6 type (255), length 20
0x0000: 6000 0000 0014 3a40 fd7e b200 0ced 0000
0x0010: 0000 0000 0a2e 0024 0064 ff9b 0000 0000
0x0020: 0000 0000 c0a8 0131 ffbe 049e 009c 1980
0x0030: 0000 0000 5002 0500 039e 0000

The fields that are corrupted are the "ip: next header" and "tcp: destination port", the rest of the packet is correct.

ip:next packet - the 7th byte is 3a, should be 06
tcp:destination port - bytes 43, 44 with value 049e should be 04d2

I have contacted the support for the border router, but I would greatly appreciate if someone could take a look at the configuration file to verify that it is ok on "our" side.

[Charlie]

Hi Unitware,

As far as I know, TCP is not supported on OpenThread API. Thread protocol is designed for resource-limited IoT devices, so on top of it UDP/CoAP is the preferred choice for this kind of applications.

Best regards,

Charlie

[Unitware]

Hi Charlie, thanks for looking at this.

I agree that TCP is not the best choice but it would enable us to get direct contact with AWS IoT over MQTT with no middlemen.

The openthread API does not have TCP, but it is possible to open a ipv6 socket and transmit that way.

according to this document at least it is allowed: 
https://portal.threadgroup.org/DesktopModules/Inventures_Document/FileDownload.aspx?ContentID=633

page 13:

UDP and TCP

Thread Devices use UDP (User Datagram Protocol) as defined in [RFC 768] for messaging between devices for mesh establishment and maintenance. Thread Networks also support TCP (or any other IPv6-based transport protocol) for application layer communication.

And reading the spec it is mandatory to support UDP, and optional to support TCP.

Anyway, If anybody know of a good scalable solution to connect threaddevices to AWS I'd be interested to hear about it.