MQTT-SN event: Retransmission retries limit has been reached

[Maboo]

Hi,

I am using an pca10059 as a SED together with a border router in my thread network. The SED shall publish every 5 min sensor data. That works fine. But after some time (randomly after 40 min or 6h) I get this error message via logger and I cant publish the sensor data anymore:

<info> app: MQTT-SN event: Retransmission retries limit has been reached.

<info> app: MQTT-SN event: Timed-out message: 5. Message ID: 0.

While testing where the problem lies, I added a pca10056 (SED) and got the same error at the same time for both devices.

These are my parameters for the SED devices (standard):

#define DEFAULT_CHILD_TIMEOUT    40                                         /**< Thread child timeout [s]. */
#define DEFAULT_POLL_PERIOD      1000                                       /**< Thread Sleepy End Device polling period when MQTT-SN Asleep. [ms] */
#define SHORT_POLL_PERIOD        100                                        /**< Thread Sleepy End Device polling period when MQTT-SN Awake. [ms] */
#define SEARCH_GATEWAY_TIMEOUT 5  

#define SCHED_QUEUE_SIZE      32                              /**< Maximum number of events in the scheduler queue. */
#define SCHED_EVENT_DATA_SIZE APP_TIMER_SCHED_EVENT_DATA_SIZE /**< Maximum app_scheduler event size. */

Do I have to change some timeout parameter for the gateway? Or does the thread network need at least one FTD next to the border router?

Ps: I use Linux, SDK17, VS-Code, pca10056/59

[Sigurd Hellesvik]

Hi Maboo

I do not think this is a property of the Thread network. The Border Router/Gateway should act as a FTD, and would in this case be the Thread Leader.

It is possibly a bug in the Gateway/Border Router.
Do you have access to debug logs from your Gateway?

In the meantime, I will also try to replicate the problem.

Regards,
Sigurd Hellesvik

[Maboo]

Hi Sigurd,

thanks for your reply. I tried to find a way to debug the gateway. The standard Paho-mqtt-sn-gateway seems to have a LogMonitor. But i couldn't find one within the nrf gateway (nrf border router image). Is there any LogMonitor available?

Thats my paho-mqtt-sn-gateway.conf:

#**************************************************************************
# Copyright (c) 2016-2019, Tomoaki Yamaguchi
#
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Eclipse Public License v1.0
# and Eclipse Distribution License v1.0 which accompany this distribution.
#
# The Eclipse Public License is available at
#    http://www.eclipse.org/legal/epl-v10.html
# and the Eclipse Distribution License is available at
#   http://www.eclipse.org/org/documents/edl-v10.php.
#***************************************************************************
#
# config file of MQTT-SN Gateway
#

# IPv4 Address of iot.eclipse.org
BrokerName=137.135.83.217
BrokerPortNo=1883
BrokerSecurePortNo=8883

#
# When AggregatingGateway=YES or ClientAuthentication=YES,
# All clients must be specified by the ClientList File
#

ClientAuthentication=NO
AggregatingGateway=NO
QoS-1=NO
Forwarder=NO

#ClientsList=/path/to/your_clients.conf

PredefinedTopic=NO
#PredefinedTopicList=/path/to/your_predefinedTopic.conf

#RootCAfile=/etc/ssl/certs/ca-certificates.crt
#RootCApath=/etc/ssl/certs/
#CertsFile=/path/to/certKey.pem
#PrivateKey=/path/to/privateKey.pem

GatewayID=1
GatewayName=PahoGateway-01
KeepAlive=900
#LoginID=your_ID
#Password=your_Password

# UDP
GatewayPortNo=10000
MulticastIP=225.1.1.1
MulticastPortNo=1883

# GatewayUDP6Broadcast address is set to all Thread devices address
# in order to enable Thread Sleepy Devices to receive multicast messages
# sent from the gateway.
# UDP6
GatewayUDP6Port = 47193
GatewayUDP6Broadcast = ff33:40:fdde:ad00:beef:0:0:1
GatewayUDP6If = wpan0

# XBee
Baudrate=38400
SerialDevice=/dev/ttyUSB0
ApiMode=2

# LOG
ShearedMemory=YES;

Regards Maboo

[Sigurd Hellesvik]

Hi

I have not been able to make this work yet.
But the MQTT-SN is run on the nRF5 border router from the "paho-mqttsn-gateway.service". This can be seen in "/etc/systemd/system/paho-mqttsn-gateway.service". From this file, we can see that the MQTT-SN HGateway runs from "/usr/sbin/MQTT-SNGateway".
In the same folder, you can find "/usr/sbin/MQTT-SNLogmonitor".
This relates to paho MQTT-Sn on How to monitor the gateway from a remote terminal.

Try running this Logmonitor. Do you get a log?

Regards,
Sigurd Hellesvik

[Maboo]

Hi,

that is what i get:

I set: ShearedMemory=YES;

pi@raspberrypi:/usr/sbin $ ./MQTT-SNLogmonitor
terminate called after throwing an instance of 'MQTTSNGW::Exception'
  what():  Semaphore can't be created.
Aborted

I couldn't even find one hint on the internet what that error means...

Regards,
Maboo


[Sigurd Hellesvik]

Hi

I get the same error as you.
I will contact our developers to see if i am able to find out how to print these logs, and hope to be back with more by Monday.

Regards,
Sigurd Hellesvik

[Maboo]

Perfect, thank you very much!

[Sigurd Hellesvik]

Hi

We do not seem to have any available way to print out logs for the nRF5 Border Router.

Maboo said:

unfortunately I do have some problems to get the OTBR running.

If you want help with getting this to work, let me know some details, and we can try to solve it.

Regards,
Sigurd Hellesvik

[Maboo]

Hi Sigurd,

thank you anyway for your help. I will come back to you, if i need some help with the OTBR.

Thanks!

Markus

[Maboo]

Hi Sigurd,

it has been a while. I was working on the OTBR. I was able to install it to an RPi 4 using this tutorial and an pca10059 as RCP. Afterwards I installed the paho-mqtt-sn gateway to the RPi 4 using that website. Then i flashed a pca10056 with the standard "mqttsn_client_publisher" example and connect it to the thread network from the OTBR. Both devices are visible via CLI "> router table". OTBR is leader, DK is router, ping each other is also possible as well.

After starting the mqtt-sn-gateway I tried to connect to it with the pca10056 by pressing the buttons. That is what I get:

mqtt-sn-gateway:

pi@raspberrypi:/etc/paho.mqtt-sn.embedded-c/MQTTSNGateway/bin $ sudo ./MQTT-SNGateway 

 ***************************************************************************
 * MQTT-SN Gateway
 * Part of Project Paho in Eclipse
 * (https://github.com/eclipse/paho.mqtt-sn.embedded-c.git)
 *
 * Author : Tomoaki YAMAGUCHI
 * Version: 1.5.1
 ***************************************************************************
 ConfigFile  : ./gateway.conf
 ClientList  : /path/to/your_clients.conf
 Broker      : mqtt.eclipseprojects.io : 1883, 8883
 RootCApath  : (null)
 RootCAfile  : (null)
 CertKey     : (null)
 PrivateKey  : (null)
 SensorN/W   : Multicast Address: [ff33:40:fdde:ad00:beef:0:0:1]:1883, Gateway Port:47193, Interface: wpan0, Hops:1
 Max Clients : 30

20211202 092716.374 PahoGateway-01 starts running.


20211202 092719.544   SEARCHGW          <---  Client                              03 01 01
20211202 092719.545   GWINFO            --->  Clients                             03 02 01

DK - JLinkLog:

SEGGER J-Link V7.50 - Real time terminal output
J-Link OB-SAM3U128-V2-NordicSemi compiled Feb  2 2021 16:47:20 V1.0, SN=683835218
Process: JLinkExe
<info> app_timer: RTC: initialized.
<info> app: Thread version   : OPENTHREAD/20191113-00534-gc6a258e3; NRF52840; Apr  5 2020 21:51:18
<info> app: Network name     : NordicOpenThread
<info> app: Thread interface has been enabled.
<info> app: 802.15.4 Channel : 11
<info> app: 802.15.4 PAN ID  : 0xABCD
<info> app: Radio mode       : rx-on-when-idle
<warning> app_timer: RTC instance already initialized.
<info> app: State changed! Flags: 0x011FD33D Current role: 1

<info> app: State changed<info> app: State changed! Flags: 0x00000064 Current role: 3

<info> app: MQTT-SN event: Gateway discovery procedure has finished.

<info> app: MQTT-SN event: Gateway discovery result: 0x3.

<error> app: CONNECT message could not be sent. Error: 0xE

So the buttonpress somehow triggers the SEARCHGW and GWINFO but i still cant connect.

While searching for a solution, I came across the gateway.conf.

#  SensorNetworks parameters
#==============================
#
# UDP | DTLS 
# 

GatewayPortNo=10000
MulticastPortNo=1883
MulticastIP=225.1.1.1
MulticastTTL=1

#
# UDP6 | DTLS6
#

GatewayIPv6PortNo=47193
MulticastIPv6PortNo=1883
MulticastIPv6=ff33:40:fdde:ad00:beef:0:0:1
MulticastIPv6If=wpan0
MulticastHops=1

I tried to configure it like on the nrf border router image. Especially the GatewayIPv6PortNo, MulticastIPv6 and MulticastIPv6If was different. It still didn't work.

I guess it has something to do with the network prefixes, since when I use "~$ sudo ot-ctl netdata show", there are none.

pi@raspberrypi:~ $ sudo ot-ctl netdata show
Prefixes:
Routes:
Services:
44970 01 4a04b000000e10 s 6000
44970 5d fdf4f30b7d7292086c737a4e8b86d1eed11f s 6000
Done

I know I can add them by:

~ $ sudo ot-ctl add prefix <ipv6-prefix> paros med
~ $ sudo ot-ctl netdata register //activates them

But I couldn't figure how to get the correct prefix.

Do you have any idea?

[Sigurd Hellesvik]

Hi Maboo

You get error code 0xE (14) which is NRF_ERROR_NULL.  (See nRF5_Thread_Zigbee/components/drivers_nrf/nrf_soc_nosd/nrf_error.h)
This is likely from the function:

uint32_t mqttsn_client_connect(mqttsn_client_t      * p_client,
                               mqttsn_remote_t      * p_remote,
                               uint8_t                gateway_id,
                               mqttsn_connect_opt_t * p_options)
{
    NULL_PARAM_CHECK(p_client);
    NULL_PARAM_CHECK(p_remote);
    NULL_PARAM_CHECK(p_options);

    if (gateway_id == 0 || p_options->alive_duration == 0 || p_options->client_id_len == 0)
    {
        return NRF_ERROR_NULL;
    }

    if (p_options->will_flag == 1 && (p_options->will_msg_len == 0 || p_options->will_topic_len == 0)) 
    {
        return NRF_ERROR_NULL;
    }

    if (!is_initialized(p_client))
    {
        return NRF_ERROR_FORBIDDEN;
    }

    if (!is_eligible_for_establishing_connection(p_client))
    {
        return NRF_ERROR_INVALID_STATE;
    }

    memset(&(p_client->gateway_info.addr), 0, sizeof(p_client->gateway_info.addr));
    memcpy(&(p_client->gateway_info.addr), p_remote, sizeof(p_client->gateway_info.addr));

    p_client->gateway_info.id = gateway_id;

    connect_info_init(p_client, p_options);

    uint32_t err_code = mqttsn_packet_sender_connect(p_client);
    if (err_code == NRF_SUCCESS)
    {   
        mqttsn_client_state_update(p_client, SENT_CONNECT);
    }   

    return err_code;
}

Could you find the values of these parameters, and which one causes the error?

The  OTBR with MQTT-SN I linked to earlier should do about the same thing you are trying to do. Maybe have a look at this to see how it does things.

Let me quote you from some time back:

Maboo said:

unfortunately I do have some problems to get the OTBR running.

The OTBR I linked to was running in Docker, which should be relatively straight forward. If you describe the issues you are facing setting it up, I can try to help you make it work.

If it is fast to set up, it would be interesting to see if it behaves differently that the version you are building.

Regards,
Sigurd Hellesvik