Can't reconnect to MQTTSN gateway

[Mary]

I have been running a mqttsn client publisher/subscriber project (based on the example) successfully for some months.  I have added a mosquitto broker to the raspberry pi border router.  Each thread node publishes data every 10 seconds.

SDK:   nRF5_SDK_for_Thread_and Zigbee_2.0.0

RaspPi image:   RaspPi_Thread_Border_Router_Demo_2.0.0-1.alpha

I turned both the raspberry pi and thread node on at the same time several weeks ago.  Today, the connection to the broker was lost and cannot be regained.  A second thread node with debugger attached shows that the gateway is found, but the connection ( mqttsn_client_connect(&m_client, &m_gateway_addr, m_gateway_id, &m_connect_opt); ) times out repeatedly.   I am assuming the Thread network connection is ok, since the gateway was found.

My PC remains connected to the broker on the raspberry pi, but is not receiving any messages.  I have repeatedly restarted the node attached to the debugger, and it continues to timeout on the mqtt gateway connection.

This error occurs infrequently (only after running continuously for several weeks).  What can I check on the raspberry pi to diagnose the problem?

Thanks,

Mary

[Mary]

I tried killing the MQTT-SNGateway:

sudo kill -9 <PID>

then restar it:

/usr/sbin/MQTT-SNGateway

but got an error:

pi@raspberrypi:/etc/init.d $ /usr/sbin/MQTT-SNGateway
terminate called after throwing an instance of 'MQTTSNGW::Exception'
what(): Semaphore can't be created.
Aborted

I will have to reboot.  I will also download the latest image.

Mary

[Mary]

I stopped and restarted mosquitto:

/etc/init.d/mosquitto stop

/etc/init.d/mosquitto start

then restarted the gateway:

sudo /usr/sbin/MQTT-SNGateway -f paho-mqtt-sn-gateway.conf &

and it ran.

Message publishing is now restored, (but all the messages from the gateway are printing to my raspberry pi console.)

[Mary]

I have reproduce the problem.

I am at the moment testing the watchdog timer function.  To test the timeout function, I comment out the call to feed the watchdog timer so that the timeout and reset occurs after 5 seconds.  This happens repeatedly.  So, it's rebooting and reconnecting to the mqttsn gateway every 5 seconds.  

After a while the gateway starts rejecting the connection.  The attached file shows the mqttsn gateway log:

Client "Dal2599e811bfb8c985"  is running continuously without rebooting.

Client "Dalcc970a4319322c55 " is rebooting after 5 seconds.

See line 1282 for the first client rejection.

Further down (line 1381), even the node that is not rebooting gets rejected

mqtt-gateway_log_connect_rejected.txt

I know this is not a realistic scenario, but I'd like to know why the gateway starts rejecting ANY connection.  

The only recovery is to stop and restart the gateway.

Mary

[AndreasF]

Hi.

This requires some digging, I have asked if our developer team can take a look at it if that's ok for you.

Best regards,

Andreas

[Mary]

Thanks. 

Today, I installed the latest Border Router on the raspberry pi,3.1.0 and the same thing happened.  The gateway starts refusing connections after the repeated re-connects from Dalcc970a4319322c55 .

MQTT-SNGateway is version 1.01

I searched for the error message "CONNECT message has been discarded" and found it here.

The discussion includes this code snippet from MQTTSNGWClientRecvTask.cpp:

                /* create a client */
				client = _gateway->getClientList()->createClient(_sensorNetwork->getSenderAddress(), &data.clientID, false, false);
				log(client, packet, &data.clientID);
				if (!client)
				{
                                        //char buf[128]; 
					WRITELOG("%s Client(%s) was rejected. CONNECT message has been discarded.%s\n", ERRMSG_HEADER, _sensorNetwork->getSenderAddress()->sprint(buf), ERRMSG_FOOTER);
					delete packet;
					continue;
				}

It apparently is unable to create a new client, but I have no idea why.

Mary

[AndreasF]

Hi Mary.

I've forwarded your latest reply to our developers, I have to wait and see what they come up with. Note that it is holiday season so it might take some time.

Best regards,

Andreas

[Mary]

I'm wondering if just updating to the latest version of MQTT-SNGateway would help.  Can you provide me with instructions on how to do that?

[AndreasF]

Hi.

Have you tried to update to the latest SDK, 3.1.0?

Best regards,

Andreas

[Mary]

Yes, as I mentioned above.  

I've also looked at the MQTT-SNGateway code at github.  There's a limit of 100 connections, after that no more can be accepted.  The current version of the software provides for re-using a connection when a device re-connects after a disconnection.  I wonder if this provision is missing in the 1.0.1 version that is installed on the 3.1.0 Border Router, such that after 100 Thread device reboots all connections fail.

I'd like to try the latest version of MQTT-SNGateway, and just wanted some guidance on how to do that without breaking anything.

Mary

[AndreasF]

Hi.

Sorry for the late reply, I've been out of office for a while.

I have forwarded this to our development team to get some assistance from them.

Best regards,

Andreas

[AndreasF]

Hi.

Still waiting on reply from the development team.

Best regards,

Andreas

[Mary]

I was able to download and compile the latest MQTTSN-Gateway.  This solved my problem.  I was able to have a node reboot every 10 seconds and successfully reconnect for several hours.

Instructions for compiling the MQTTSN-Gateway are here.

Here's what I did:

Before compiling on the RaspberryPi, first install ssl-dev.  This adds header files, etc so you can compile OpenSSL dependent code..

$ sudo apt-get update
$ sudo apt-get install libssl-dev

Create a working directory.  I used /home/pi/Projects. 

Next, build the gateway:

$ cd ~
$ mkdir Projects
$ cd Projects
$ git clone -b experiment https://github.com/eclipse/paho.mqtt-sn.embedded-c   
$ cd paho.mqtt-sn.embedded-c/MQTTSNGateway       
$ make SENSORNET=udp6 
$ make install INSTALL_DIR=/home/pi/Projects/MQTTSNGateway_v1.3.1 CONFIG_DIR=/home/pi/Projects/MQTTSNGateway_v1.3.1   
$ make clean    

Stop the currently running gateway:

$ sudo systemctl stop paho-mqttsn-gateway.service

Edit the service file:

$ sudo nano /etc/systemd/system/paho-mqttsn-gateway.service

Change the ExecStart line to explicitly specify the configuration file:

ExecStart=/usr/sbin/MQTT-SNGateway -f /etc/paho-mqtt-sn-gateway.conf

Copy the predefinedTopics.conf file from the CONFIG_DIR to /etc or add a new file for predefined topics and edit as needed.

$ sudo nano /etc/predefinedTopic.conf

Make a backup of the original executable and copy the new one to /usr/sbin:

$ sudo cp /usr/sbin/MQTT-SNGateway /usr/sbin/MQTT-SNGateway_v1.0.1

$ sudo cp /home/pi/Projects/MQTTSNGateway_v1.3.1/MQTT-SNGateway /usr/sbin/MQTT-SNGateway

$ make clean

Restart the gateway:

$ sudo systemctl start paho-mqttsn-gateway.service

Viewing the systemd log will show the gateway activity.

$ journalctl -ef