Easiest way to subscribe to MQTT messages from Thingy91 on the nRF Cloud

[Joakim U]

Hi,

What's the easiest way to subscribe to the MQTT messages that the thingy91 sends to the nRF cloud?

I have tried setting it up with a node-red and a python client, but I'm never able to connect to the MQTT broker that is set up in the nRF cloud. I want to find out the easiest way to subscribe to these messages so that I can present them in a web application like node-red or similar.

The documentation that is found under APIs doesn't seem to help me, as I'm unable to connect to the broker. As I can see the data being sent to the nRF cloud, I figured it wouldn't be too difficult to just parse that data somewhere else.

Thanks

Best regards

Joakim Uddenfeldt

[Øyvind]

Hello Joakim, 
 

The documentation that is found under APIs doesn't seem to help me, as I'm unable to connect to the broker.

I assume you mean nRF Cloud API documentation, specifically the MQTT chapter?

Can you share how you are trying to connect?

Kind regards,
Øyvind

[Joakim U]

Hi Øyvind,

Here is what I have done:

I start by downloading and installing new certificates on my Thingy using these instructions:

https://devzone.nordicsemi.com/nordic/cellular-iot-guides/b/getting-started-cellular/posts/nrf-cloud-certificate-update

After that, I register my Thingy to my nRF account using the GUI on the nRF Cloud website

I then use the swagger gui to download the MQTT information, such as endpoint and topics using the following curl command

curl -X GET "">api.nrfcloud.com/.../account" -H "accept: application/json" -H "Authorization: Bearer

With the following python code, I try to subscribe to the messages that my thingy sends (which I can see in the nRF cloud GUI

# Import package
import paho.mqtt.client as mqtt
import ssl
import certifi

#Defining mqtt input
mqttEndpoint = "a2n7tk1kp18wix-ats.iot.us-east-1.amazonaws.com"
mqttTopicPrefix = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/"
#topics
gateways = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/a/gateways"
connections = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/a/connections"
alerts = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/a/alerts"
notifications ="prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/a/alerts/notifications"
messagesPrefix = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/m"

# Define Variables
MQTT_PORT = 8883
MQTT_KEEPALIVE_INTERVAL = 45
MQTT_TOPIC = "prod/7215a082-a8f3-4507-a03d-a7e2e0d55387/m/"


MQTT_HOST = "a2n7tk1kp18wix-ats.iot.us-east-1.amazonaws.com"
CA_ROOT_CERT_FILE = "root-CA.cert"
THING_CERT_FILE = "certificate.pem"
THING_PRIVATE_KEY = "privatekey.key"

# Define on connect event function
# We shall subscribe to our Topic in this function
def on_connect(mosq, obj, rc):
    mqttc.subscribe(MQTT_TOPIC, 0)

# Define on_message event function. 
# This function will be invoked every time,
# a new message arrives for the subscribed topic 
def on_message(mosq, obj, msg):
	print "Topic: " + str(msg.topic)
	print "QoS: " + str(msg.qos)
	print "Payload: " + str(msg.payload)

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed to Topic: " + 
	MQTT_MSG + " with QoS: " + str(granted_qos))

# Initiate MQTT Client
mqttc = mqtt.Client()

# Assign event callbacks
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe

# Configure TLS Set
mqttc.tls_set(certfile=THING_CERT_FILE, keyfile=THING_PRIVATE_KEY, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)


# Connect with MQTT Broker
mqttc.connect(MQTT_HOST, MQTT_PORT, MQTT_KEEPALIVE_INTERVAL)


# Continue monitoring the incoming messages for subscribed topic
mqttc.loop_forever()

However, i'm never able to connect successfully? I have tried the same with node-red, but i'm unable to connect using that as well.

Do you have any idea what I might be doing incorrectly? 

Best regards

Joakim Uddenfeldt

[Øyvind]

Hi Joakim, 

So I tried using Mosquitto to subscribe to my Thingy:91 using: 

 mosquitto_sub.exe -h a2n7tk1kp18wix-ats.iot.us-east-1.amazonaws.com -t prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/# --cert 237059_cert-2.pem.crt --cafile 237059_CA.pem -p 8883 --key 237059_PK-2.pem.key -d -i prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account

This outputs: 

Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account sending CONNECT
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received CONNACK (0)
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account sending SUBSCRIBE (Mid: 1, Topic: prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/#, QoS: 0, Options: 0x00)
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received SUBACK
Subscribed (mid: 1): 0
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (51 bytes))
{"appId":"TEMP","data":"25.8","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (52 bytes))
{"appId":"HUMID","data":"16.8","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (56 bytes))
{"appId":"AIR_PRESS","data":"98.6","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (51 bytes))
{"appId":"RSRP","data":"-115","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (51 bytes))
{"appId":"TEMP","data":"25.8","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (52 bytes))
{"appId":"HUMID","data":"16.7","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PUBLISH (d0, q0, r0, m0, 'prod/f1b2cfb2-e82e-4b13-819a-0db2bf5fad69/m/d/nrf-352656100830234/d2c', ... (56 bytes))
{"appId":"AIR_PRESS","data":"98.6","messageType":"DATA"}
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account sending PINGREQ
Client prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account received PINGRESP

Where did you get the certificates from? Did you use the following command from the MQTT API?

http POST https://api.nrfcloud.com/v1/account/certificates Authorization:"Bearer b09f76779f504da08ffea13814024ce545f68b3b"

Which will output:

http POST https://api.nrfcloud.com/v1/account/certificates Authorization:"Bearer b09f76779f504da08ffea13814024ce545f68b3b"
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 3594
Content-Type: application/json
Date: Mon, 07 Oct 2019 13:21:19 GMT
Via: 1.1 4f48f90c7cc5834331dc3e65cd576297.cloudfront.net (CloudFront)
X-Amz-Cf-Id: b-pIQ0BhrpaY2QkFeNaGCX2wae7F5SLPUBkexpZOfs9U0puMBZJdJw==
X-Amz-Cf-Pop: OSL50-C1
X-Amzn-Trace-Id: Root=1-5d9b3bcd-fc5d62902b2ffacdfb8092b3;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: BMZIKFKmoAMFmsg=
x-amzn-RequestId: 950d3186-681c-4a18-b152-b5017f74afd9

{
    "certificate": "-----BEGIN CERTIFICATE-----\n[REMOVED]\n-----END CERTIFICATE-----\n",
    "clientId": "prod-f1b2cfb2-e82e-4b13-819a-0db2bf5fad69-account",
    "id": "[REMOVED]",
    "privateKey": "-----BEGIN RSA PRIVATE KEY-----\n[REMOVED]\n-----END RSA PRIVATE KEY-----\n",
    "publicKey": "-----BEGIN PUBLIC KEY-----\n[REMOVED]\n-----END PUBLIC KEY-----\n"
}

Or did you download from your nRF Cloud Account? See image below, which downloads a .json file to import to LTE Link Monitor - Certificate Manager

Kind regards,

Øyvind

[Joakim U]

Hi Øyvind,

Thank you for testing it out, it's much appreciated. It seems that I might ve done something incorrectly if you were able to get it working.

I downloaded the certificates from the nRF Cloud Account, but perhaps I should do it using the curl command? It should be the same I assume?

I get an AT OK from the Link Monitor when I load the certificates, so it should be correct and I'm also able to read messages from my nRF Cloud account. I will have a look at your output above to see if I have done something incorrectly

Best regards

Joakim Uddenfeldt

[Joakim U]

Hmm it seems that it might be something wrong with my private key. When I check my CA certificate and Client certificate, with a certificate checker, I get an OK back. However when I check my private key, I get an error back. If this is the case, it makes sense why I can't get it to work. As I mentioned earlier, I'm using nRF cloud to download the certificates and then load the .json file in the certificates manager, this helps me copy the certificates in the right format. When I use the curl command, I only receive a private key, which also gets an error when I check it.

[Øyvind]

When you imported the .json file to the Certificate Manager, did you also click on Update Certificates? Hence the AT OK from Link Monitor.

I have forwarded this topic to our nRF Cloud team and will get back to you with an answer.

[Joakim U]

Hi Øyvind,

Yes, I did exactly that and I also received an AT OK from the Link Monitor. I will test it out today again, as I re-did this step yesterday, but then ended up with the POLLHUP error which I later solved.

Thanks for forwarding this issue, I appreciate that.

Best regards

Joakim Uddenfeldt

[Joakim U]

Hi Øyvind,

Unfortunately i'm still unable to subscribe to anything that my Thingy91 is sending. I have tested that my tools work with another MQTT broker and it works fine. I'm still uncertain what the issue can be, but the error I receive is that I can't connect to the broker, which might be due to the certification error that I pointed out above.

As I mentioned earlier, two of my 3 certificates checks out where I can read information such as

Common Name: nrf-IMEI

Organization: Nordic Semiconductor ASA

Locality: Trondheim

State: Trondelag

Country: NO

Valid From: October , 2019

Valid To: September , 2049

Issuer: Nordic Semiconductor

Serial Number: serial number

[Øyvind]

Hello Joakim, 

First I must apologize for not being very clear when asking about the certificate manager. You are not supposed to click on Update Certificates. The only reason to use this is to get the certificates in the correct format to copy and paste into separate certificates, and then close the Certificate Manager. You will need to remove the device and add the device again to your nRF Cloud account.

Joakim U said:

I'm still uncertain what the issue can be, but the error I receive is that I can't connect to the broker, which might be due to the certification error that I pointed out above.

Where are you receiving the error? And what error are you receiving? 

Joakim U said:

have tested that my tools work with another MQTT broker and it works fine.

What MQTT broker did you use?

Thanks!

[Joakim U]

Hi Øyvind,

No worries, I might have been a bit unclear myself as well. What I have done is actually to use the certificate manager in the same way that you are describing, in other words, i'm only using it to get the certificates in the right format. As I'm using Windows, I assumed this was the best way of formatting the certificates. Once I have done that, I check the certificates against  this tool: https://www.sslshopper.com/certificate-decoder.html, which tells me if the certificate is correctly formatted.

I'm currently using Node-red (https://nodered.org) to test out these connections. I haven't set up a broker, instead I'm trying to subscribe to the broker that my Thingy is sending its data to, which is the one you can see below. The error I receive is the following:

8 Oct 08:31:15 - [info] [mqtt-broker:Thingy91] Connection failed to broker: mqtts://a2n7tk1kp18wix-ats.iot.us-east-1.amazonaws.com:8883

 I know it perhaps doesn't say much, but that's the feedback I get from that program.

My workaround is to use HTTP requests to retrieve the messages, but it is not as nice implementation as MQTT.

Best regards

Joakim Uddenfeldt

[Øyvind]

Hello Joakim, 

there have been updates on the nRF Cloud API. To get the correct certificates and client-id to use when connecting to the Thingy:91 broker, please run the following:

 http POST https://api.nrfcloud.com/v1/account/certificates Authorization:"Bearer b09f76779f504da08ffea13814024ce545f68b3b"

This will output CA certificate, Client Certificate and Private Key, which you need to format correctly in separate files.

As I am not familiar with Node-Red, I do not know how to answer questions about this. I have tested using Mosquitto, and I urge you test with this first to validate the connection. 

[Joakim U]

Hi Øyvind,

I will start testing with Mosquitto and also the Curl command you sent above, in order to see if it helps. Did you use a Windows to format it or did it work better in Linux?

Thanks

Best regards

Joakim Uddenfeldt