MQTT with TLS on AWS EC2

[flau]

Hi,

We have our own MQTT broker setup with TLS running on a AWS EC2 cloud service.  I am building my code based on the MQTT_SIMPLE example.  I have only a CA certificate in the certificates.h file.  There is no client private key and no client public certificate.  SEC_TAG is set to 2840.  Port is set to 8883.  Below is the terminal output.  IP address was resolved correctly.  Unfornately, I wasn't able to connect to our MQTT broker.  I kept getting error code 95.

[00:00:00.266,235] <inf> MQTT_SIMPLE: MQTT started
[00:00:00.271,270] <inf> MQTT_SIMPLE: Provisioning certificates
[00:00:00.428,375] <inf> MQTT_SIMPLE: Disabling PSM and eDRX
[00:00:00.439,147] <inf> MQTT_SIMPLE: LTE Link Connecting
[00:00:15.602,569] <inf> MQTT_SIMPLE: LTE Link Connected
[00:00:17.040,008] <inf> MQTT_SIMPLE: IPv4 Address found xxx.xxx.215.238
[00:00:17.047,363] <inf> MQTT_SIMPLE: TLS enabled
[00:00:17.707,611] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:17.713,256] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds
[00:00:48.585,479] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:48.591,125] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds
[00:01:19.484,619] <err> MQTT_SIMPLE: mqtt_connect -95
[00:01:19.490,264] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds

The same code was able to connect with test.mosquitto.org with its mosquitto.org.crt (PEM format) and port 8883.  Am I missing anything?  What have I done wrong?

By the way, I am using SDK 1.8.0.

[flau]

We put more log messages in the code (mainly in mqtt.c and mqtt_transport_socket_tls.c) and found that there is an error when calling the zsock_connect funcation.  After that an error when calling the mqtt_transport_connect function.  And finally the mqtt_connect -95 error.

[Didrik Rokhaug]

flau said:

Does Trace Collector v2 work on custom board, or nRF9160DK only? 

It should work for any boards that has a serial port.

flau said:

We put more log messages in the code (mainly in mqtt.c and mqtt_transport_socket_tls.c) and found that there is an error when calling the zsock_connect funcation.  After that an error when calling the mqtt_transport_connect function.  And finally the mqtt_connect -95 error.

Yes, that's where it comes from. But to know why, we need to inspect the traffic between the modem and the server.

[leo_nam]

Hello Didrik, 

We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10 which couldnt be done until we moved the broker to a newer version of AWS (Amazon Linux 2). The new broker has the required cipher suites for a tls connection. 

But now the error I'm getting is

Fullscreen 4628.output.txt Download

*** Booting Zephyr OS build v2.7.0-ncs1  ***
Flash regions           Domain          Permissions
00 01 0x00000 0x10000   Secure          rwxl
02 31 0x10000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 1 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU0             Non-Secure      OK
22 NRF_EGU1             Non-Secure      OK
23 NRF_EGU2             Non-Secure      OK
24 NRF_EGU3             Non-Secure      OK
25 NRF_EGU4             Non-Secure      OK
26 NRF_EGU5             Non-Secure      OK
27 NRF_DPPIC            Non-Secure      OK
28 NRF_REGULATORS       Non-Secure      OK
29 NRF_PDM              Non-Secure      OK
30 NRF_I2S              Non-Secure      OK
31 NRF_GPIOTE1          Non-Secure      OK

SPM: NS image at 0x10000
SPM: NS MSP at 0x2001d878
SPM: NS reset vector at 0x137e1
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.7.0-ncs1  ***
[00:00:00.243,072] <wrn> at_notif: Already initialized. Nothing to do
[00:00:00.258,697] <inf> mqtt_simple: The MQTT simple sample started
[00:00:00.258,697] <inf> mqtt_simple: Provisioning certificates
[00:00:00.473,388] <inf> mqtt_simple: Disabling PSM and eDRX
[00:00:00.490,509] <inf> mqtt_simple: LTE Link Connecting...
[00:00:11.282,867] <inf> mqtt_simple: LTE Link Connected!
[00:00:11.285,430] <inf> mqtt_simple: IPv4 Address found 3.97.181.128
[00:00:11.285,461] <dbg> mqtt_simple.client_id_get: client_id = my-client-id
[00:00:11.285,491] <inf> mqtt_simple: TLS enabled
[00:00:11.286,407] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20018af8): Created socket 1
[00:00:13.258,056] <dbg> net_mqtt_enc.connect_request_encode: (0x20018af8): Encoding Protocol Version 04.
--- 5 messages dropped ---
[00:00:13.258,087] <dbg> net_mqtt_enc.pack_uint8: (0x20018af8): >> val:00 cur:0x20019843, end:0x20019a36
[00:00:13.258,087] <dbg> net_mqtt_enc.connect_request_encode: (0x20018af8): Encoding Keep Alive Time 003c.
[00:00:13.258,087] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:003c cur:0x20019844, end:0x20019a36
[00:00:13.258,117] <dbg> net_mqtt_enc: Encoding Client Id.
                                       6d 79 2d 63 6c 69 65 6e  74 2d 69 64             |my-clien t-id
[00:00:13.258,117] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:0000000e cur:0x20019846, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:000c cur:0x20019846, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc: Encoding Username.
                                       70 61 6c 64 65 6e 31 32  33                      |palden12 3
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:0000000b cur:0x20019854, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:0009 cur:0x20019854, end:0x20019a36
[00:00:13.258,178] <dbg> net_mqtt_enc: Encoding Password.
                                       31 32 33                                         |123
[00:00:13.258,178] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:00000005 cur:0x2001985f, end:0x20019a36
[00:00:13.258,178] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:0003 cur:0x2001985f, end:0x20019a36
[00:00:13.258,209] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20018af8): << msg type:0x10 length:0x00000029
[00:00:13.258,209] <dbg> net_mqtt_enc.packet_length_encode: (0x20018af8): >> length:0x00000029 cur:(nil), end:(nil)
[00:00:13.258,239] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20018af8): Fixed header length = 02
[00:00:13.258,239] <dbg> net_mqtt_enc.pack_uint8: (0x20018af8): >> val:10 cur:0x20019839, end:0x20019a36
[00:00:13.258,270] <dbg> net_mqtt_enc.packet_length_encode: (0x20018af8): >> length:0x00000029 cur:0x2001983a, end:0x20019a36
[00:00:13.259,185] <dbg> net_mqtt.client_connect: (0x20018af8): Connect completed
[00:00:13.568,634] <dbg> net_mqtt.mqtt_input: (0x20018af8): state:0x00000002
[00:00:13.568,634] <err> net_mqtt_rx: [536974904]
[00:00:13.568,695] <dbg> net_mqtt_rx.mqtt_read_message_chunk: (0x20018af8): [CID 0x20018c20]: Connection closed.
[00:00:13.568,695] <dbg> net_mqtt_sock_tls.mqtt_client_tls_disconnect: (0x20018af8): Closing socket 1
[00:00:13.568,786] <inf> mqtt_simple: MQTT client disconnected: -128
[00:00:13.568,786] <err> mqtt_simple: mqtt_input: -128
[00:00:13.568,817] <inf> mqtt_simple: Disconnecting MQTT client...
[00:00:13.568,878] <err> mqtt_simple: Could not disconnect MQTT client: -128
[00:00:13.568,878] <inf> mqtt_simple: Reconnecting in 60 seconds...

on the client side.

And on the broker side I see: 

New connection from <ip address> on port 8883.

Client <unknown> disconnected due to protocol error.

I'm able to connect to the broker using the mosquitto_sub command. 

[Didrik Rokhaug]

leo_nam said:

We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10

That's great to hear!

leo_nam said:

But now the error I'm getting is

Do you have a modem trace showing this error?

[leo_nam]

I have a modem trace from the server side (Amazon Linux 2), I didn't save any of the ones I made client side (Nordic).

 server_pcap.pcap

I was wondering if I'd get something different if I did a trace from the server side but I didn't notice any difference. Let me know if you still need one from the client side and I will get it to you. 

Edit: I upgraded my broker to 2.0.11 but to do that I changed to a ubuntu server. However, same output from the server: "Client <unknown> disconnected due to protocol error." 

Here is the pcap from the nordic client.

 client_trace.pcapng

[Didrik Rokhaug]

Thanks,

The traces shows that the TLS handshake is successful, but the server closes the connection after the client sends (presumably) the MQTT Connect message.

However, as the Connect message is sent after the TLS handshake, it is encrypted, so I can't check it for anything weird.

How have you configured your broker?

In most cases I coud on the internet with the same error ended up being configuration errors.

[leo_nam]

Hello, 

The broker has been configured, and over the weekend flau was able to connect successfully over MQTT with TLS to the Amazon Linux 2 server running MQTT broker 1.6.10. 

I was unable to connect using my mqtt_simple project with the development board. However, by restarting with a clean version of the mqtt_simple project by removing and then readding "nrf Connect SDK v1.8.0" and then making then necessary changes to enable tls and certificate provisioning, I was able to connect to the broker using the mqtt_simple project and the development board.

Thank you for your help,

Palden