MQTT with TLS on AWS EC2

[flau]

Hi,

We have our own MQTT broker setup with TLS running on a AWS EC2 cloud service.  I am building my code based on the MQTT_SIMPLE example.  I have only a CA certificate in the certificates.h file.  There is no client private key and no client public certificate.  SEC_TAG is set to 2840.  Port is set to 8883.  Below is the terminal output.  IP address was resolved correctly.  Unfornately, I wasn't able to connect to our MQTT broker.  I kept getting error code 95.

[00:00:00.266,235] <inf> MQTT_SIMPLE: MQTT started
[00:00:00.271,270] <inf> MQTT_SIMPLE: Provisioning certificates
[00:00:00.428,375] <inf> MQTT_SIMPLE: Disabling PSM and eDRX
[00:00:00.439,147] <inf> MQTT_SIMPLE: LTE Link Connecting
[00:00:15.602,569] <inf> MQTT_SIMPLE: LTE Link Connected
[00:00:17.040,008] <inf> MQTT_SIMPLE: IPv4 Address found xxx.xxx.215.238
[00:00:17.047,363] <inf> MQTT_SIMPLE: TLS enabled
[00:00:17.707,611] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:17.713,256] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds
[00:00:48.585,479] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:48.591,125] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds
[00:01:19.484,619] <err> MQTT_SIMPLE: mqtt_connect -95
[00:01:19.490,264] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds

The same code was able to connect with test.mosquitto.org with its mosquitto.org.crt (PEM format) and port 8883.  Am I missing anything?  What have I done wrong?

By the way, I am using SDK 1.8.0.

[flau]

Here is the tls_cfg in the main.c.

#if defined(CONFIG_MQTT_LIB_TLS)
     struct mqtt_sec_config *tls_cfg = &(client->transport).tls.config;
     static sec_tag_t sec_tag_list[] = { CONFIG_MQTT_TLS_SEC_TAG };

     client->transport.type = MQTT_TRANSPORT_SECURE;

     tls_cfg->peer_verify = CONFIG_MQTT_TLS_PEER_VERIFY;
     tls_cfg->cipher_count = 0;
     tls_cfg->cipher_list = NULL;
     tls_cfg->sec_tag_count = ARRAY_SIZE(sec_tag_list);
     tls_cfg->sec_tag_list = sec_tag_list;
     tls_cfg->hostname = CONFIG_MQTT_BROKER_HOSTNAME;

#if defined(CONFIG_NRF_MODEM_LIB)
     tls_cfg->session_cache = IS_ENABLED(CONFIG_MQTT_TLS_SESSION_CACHING) ?
     TLS_SESSION_CACHE_ENABLED :
     TLS_SESSION_CACHE_DISABLED;
#else
/* TLS session caching is not supported by the Zephyr network stack */
     tls_cfg->session_cache = TLS_SESSION_CACHE_DISABLED;

#endif

#else
     client->transport.type = MQTT_TRANSPORT_NON_SECURE;
#endif

And in prj.conf, I have

CONFIG_MQTT_BROKER_HOSTNAME="ec2-xxx-xxx-xxx-xxx.ca-central-1.compute.amazonaws.com"
CONFIG_MQTT_BROKER_PORT=8883

Regards,

Floyd

[Didrik Rokhaug]

That should be enough to enable SNI, though I can't see that it is enabled in the .pcap you shared.

Could you share the full URL of your server (either here or in a private message), so I can see what TLS features the server requires/supports?

[leo_nam]

Hello Didrik, 

We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10 which couldnt be done until we moved the broker to a newer version of AWS (Amazon Linux 2). The new broker has the required cipher suites for a tls connection. 

But now the error I'm getting is

Fullscreen 4628.output.txt Download

*** Booting Zephyr OS build v2.7.0-ncs1  ***
Flash regions           Domain          Permissions
00 01 0x00000 0x10000   Secure          rwxl
02 31 0x10000 0x100000  Non-Secure      rwxl

Non-secure callable region 0 placed in flash region 1 with size 32.

SRAM region             Domain          Permissions
00 07 0x00000 0x10000   Secure          rwxl
08 31 0x10000 0x40000   Non-Secure      rwxl

Peripheral              Domain          Status
00 NRF_P0               Non-Secure      OK
01 NRF_CLOCK            Non-Secure      OK
02 NRF_RTC0             Non-Secure      OK
03 NRF_RTC1             Non-Secure      OK
04 NRF_NVMC             Non-Secure      OK
05 NRF_UARTE1           Non-Secure      OK
06 NRF_UARTE2           Secure          SKIP
07 NRF_TWIM2            Non-Secure      OK
08 NRF_SPIM3            Non-Secure      OK
09 NRF_TIMER0           Non-Secure      OK
10 NRF_TIMER1           Non-Secure      OK
11 NRF_TIMER2           Non-Secure      OK
12 NRF_SAADC            Non-Secure      OK
13 NRF_PWM0             Non-Secure      OK
14 NRF_PWM1             Non-Secure      OK
15 NRF_PWM2             Non-Secure      OK
16 NRF_PWM3             Non-Secure      OK
17 NRF_WDT              Non-Secure      OK
18 NRF_IPC              Non-Secure      OK
19 NRF_VMC              Non-Secure      OK
20 NRF_FPU              Non-Secure      OK
21 NRF_EGU0             Non-Secure      OK
22 NRF_EGU1             Non-Secure      OK
23 NRF_EGU2             Non-Secure      OK
24 NRF_EGU3             Non-Secure      OK
25 NRF_EGU4             Non-Secure      OK
26 NRF_EGU5             Non-Secure      OK
27 NRF_DPPIC            Non-Secure      OK
28 NRF_REGULATORS       Non-Secure      OK
29 NRF_PDM              Non-Secure      OK
30 NRF_I2S              Non-Secure      OK
31 NRF_GPIOTE1          Non-Secure      OK

SPM: NS image at 0x10000
SPM: NS MSP at 0x2001d878
SPM: NS reset vector at 0x137e1
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.7.0-ncs1  ***
[00:00:00.243,072] <wrn> at_notif: Already initialized. Nothing to do
[00:00:00.258,697] <inf> mqtt_simple: The MQTT simple sample started
[00:00:00.258,697] <inf> mqtt_simple: Provisioning certificates
[00:00:00.473,388] <inf> mqtt_simple: Disabling PSM and eDRX
[00:00:00.490,509] <inf> mqtt_simple: LTE Link Connecting...
[00:00:11.282,867] <inf> mqtt_simple: LTE Link Connected!
[00:00:11.285,430] <inf> mqtt_simple: IPv4 Address found 3.97.181.128
[00:00:11.285,461] <dbg> mqtt_simple.client_id_get: client_id = my-client-id
[00:00:11.285,491] <inf> mqtt_simple: TLS enabled
[00:00:11.286,407] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20018af8): Created socket 1
[00:00:13.258,056] <dbg> net_mqtt_enc.connect_request_encode: (0x20018af8): Encoding Protocol Version 04.
--- 5 messages dropped ---
[00:00:13.258,087] <dbg> net_mqtt_enc.pack_uint8: (0x20018af8): >> val:00 cur:0x20019843, end:0x20019a36
[00:00:13.258,087] <dbg> net_mqtt_enc.connect_request_encode: (0x20018af8): Encoding Keep Alive Time 003c.
[00:00:13.258,087] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:003c cur:0x20019844, end:0x20019a36
[00:00:13.258,117] <dbg> net_mqtt_enc: Encoding Client Id.
                                       6d 79 2d 63 6c 69 65 6e  74 2d 69 64             |my-clien t-id
[00:00:13.258,117] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:0000000e cur:0x20019846, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:000c cur:0x20019846, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc: Encoding Username.
                                       70 61 6c 64 65 6e 31 32  33                      |palden12 3
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:0000000b cur:0x20019854, end:0x20019a36
[00:00:13.258,148] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:0009 cur:0x20019854, end:0x20019a36
[00:00:13.258,178] <dbg> net_mqtt_enc: Encoding Password.
                                       31 32 33                                         |123
[00:00:13.258,178] <dbg> net_mqtt_enc.pack_utf8_str: (0x20018af8): >> str_size:00000005 cur:0x2001985f, end:0x20019a36
[00:00:13.258,178] <dbg> net_mqtt_enc.pack_uint16: (0x20018af8): >> val:0003 cur:0x2001985f, end:0x20019a36
[00:00:13.258,209] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20018af8): << msg type:0x10 length:0x00000029
[00:00:13.258,209] <dbg> net_mqtt_enc.packet_length_encode: (0x20018af8): >> length:0x00000029 cur:(nil), end:(nil)
[00:00:13.258,239] <dbg> net_mqtt_enc.mqtt_encode_fixed_header: (0x20018af8): Fixed header length = 02
[00:00:13.258,239] <dbg> net_mqtt_enc.pack_uint8: (0x20018af8): >> val:10 cur:0x20019839, end:0x20019a36
[00:00:13.258,270] <dbg> net_mqtt_enc.packet_length_encode: (0x20018af8): >> length:0x00000029 cur:0x2001983a, end:0x20019a36
[00:00:13.259,185] <dbg> net_mqtt.client_connect: (0x20018af8): Connect completed
[00:00:13.568,634] <dbg> net_mqtt.mqtt_input: (0x20018af8): state:0x00000002
[00:00:13.568,634] <err> net_mqtt_rx: [536974904]
[00:00:13.568,695] <dbg> net_mqtt_rx.mqtt_read_message_chunk: (0x20018af8): [CID 0x20018c20]: Connection closed.
[00:00:13.568,695] <dbg> net_mqtt_sock_tls.mqtt_client_tls_disconnect: (0x20018af8): Closing socket 1
[00:00:13.568,786] <inf> mqtt_simple: MQTT client disconnected: -128
[00:00:13.568,786] <err> mqtt_simple: mqtt_input: -128
[00:00:13.568,817] <inf> mqtt_simple: Disconnecting MQTT client...
[00:00:13.568,878] <err> mqtt_simple: Could not disconnect MQTT client: -128
[00:00:13.568,878] <inf> mqtt_simple: Reconnecting in 60 seconds...

on the client side.

And on the broker side I see: 

New connection from <ip address> on port 8883.

Client <unknown> disconnected due to protocol error.

I'm able to connect to the broker using the mosquitto_sub command. 

[Didrik Rokhaug]

leo_nam said:

We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10

That's great to hear!

leo_nam said:

But now the error I'm getting is

Do you have a modem trace showing this error?

[leo_nam]

I have a modem trace from the server side (Amazon Linux 2), I didn't save any of the ones I made client side (Nordic).

 server_pcap.pcap

I was wondering if I'd get something different if I did a trace from the server side but I didn't notice any difference. Let me know if you still need one from the client side and I will get it to you. 

Edit: I upgraded my broker to 2.0.11 but to do that I changed to a ubuntu server. However, same output from the server: "Client <unknown> disconnected due to protocol error." 

Here is the pcap from the nordic client.

 client_trace.pcapng