MQTT with TLS on AWS EC2

RE: MQTT with TLS on AWS EC2

leo_nam — Tue, 24 May 2022 16:04:50 GMT

Hello,

The broker has been configured, and over the weekend flau was able to connect successfully over MQTT with TLS to the Amazon Linux 2 server running MQTT broker 1.6.10.

I was unable to connect using my mqtt_simple project with the development board. However, by restarting with a clean version of the mqtt_simple project by removing and then readding "nrf Connect SDK v1.8.0" and then making then necessary changes to enable tls and certificate provisioning, I was able to connect to the broker using the mqtt_simple project and the development board.

Thank you for your help,

Palden

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Mon, 23 May 2022 14:19:19 GMT

Thanks,

The traces shows that the TLS handshake is successful, but the server closes the connection after the client sends (presumably) the MQTT Connect message.

However, as the Connect message is sent after the TLS handshake, it is encrypted, so I can't check it for anything weird.

How have you configured your broker?

In most cases I coud on the internet with the same error ended up being configuration errors.

RE: MQTT with TLS on AWS EC2

leo_nam — Fri, 20 May 2022 15:37:14 GMT

I have a modem trace from the server side (Amazon Linux 2), I didn't save any of the ones I made client side (Nordic).

devzone.nordicsemi.com/.../server_5F00_pcap.pcap

I was wondering if I'd get something different if I did a trace from the server side but I didn't notice any difference. Let me know if you still need one from the client side and I will get it to you.

Edit: I upgraded my broker to 2.0.11 but to do that I changed to a ubuntu server. However, same output from the server: "Client <unknown> disconnected due to protocol error."

Here is the pcap from the nordic client.

devzone.nordicsemi.com/.../client_5F00_trace.pcapng

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Fri, 20 May 2022 14:18:37 GMT

[quote user="leo_nam"]We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10[/quote]

That's great to hear!

[quote user="leo_nam"]But now the error I'm getting is[/quote]

Do you have a modem trace showing this error?

RE: MQTT with TLS on AWS EC2

leo_nam — Wed, 18 May 2022 18:09:31 GMT

Hello Didrik,

We've gotten past the error 95 by upgrading our Mosquitto broker from 1.4.x to 1.6.10 which couldnt be done until we moved the broker to a newer version of AWS (Amazon Linux 2). The new broker has the required cipher suites for a tls connection.

But now the error I'm getting is

devzone.nordicsemi.com/.../4628.output.txt

on the client side.

And on the broker side I see:

New connection from <ip address> on port 8883.

Client <unknown> disconnected due to protocol error.

I'm able to connect to the broker using the mosquitto_sub command.

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Wed, 18 May 2022 14:31:48 GMT

That should be enough to enable SNI, though I can't see that it is enabled in the .pcap you shared.

Could you share the full URL of your server (either here or in a private message), so I can see what TLS features the server requires/supports?

RE: MQTT with TLS on AWS EC2

flau — Mon, 16 May 2022 13:43:58 GMT

Here is the tls_cfg in the main.c.

#if defined(CONFIG_MQTT_LIB_TLS)
struct mqtt_sec_config *tls_cfg = &(client->transport).tls.config;
static sec_tag_t sec_tag_list[] = { CONFIG_MQTT_TLS_SEC_TAG };

client->transport.type = MQTT_TRANSPORT_SECURE;

tls_cfg->peer_verify = CONFIG_MQTT_TLS_PEER_VERIFY;
tls_cfg->cipher_count = 0;
tls_cfg->cipher_list = NULL;
tls_cfg->sec_tag_count = ARRAY_SIZE(sec_tag_list);
tls_cfg->sec_tag_list = sec_tag_list;
tls_cfg->hostname = CONFIG_MQTT_BROKER_HOSTNAME;

#if defined(CONFIG_NRF_MODEM_LIB)
tls_cfg->session_cache = IS_ENABLED(CONFIG_MQTT_TLS_SESSION_CACHING) ?
TLS_SESSION_CACHE_ENABLED :
TLS_SESSION_CACHE_DISABLED;
#else
/* TLS session caching is not supported by the Zephyr network stack */
tls_cfg->session_cache = TLS_SESSION_CACHE_DISABLED;

#endif

#else
client->transport.type = MQTT_TRANSPORT_NON_SECURE;
#endif

And in prj.conf, I have

CONFIG_MQTT_BROKER_HOSTNAME="ec2-xxx-xxx-xxx-xxx.ca-central-1.compute.amazonaws.com"
CONFIG_MQTT_BROKER_PORT=8883

Regards,

Floyd

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Mon, 16 May 2022 13:09:58 GMT

Do you set the hostname in the tls_config struct?

Your server/AWS probably requires SNI support to route the packets correctly. You enable SNI by adding the hostname in the tls_config struct.

[quote user="flau"]If we set tls_cfg->cipher_count = 0 and tls_cfg->cipher_list = NULL, does it mean the modem will not use any of the 15 cipher suites?[/quote]

If the cipher_count is 0, those fields are ignored, and the stack will use all supported cipher suites.

RE: MQTT with TLS on AWS EC2

flau — Sun, 15 May 2022 22:39:38 GMT

We are running modem firmware v1.3.1. I suppose that means we have 15 different cipher suites to choose from. Am I correct?

If we change the cipher suite of our broker to match with one that is available with the nRF9160, what else do we need to do other than restart the broker? Do we need to generate a new sets of certificates, etc.?

If we set tls_cfg->cipher_count = 0 and tls_cfg->cipher_list = NULL, does it mean the modem will not use any of the 15 cipher suites?

RE: MQTT with TLS on AWS EC2

leo_nam — Thu, 12 May 2022 18:03:42 GMT

I was unable to connect to the broker with the nRF9160.

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Thu, 12 May 2022 16:00:02 GMT

[quote user="leo_nam"]I tried restarting the broker with a cipher that's available in both the mosquitto_sub and nrf modem cipher suite by adding 'ciphers ECDHE-ECDSA-AES128-SHA256' to the mosquitto.conf but then i got the no shared cipher message when using mosquitto_sub. I'm not really sure where to go from here[/quote]

Were you able to connect to the broker with the nRF9160 after changing the cipher suite of the broker?

RE: MQTT with TLS on AWS EC2

leo_nam — Wed, 11 May 2022 18:22:40 GMT

Hello, I'm working with flau on this problem.

devzone.nordicsemi.com/.../nrf9160_5F00_error95_5F00_trace.pcapng This is the trace of the modem when it receives the 'mqtt_connect -95' error.

And from the server side the broker outputs:

OpenSSL Error: error:1408A0C1:SSL routines: ssl3_get_client_hello:no shared cipher

I ran another trace while connecting successfully to the broker using mosquitto_sub command on powershell and when looking at the trace I get a 'Server Hello' with Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d) which isn't supported based on https://www.nordicsemi.com/Products/nRF9160/Download#infotabs.

I tried restarting the broker with a cipher that's available in both the mosquitto_sub and nrf modem cipher suite by adding 'ciphers ECDHE-ECDSA-AES128-SHA256' to the mosquitto.conf but then i got the no shared cipher message when using mosquitto_sub. I'm not really sure where to go from here.

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Fri, 06 May 2022 14:56:40 GMT

[quote user="flau"]Does Trace Collector v2 work on custom board, or nRF9160DK only? [/quote]

It should work for any boards that has a serial port.

[quote user="flau"]We put more log messages in the code (mainly in mqtt.c and mqtt_transport_socket_tls.c) and found that there is an error when calling the zsock_connect funcation. After that an error when calling the mqtt_transport_connect function. And finally the mqtt_connect -95 error.[/quote]

Yes, that's where it comes from. But to know why, we need to inspect the traffic between the modem and the server.

RE: MQTT with TLS on AWS EC2

flau — Wed, 04 May 2022 16:04:12 GMT

We put more log messages in the code (mainly in mqtt.c and mqtt_transport_socket_tls.c) and found that there is an error when calling the zsock_connect funcation. After that an error when calling the mqtt_transport_connect function. And finally the mqtt_connect -95 error.

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Wed, 04 May 2022 15:44:36 GMT

[quote user="flau"]Does Trace Collector v2 work on custom board, or nRF9160DK only? I have the tool installed but it only detects my nRF9160DK.[/quote]

It should be able to detect any serial devices, though I don't have a lot of non-Nordic boards to test with.

I have asked the developers to comment.

However, the Trace Collector V2 is also able to decode already captured traces. So you can use the "old" Trace Collector to capture the trace, then decode it with the Trace Collector v2 (when you haven't selected a device, there is a button for converting raw traces to pcap).

RE: MQTT with TLS on AWS EC2

flau — Wed, 04 May 2022 12:15:10 GMT

Thanks Didrik. Does Trace Collector v2 work on custom board, or nRF9160DK only? I have the tool installed but it only detects my nRF9160DK.

Anyway, I turned on the debug log on my board hoping to see more debug messages. Here is the terminal output.

[00:00:00.266,174] <inf> MQTT_SIMPLE: MQTT started
[00:00:00.271,209] <inf> MQTT_SIMPLE: Provisioning certificates
[00:00:01.728,515] <inf> MQTT_SIMPLE: Disabling PSM and eDRX
[00:00:01.739,288] <inf> MQTT_SIMPLE: LTE Link Connecting
[00:00:15.933,807] <inf> MQTT_SIMPLE: LTE Link Connected
[00:00:17.322,906] <inf> MQTT_SIMPLE: IPv4 Address found xxx.xxx.215.238
[00:00:17.330,444] <inf> MQTT_SIMPLE: TLS enabled
[00:00:17.335,815] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20014da0): Created socket 1
[00:00:17.953,979] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:17.959,625] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds
[00:00:47.974,792] <dbg> net_mqtt_sock_tls.mqtt_client_tls_connect: (0x20014da0): Created socket 1
[00:00:48.845,336] <err> MQTT_SIMPLE: mqtt_connect -95
[00:00:48.850,982] <inf> MQTT_SIMPLE: Reconnecting in 30 seconds

Regards,

Floyd

RE: MQTT with TLS on AWS EC2

Didrik Rokhaug — Wed, 04 May 2022 11:00:16 GMT

Hi,

[quote user=""]Am I missing anything? What have I done wrong?[/quote]

It is hard to say without seeing the IP traffic.

You can get the IP traffic if you capture a modem trace with the Trace Collector v2, and use either the "pcap" or "live" options.

My current guesses though, is that you either have written the wrong certificate to the device, or you are not providing the hostname of the server for SNI.

Again, a modem trace would be very helpful here.

Best regards,

Didrik