LWM2M + DTLS + NBIoT proper implementation

RE: LWM2M + DTLS + NBIoT proper implementation

jp — Fri, 05 Jun 2020 13:34:51 GMT

Thanks for all your work on this fastfox , your workaround worked for me, and the DTLS Connection ID extension was something I was not aware of.

Hoping Nordic will support this, as soon as they can, as it will save 100s of bytes every registration update.

Thanks,
John

RE: LWM2M + DTLS + NBIoT proper implementation

Jan Tore Guggedal — Wed, 03 Jun 2020 13:20:46 GMT

Yeah, I agree that there could be an option to skip straight to the session resumption.
It makes sense to keep the ACK timeout at 10 seconds, especially on NB-IoT networks.

BR,

Jan Tore

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 03 Jun 2020 12:36:30 GMT

Ok,

I already started to agree with you that this is LWM2M thing and I wrote and issue to zephyr to get some feedback about what they think

https://github.com/zephyrproject-rtos/zephyr/issues/25935

I have the CONFIG_COAP_INIT_ACK_TIMEOUT_MS already set to 10s, because with very small values you got bursts of retransmissions.

Anyway, I believe it would be better to go to the resumption immediately and not via some fail -> retry -> fail -> close&reopen scenario

RE: LWM2M + DTLS + NBIoT proper implementation

Jan Tore Guggedal — Wed, 03 Jun 2020 12:04:17 GMT

Adding a bit more to my previous answer, I didn't properly explaing why I think this is LwM2M (implementation) issue.

It looks like LwM2M has mechanism for retrying packet sends 3 times with doubling back-off time, starting at 10 seconds. After that it closes and reopens socket, which triggers DTLS resumption. So to properly work around this, there would have to be a way to detect that the NAT rules are removed. Usually you can assume this after ~60 seconds of UDP inactivity between client and server, but this is highly network dependent.

From the Leshan PCAP file, it looks like the client just assumes that the DTLS session is gone before attempting to send data. This is a viable workaround also for the nRF91 device, though I don't know if it's configurable or if you will have to make some code changes in the LwM2M engine.

Update: I poked around a bit, and this seems to actually be a CoAP thing. You can adjust CONFIG_COAP_INIT_ACK_TIMEOUT_MS in your prj.conf to reduce the retry time and have resumption kick in faster.

RE: LWM2M + DTLS + NBIoT proper implementation

Jan Tore Guggedal — Wed, 03 Jun 2020 11:34:57 GMT

The session resumption happens because the (D)TLS session caching socket option is set by default on the version of BSD library that you're using. So if you stick to that, the modem will always try to resume the previoud session and let the server decide what to do.

So this mainly sounds like an LwM2M issue now that it looks like session resumption is working as intended. Let me know if you don't fully agree on this.
My familiarity with LwM2M is unfortunately very limited, so I'm not much help on that front.
I know the issue has been reported internally, which will hopefully give results on the LwM2M debugging.

Best regards,

Jan Tore

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 03 Jun 2020 11:17:06 GMT

Indeed, I completely missed that session ID in packet #42.

What happens in the upper layer(lwm2m) is that registration update timeouts(because server does not respond). Registration update and retries are in packets 38, 39 and 40.

After this lwm2m closes the socket(which causes the Alert, packet 41) and starts a new socket to re-register. This socket open causes the Client hello(42), which, as you said, looks like resume.

Don't know what to think here. Something obviously triggers the resume, but I hope it is not only closing and opening the socket. I could try to hack lwm2m to do that anyway

I also need to continue working to get those modem traces...

RE: LWM2M + DTLS + NBIoT proper implementation

Jan Tore Guggedal — Wed, 03 Jun 2020 10:54:21 GMT

Hi,

I looked up that nrfxlib version, and it is with TLS caching enabled by default.

Thanks for claryfying the bootstrap at the beginning. I think the Encrypted alert is sent to notify the server that the connection can be closed. The server also responds in packet 21. I'm not sure about the error message in the application log, unfortunately. It seems to happen after the bootstrap procedure is completed, so it could be unrelated to these issues. I think we would need modem traces from the device side to get to the bottom of that one.

I'm starting to wonder if might have misunderstood the issue here, as I see session resumption is being used in ip_port_changes_90s.pcapng:

Looking at the next time the NAT routing has timed out (new src port), we get the seconds Encrypted alert in packet 41. After that, session resumption is attemped in client hello in packet 42. The server responds that it recognizes the session ID in packet 43, and the session is resume, and both server and client notify each other to change cipher spec.

Please correct me if I'm missing the issue here.

Thanks,

Jan Tore

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 03 Jun 2020 10:11:16 GMT

Hi,

Did you notice that the connection that ends to alerts is a separate connection to different port? It is actually the bootstrap server connection. Maybe I should try to do the connection completely without the bootstrap.

nrfxlib is at this version https://github.com/nrfconnect/sdk-nrfxlib/commit/7d5e7624ec7fdb071ab8def3ae772e5ee5b65f21

and yes, there is one error printed when boostrap operation completes

00> [00:00:37.000,762] <inf> net_lwm2m_rd_client: Bootstrap data transfer done!
00> [network2.c] [0x2002302c] D Bootstrap transfer complete
00> [network2.c] [0x2002302c] D '0/1/0': 'coaps://lwm2m.ycfwydu.com:8601' '0x00'
00> [00:00:37.001,159] <dbg> net_lwm2m_engine.lwm2m_engine_get: path:1/0/0, buf:0x20029e0e, buflen:2
00> [00:00:37.001,251] <dbg> net_lwm2m_engine.lwm2m_engine_get: path:1/0/1, buf:0x20029e0e, buflen:2
00> [00:00:37.001,678] <err> net_lwm2m_engine: Err sending response: -22

RE: LWM2M + DTLS + NBIoT proper implementation

Jan Tore Guggedal — Wed, 03 Jun 2020 09:26:17 GMT

Hi,

I had a look at ip_port_changes_90s.pcapng, and it seems like (D)TLS caching is already enabled. The session ID is at least provided in the client hello message, but it seems like the server side has closed that session, possibly because of the preceeding alert message. You can check which nrfxlib revision you are using by executing "west status" which should print the revision of all repositories west has imported for you. That will show if you have the revision where TLS caching is enabled by default or not.

The alert messages from the device are curious, and looks likely to be the source of trouble here. Could you try to enable CONFIG_LWM2M_LOG_LEVEL_DBG=y and see if that yields any useful information, in particular related to the alert message?

Best regards,

Jan Tore

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 03 Jun 2020 06:12:21 GMT

I did try to set cache before like this:

nrf_sec_session_cache_t session_cache = 1;
if (nrf_setsockopt(client_ctx->sock_fd, NRF_SOL_SECURE, NRF_SO_SEC_SESSION_CACHE, &session_cache, sizeof(session_cache)) < 0) {
	LOG_ERR("setsockopt NRF_SO_SEC_SESSION_CACHE: %d", errno);
}

Right after this setsockopt https://github.com/zephyrproject-rtos/zephyr/blob/358dcc1bde6a79abb3167e994fe7aec66968adaf/subsys/net/lib/lwm2m/lwm2m_engine.c#L4241

But that had no effect, and I just retried and confirmed that

I can see TLS_SESSION_CACHE was just added to master, so I can't use that in 1.2. Does it make any difference in which way the opt is set? Should I try it on master?(I assume that the problem lies in modem side)

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Tue, 02 Jun 2020 10:52:26 GMT

DO you see a different behavior if you set the TLS_SESSION_CACHE socket option explicitly?

https://github.com/nrfconnect/sdk-zephyr/blob/master/include/net/socket.h#L126

Edit: Note that the option must be set using setsockopt(), not the non-offloaded nrf_setsockopt().

Or, you can use nrf_setsockopt() and NRF_SO_SEC_SESSION_CACHE.

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Fri, 29 May 2020 10:13:14 GMT

Hi, both of you.

These are good questions, but I am not familiar enough with the fine details to give an answer myself.

I have forwarded them to our developers and will get back to you when I have an answer.

Best regards,

Didrik

RE: LWM2M + DTLS + NBIoT proper implementation

jp — Thu, 28 May 2020 10:21:48 GMT

Yes, I think it needs to be a configurable timeout, as it is network dependent, and using the worst case of 20 seconds would cut short the access window from the server, if your network keeps the session for 2 minutes for example.

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Thu, 28 May 2020 06:38:49 GMT

I've tried to read all the related RFCs bacwards and fowards, but can't really find any detail spec/timer/something about "When should client do session resumption" There are some hints e.g. in https://tools.ietf.org/html/rfc7925

For the case where the IP address and port number
changes, it is necessary to recreate the record layer using
session resumption.

and LWM2M spec itself http://openmobilealliance.org/release/LightweightM2M/V1_0_2-20180209-A/OMA-TS-LightweightM2M-V1_0_2-20180209-A.pdf

Note: During
the time the LwM2M Client has been sleeping the IP address assigned to it may have been released and / or existing
NAT bindings may have been released. If this is the case, then the client needs to re-run the DTLS handshake with
the LwM2M Server since an IP address and/or port number change will destroy the existing security context. For
performance and efficiency reasons it is RECOMMENDED to utilize the DTLS session resumption.

So my conclusion is that it is CLIENT side implementation detail about when to do resumption. And since client can't really know when IP address changes, I would assume that NRF91 has some internal logic to start the resumption. Whether it is a hardcoded timeout, configurable timeout or something else, I hope that Didrik Rokhaug can shed some more light here

RE: LWM2M + DTLS + NBIoT proper implementation

jp — Wed, 27 May 2020 16:21:57 GMT

Hi Didrik, how can the server reject the DTLS session when the communication is coming from a different IP address, and it doesn't recognise the packet?

Is there some way for the nRF9160 to invalidate the DTLS session?

Thanks,
John

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 27 May 2020 12:17:50 GMT

That is a good question, I need to investigate that.

I thought that it is up to the client to decide that, but it might very well be that it is somehow negotiated in initial handshake. Is there any default values for this timeout in modem or can it be adjusted somehow?

I am not even sure what is the term to look for here. I think RFCs are talking about "idle timeout" and "session timeout" both meaning that session is no longer valid. But what we need is "do resumption timeout", which is network depended and always smaller than "idle timeout"🤔

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Wed, 27 May 2020 09:30:07 GMT

Do you know what the DTLS timeout on the server is?

The modem assumes that the connection is valid, and will not initiate a new DTLS session unless the server rejects the old one.

So what might have happened is that the modem tries to use the "old" session, which is still valid. That means that a new DTLS handshake is unnecessary, hence no client hello.

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Wed, 27 May 2020 08:56:17 GMT

Hi, I already tried to take modem traces, but failed to figure out exactly how to do that in my custom PCB. I'll give it another try or rewrite the SW to evaluation board.

[quote userid="81181" url="~/f/nordic-q-a/61438/lwm2m-dtls-nbiot-proper-implementation/251849"][/quote]

Are you sure that you have configured the client and the server to use TLS?

Where the data encrypted?

Or, could it be that you simply missed the client hello?

Yes, I am sure. But it is DTSL to be exact.

Yes, data is encrypted

Possible of course, but very unlikely since I have tested this tens of times. I'll attach two pcaps here. From there(90s) you can see that packet #37 at time 102 is application data. This is the registration update. With a working client(leshan_client) the same registration update starts with client hello + resumptiondevzone.nordicsemi.com/.../ip_5F00_port_5F00_changes_5F00_90s.pcapng devzone.nordicsemi.com/.../ip_5F00_port_5F00_changes_5F00_leshan_5F00_client.pcapng

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Wed, 27 May 2020 08:29:39 GMT

If you take a modem trace, we can see what is happening from the device side as well.

However, if the modem provides a session ID in the client hello message, that is the session cache in action. If the server still recognizes the ID, then a full TSL negotiation is not needed.

[quote user="fastfox"]If I connect to my server using NRF91, I can only see application data(no client hello and thus no session resumption) when it is making registration update[/quote]

Are you sure that you have configured the client and the server to use TLS?

Where the data encrypted?

Or, could it be that you simply missed the client hello?

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Mon, 25 May 2020 11:21:00 GMT

Hi,

This is just not sinking into my brains now 😞

I am using 1.2.0 modem firmware and tag v1.2.0 from nrf-connect SDK

If I connect to my server using, e.g. leshan client, session resumption works and I see that the client is sending client hello when making registration update

If I connect to my server using NRF91, I can only see application data(no client hello and thus no session resumption) when it is making registration update

And I can't find any code that would set any session caches in nrf-connect SDK(or any of the sub repositories).

So you say that session cache is enabled by default, but it does not look like that from what I see.

Edit: While staring at the pcaps, I also realized that modem is sending session ID in the very first client hello to the server. I really mean the first message to the server after device power-cycle. That is strange...

Edit2: Ok that was probably TCP+TLS connection to the same domain name that caused the session ID to exist. Unfortunately, removing another connection does not make it work any better.

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Tue, 19 May 2020 11:38:15 GMT

[quote user="fastfox"]And the ?random? crashes caused by activating the cache will be fixed in 1.3.0, which will be release at unknown date?[/quote]

I am sorry, but my memory failed me a little. The bug was fixed in mwf v1.2.0, so the fix is already out.

If you are using NCS v1.2.0, TLS session caching is enabled by default.

[quote user="fastfox"]What is the API?[/quote]

Here is you would disable session caching:

#include <nrf_socket.h>

nrf_sec_session_cache_t session_cache = 0;

nrf_setsockopt(socket, NRF_SOL_SECURE,

        NRF_SO_SEC_SESSION_CACHE, &session_cache,

        sizeof(session_cache));

RE: LWM2M + DTLS + NBIoT proper implementation

fastfox — Tue, 19 May 2020 10:22:38 GMT

Thanks for clarifications. Just to make sure that I understood this correctly

[quote userid="81181" url="~/f/nordic-q-a/61438/lwm2m-dtls-nbiot-proper-implementation/250430"][/quote]

It is, but at the moment, there is a bug that can casue the modem to crash when (D)TLS session caching is enabled. The bug will be fixed in the next modem release.

As I am not seeing session resumption, it means that I have to explicitly enable the session caching somehow, right? What is the API? I found this post where it says that you have to use nrf_setsock_opt https://devzone.nordicsemi.com/f/nordic-q-a/55335/dtls-session-resumption-on-nrf9160-modem-fw-v1-1-0

And the ?random? crashes caused by activating the cache will be fixed in 1.3.0, which will be release at unknown date?

And to make this work right now, the only option is to not use offloaded NRF sockets at all, but something else instead, right?

RE: LWM2M + DTLS + NBIoT proper implementation

Didrik Rokhaug — Mon, 18 May 2020 12:08:31 GMT

Hi.

[quote user=""]Question 1: Operator NAT/firewall rules and NRF9160 DTLS implementation[/quote]

In the DTLS header, there will be a session token indicating which DTLS session the packet belongs to.

It is up to the server to decide if the session has timed out or not. So as long as all communication is initiated by the device, the NAT/firewall rules should not be a huge problem.

[quote user=""]It looks like resumption does not work in NRF or is not implemented at all?[/quote]

It is, but at the moment, there is a bug that can casue the modem to crash when (D)TLS session caching is enabled. The bug will be fixed in the next modem release.

[quote user=""]Question 2: It looks like DTLS connection(to LWM2M bootstrap server) always ends to TLS ALERT sent by nrf91. Any idea why? Is this a feature? Other clients do not do that.[/quote]

While I can not say for certain without more information (a modem trace is the most helpfull), I have two hyptohesis:

1. The server's certificates are too large. The nRF9160 only supports TLS frames up to 2048 bytes.

2. The nRF9160 does not support SNI at the moment. This will be added in the next modem release.

[quote user=""]Question 3: Zephyr does not support QUEUE mode. If I understood the LWM2M spec correctly, QUEUE mode is exactly what low power NB-IoT application needs, but zephyr LWM2M stack does not support it. Do you have any idea how Nordic users tend to solve this? Implement queues as server side application on top of LWM2M? Use different LWM2M client?[/quote]

Please take a look at https://github.com/NordicPlayground/fw-nrfconnect-nrf/pull/2252

Best regards,

Didrik