MQTT 128kB packet size

RE: MQTT 128kB packet size

Didrik Rokhaug — Thu, 30 Sep 2021 15:05:02 GMT

Actually, it turns out the POLLHUP issue has already been fixed in upstream Zephyr.

It will be included in the NCS fork in the next upmerge, and be included in NCS 1.8.0.

You can find the fix here: https://github.com/zephyrproject-rtos/zephyr/pull/37794

It should apply cleanly to NCS v1.7.0.

RE: MQTT 128kB packet size

Didrik Rokhaug — Thu, 30 Sep 2021 08:14:51 GMT

Hi, and again, sorry for this taking so long.

But, I have some good news this time: I have found a solution/workaround for the POLLHUP problem.

After some digging, it turned out that the connection to AWS is fine. While I still don't understand 100% why the POLLHUP flag is set, I believe that either the POLLHUP flag itself is a bug, or the application's handling of the POLLHUP is wrong. I'll have to discuss this more with our developers before I can be sure which one it is.

Regardless, you can simply ignore the POLLHUP, and everthing will work fine.

What I did was to add a 'continue' inside the if that checks if fds.revents has the POLLIN flag set, after the call to mqtt_input (and checking the return value) in mqtt_thread_fn in slm_at_mqtt.c.

In an unmodified SLM, it would be here: https://github.com/nrfconnect/sdk-nrf/blob/master/applications/serial_lte_modem/src/mqtt_c/slm_at_mqtt.c#L237

I'll try to keep you updated as we open PRs to make AWS connections with native TLS supported.

Best regards,

Didrik

RE: MQTT 128kB packet size

JWizard — Tue, 24 Aug 2021 21:37:30 GMT

I was able to get past the handshake by increasing the CONFIG_MBEDTLS_HEAP_SIZE. In my case to 81920.

Now the error is POLLHUP. From what I can tell at this moment this occurs immediately, the first time the socket is polled.

RE: MQTT 128kB packet size

JWizard — Tue, 24 Aug 2021 15:01:42 GMT

Thanks Didrik Rokhaug The Only reply button seems to be at the top so I'm replying to

[quote userid="81181" url="~/f/nordic-q-a/73647/mqtt-128kb-packet-size/325810#325810"]

Hi,

Unfortunately, I have not been able to make much progress. I have been quite busy, especially the past couple of months, due to low staffing during the summer acation period.

I am truly sorry for not having been able to work more on this.

However, staffing should soon return to normal, which will make it easier to find the time needed to look properly into this issue.

There has also been a lot of work done on the crypto side, with the inclusion of Trusted Firmware-M.

This includes adding a sample to the master branch that shows how you can run the TLS stack in the secure domain: https://github.com/nrfconnect/sdk-nrf/tree/master/samples/crypto/psa_tls

With sincere apologies,

Didrik

[/quote]

Understood. I will check that out. In the meantime I have found what looks like the point where my error first arises in rsa.c. You can see the Call stack and the line where the error code comes from in this image.

Watch on "ret" doesn't work here but I do believe the value is -0x10 = MBEDTLS_ERR_MPI_ALLOC_FAILED.

Hopefully that might be of use in the future.

As a side note it's pretty difficult to step through this code because it is optimized and there is only Common Configuration in the drop down. Is it possible to run this SLM application in an unoptimized Debug configuration?

RE: MQTT 128kB packet size

Didrik Rokhaug — Thu, 19 Aug 2021 18:21:54 GMT

Hi,

Unfortunately, I have not been able to make much progress. I have been quite busy, especially the past couple of months, due to low staffing during the summer acation period.

I am truly sorry for not having been able to work more on this.

However, staffing should soon return to normal, which will make it easier to find the time needed to look properly into this issue.

There has also been a lot of work done on the crypto side, with the inclusion of Trusted Firmware-M.

This includes adding a sample to the master branch that shows how you can run the TLS stack in the secure domain: https://github.com/nrfconnect/sdk-nrf/tree/master/samples/crypto/psa_tls

With sincere apologies,

Didrik

RE: MQTT 128kB packet size

JWizard — Wed, 18 Aug 2021 20:58:04 GMT

Hello Didrik Rokhaug

I haven't received a reply in some time. I have attempted to repeat the process in NCS 1.6.1 but found the same results. Are there any updates on this topic? Thanks, Jack

RE: MQTT 128kB packet size

Didrik Rokhaug — Mon, 03 May 2021 17:11:00 GMT

I am able to write the certificates with native TLS using the Certificate Manager in the LTE Link Monitor application.

However, I did have a problem reading the private key back out when I tried to use it. The problem was that MAX_CRDL_LEN in slm_native_tls.c was too small to hold all three certificate types.

But, the TLS handshake is still failing. I'll keep trying to debug this, and let you know when I know more (hopefully it won't be this long until my next update).

Is did you use the code you attached to try to write the certificates?

If so, you must write all certificate types as MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN, so that they can be read back out.

RE: MQTT 128kB packet size

JWizard — Wed, 28 Apr 2021 22:31:10 GMT

Thank you Didrik.

I think I misunderstood the mapping, however, I have tried using the Native TLS version to write the credentials and I could not connect to my Broker. It is possible that I did not format the creds correctly.

So I took the mapping knowledge and I used a non-native TLS App to write the creds to SecTag = 240.

I reloaded the NativeTLS application and I do read my type 0 credential with AT%CMNG=2,24,0.

However It still is not connecting. Interestingly I get this output. The first time I am using the correct port and the second time I used an incorrect port and got different output.

I tried again this time incrementing the SEC_TAG (per the mapping function) when I use the modem_key_mgmt_write function in a non-native TLS app.

    err = modem_key_mgmt_write(CONFIG_MQTT_TLS_SEC_TAG,
				   MODEM_KEY_MGMT_CRED_TYPE_CA_CHAIN,
				   CA_CERTIFICATE,
				   strlen(CA_CERTIFICATE));
	if (err) {
		LOG_ERR("Failed to provision CA certificate: %d", err);
		return err;

        
	}
        
        err = modem_key_mgmt_write(CONFIG_MQTT_TLS_SEC_TAG+1,
				   MODEM_KEY_MGMT_CRED_TYPE_PUBLIC_CERT,
				   CLIENT_PUBLIC_CERTIFICATE,
				   strlen(CLIENT_PUBLIC_CERTIFICATE));
	if (err) {
		LOG_ERR("Failed to provision public certificate: %d", err);
		return err;
        }

        err = modem_key_mgmt_write(CONFIG_MQTT_TLS_SEC_TAG+2,
				   MODEM_KEY_MGMT_CRED_TYPE_PRIVATE_CERT,
				   CLIENT_PRIVATE_KEY,
				   strlen(CLIENT_PRIVATE_KEY));
	if (err) {
		LOG_ERR("Failed to provision private key: %d", err);
		return err;
        }

Results were the same.

RE: MQTT 128kB packet size

Didrik Rokhaug — Wed, 28 Apr 2021 14:46:55 GMT

The CLAC problem has been fixed in the fork the SLM developers are working on.

If I have understood their fix correctly, they do it by only checking for CLAC responses of the second line of the respons, rather than the first.

[quote user="JWizard"]And now I am at least getting through to something happening and getting "FAILED! modem_key_mgmt_read() = -22" in my log.[/quote]

Did you write a certificate (with the SLM configured to use native TLS) to that sec_tag first?

When native TLS is enabled, the SLM remaps the sec_tags, so if the certificates was written without the remapping, you have to map backwards to find the new sec_tag you must use with %CMNG.

You can find the mapping function here: https://github.com/nrfconnect/sdk-nrf/blob/v1.5.0/applications/serial_lte_modem/src/slm_native_tls.c#L21

I'll continue to work on using the native TLS stack for connecting to an MQTT broker tomorrow.

RE: MQTT 128kB packet size

Didrik Rokhaug — Tue, 27 Apr 2021 14:52:46 GMT

Hi, and sorry for the late reply.

[quote user="JWizard"]
The documentation of SLM states I "must" use the overridden %CMNG command to provision credentials. What does this mean exactly?[/quote]

Normally, %CMNG writes the credentials to the modem, where they cannot be read by the application (except for the root CA certificates).

The overridden %CMNG command is used automatically when you enable CONFIG_SLM_NATIVE_TLS, so it is not really something you have to think about, except for %CMNG=1 not being supported in the overridden version.

[quote user="JWizard"]There is a comment in slm_at_cmng.c that says it should be like this: AT#XCMNG, but that doesn't change the outcome.[/quote]

I believe AT#XCMNG was used in an earlier version, though I might be mistaken.

[quote user="JWizard"]For good measure I ran CLAC and AT%CMNG does in fact appear in the list of available commands, AT#XCMNG does not.[/quote]

Anyway, if it was supported, I would expect it to be listed by #CLAC, not %CLAC.

[quote user="JWizard"]Do I need to issue %CMNG with the write opcode for each of my three credentials every time I start the application?[/quote]

No. The credentials are stored in the modem as CA certificates, so that the application can read them out later if it needs them.

[quote user="JWizard"]AT%CMNG=2,24,0 fails at the first check in handle_at_xcmng. Line 73 of slm_at_cmng.c. I edited the LOGERR to differentiate it from the others.[/quote]

Yes, I get this as well.

[quote user="JWizard"]Testing with other known commands such as AT+CFUN seems to show these commands are not parsed in at_parse_param of at_cmd_parser.c. Special commands of the structure AT#X... are parsed in this function, and it is apparent that the 'X' is specifically meant to differentiate a command subject to this parser from a CLAC response.[/quote]

"normal" AT commands are sent to the modem, where they are parsed and handled. The exception here is %CMNG, which normally would be sent to the modem, but we now want to intercept so that we can do something else with it. AT#-commands on the other hand are implemented by the SLM on the application core, and must therefore be parsed by the application.

The 'X' is used to differentiate between standard AT commands specified by 3GPP, and custom commands added by us.

[quote user="JWizard"]So I don't think this is probably how you should do this but I edited is_clac in at_utils.h to read:[/quote]

I haven't fully wrapped my head around the parser logic, and why CLAC responses must be detected. But it is not unreasonable for is_clac to trigger here, as you wouldn't encounter AT%<non 'X'> anywhere normally, except for this special case where we want to override an AT command, while using a parser not made to handle that scenario.

I'll discuss the best way to handle this with our developers tomorrow, along with looking into your other problems with using the credentials.

Best regards,

Didrik

RE: MQTT 128kB packet size

JWizard — Tue, 20 Apr 2021 19:01:36 GMT

Thank you for the clarifications Didrik. I did not expect it to be easy, but your reply has helped a lot in getting started.

If half of the RAM is still available while using the SLM then yes that is exactly what I should try and accomplish.

But I am first trying to establish a connection with the MQTT broker using TLS in the application core. I have not been successful and I think it has something to do with credentials. I changed "overlay-native_tls.conf" so it looks like this

# To build serial LTE modem native TLS socket support
# add the following to your west build command:
# -DOVERLAY_CONFIG=overlay-native_tls.conf
#

# TLS configuration
CONFIG_SLM_NATIVE_TLS=y
CONFIG_MODEM_KEY_MGMT=y
CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_LIBRARY=y
CONFIG_MBEDTLS_TLS_LIBRARY=y
CONFIG_MBEDTLS_PKCS1_V15=y
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_INSTALL_PATH="DUMMY"
# If larger TLS buffer is required for large CA chain,
# increase CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN to 4096
# and CONFIG_MBEDTLS_HEAP_SIZE to 32768
#CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=1280
#CONFIG_MBEDTLS_HEAP_SIZE=23040
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=4096
CONFIG_MBEDTLS_HEAP_SIZE=32768
CONFIG_NET_SOCKETS_OFFLOAD_TLS=n
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_NET_SOCKETS_TLS_MAX_CONTEXTS=2
# Increase extra FD entry for TLS contexts(2)
CONFIG_POSIX_MAX_FDS=10
CONFIG_NORDIC_SECURITY_BACKEND=y
CONFIG_NRF_SECURITY_ADVANCED=y
# Enable Socket Logging for debug
#CONFIG_NET_LOG=y
#CONFIG_NET_SOCKETS_LOG_LEVEL_DBG=y

according to the note in the middle of this file about large CA chains (in case that refers to me.) And I build with -DOVERLAY_CONFIG=overlay-native_tls.conf

The documentation of SLM states I "must" use the overridden %CMNG command to provision credentials. What does this mean exactly? Do I need to issue %CMNG with the write opcode for each of my three credentials every time I start the application?

This is what I have been trying to do but without success. I was thinking I have not found the correct way to format the certificates but then I discovered that every %CMNG command is returning a simple "ERROR."

For example if I try to read a CA Root Cert with security tag 24: AT%CMNG=2,24,0 will return ERROR. I have not been able to get a success or even a +CME Error Code

I have set CFUN=4 like it says to do if I want to write a credential.

There is a comment in slm_at_cmng.c that says it should be like this: AT#XCMNG, but that doesn't change the outcome.

For good measure I ran CLAC and AT%CMNG does in fact appear in the list of available commands, AT#XCMNG does not.

Am I misunderstanding something about the overridden %CMNG command?

Update: AT%CMNG=2,24,0 fails at the first check in handle_at_xcmng. Line 73 of slm_at_cmng.c. I edited the LOGERR to differentiate it from the others.

static int handle_at_xcmng(enum at_cmd_type cmd_type)
{
int err = -EINVAL;
uint16_t op, type;
nrf_sec_tag_t sec_tag;
uint8_t *content;
size_t len = CONFIG_AT_CMD_RESPONSE_MAX_LEN;

switch (cmd_type) {
case AT_CMD_TYPE_SET_COMMAND:
if (at_params_valid_count_get(&at_param_list) < 2) {
LOG_ERR("Parameter missed at first get count");
return -EINVAL;
} ... Update:I am struggling to wrap my mind around the full operation of the at_parser, however it appears to me that this AT%CMNG is being treated as a CLAC response, which results in the at_params_list containing just one element, the whole string.

Testing with other known commands such as AT+CFUN seems to show these commands are not parsed in at_parse_param of at_cmd_parser.c. Special commands of the structure AT#X... are parsed in this function, and it is apparent that the 'X' is specifically meant to differentiate a command subject to this parser from a CLAC response.

So I don't think this is probably how you should do this but I edited is_clac in at_utils.h to read:

static bool is_clac(const char *str)
{
if (strlen(str) < 4) {
return false;
}

if ((toupper(str[0]) != 'A') || (toupper(str[1]) != 'T')) {
/* Not an AT command */
return false;
}

if ((toupper(str[2]) != '+') && (toupper(str[2]) != '%')) {
/* Neither AT+ nor AT% */
return false;
}

if ((toupper(str[2]) == '%') && ((toupper(str[3]) == 'X') ||
(strlen(str) >= 7) && (toupper(str[3]) == 'C') && (toupper(str[4]) == 'M')
&& (toupper(str[5]) == 'N') && (toupper(str[6]) == 'G'))){
/* Ignore AT%X to avoid false detect (read resp XCOEX0 etc.) */
/* Ignore CMNG */
return false;
}

return true;
}

And now I am at least getting through to something happening and getting "FAILED! modem_key_mgmt_read() = -22" in my log.

-----------------------------------------------

Update:

It seems that provisioning credentials is now possible with the is_clac edits. I have tried many ways of formatting the credentials but I always get this message when I try to connect to the broker whether I use the URL or the IP directly:

RE: MQTT 128kB packet size

Didrik Rokhaug — Thu, 08 Apr 2021 15:35:17 GMT

Hi,

[quote user=""]When I publish a message of this size from an external device my nrf9160 device outputs the error -128. What is this error?[/quote]

ENOTCONN.

You get this error because the nRF9160 modem does not support TLS fragments this large, and closes the connection.

[quote user=""] 128 does not appear in errno.h.[/quote]

Yes, it does: https://github.com/eblot/newlib/blob/2a63fa0fd26ffb6603f69d9e369e944fe449c246/newlib/libc/include/sys/errno.h#L161

Your confusion comes from the fact that Zephyr and NCS supports two different implementations of libc. Zephyr has it's own minimal implementation (which is the default), but it also supports the Newlib implementation which is distributed with the compiler.

These two implementations uses different values for the error codes.

The aws_iot sample has enabled the Newlib implementation by setting CONFIG_NEWLIB_LIBC=y in the prj.conf file.

[quote user=""]How can I allow the maximum packet size of 128kB from AWS IoT to be received by the nrf9160?[/quote]

Realistically, you can't.

The modem only supports TLS fragments up to ~2kB.

You can also run the TLS stack on the application core, instead of in the modem. This will let you increase the buffer size, but a buffer size of 128kB would be half the RAM available on the nRF9160 (if you are interested in this, take a look at how it is done in the SLM).

You can read about how the nRF Asset Tracker works around this limitation here: https://nordicsemiconductor.github.io/asset-tracker-cloud-docs/v1.5.x/docs/aws/LimitingShadowDocumentSize.html

Best regards,

Didrik