Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Thu, 27 Jun 2019 17:04:13 GMT

We eventually realized that the bulk of the issue, and likely the original root of my troubles, was that the buffers being used by the at_host library are simply too small in both directions, causing silent truncation and loss of data. There is now a ticket for a Nordic engineer to fix this, and I expect an official fix in the not too far future.

If you can't wait, start digging through the at_host code, looking in particular at CONFIG_AT_CMD_RESPONSE_MAX_LEN or CONFIG_AT_HOST_SOCKET_BUF_SIZE depending on what version you're at. I also recommend turning on CONFIG_LOG if you can support it so that the failures aren't so silent.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

Martin Lesund — Fri, 21 Jun 2019 13:55:18 GMT

Hi Justin,
I see that you have shared your trace in a private ticket and the traces will be followed up and analyses will be posted there.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 22:42:53 GMT

For Pete's sake! I'm 0 for 5 in getting this thing to hang while bringing up MQTT with BSD trace enabled and peer_verify set back to 2! (I was 5 for 5 earlier today with BSD trace disabled...)

Are these modem traces easily cropped? I'm considering just leaving the trace always running in the background while I go back to my other development work, but I don't want to make you guys sift through hours of trace when only the final minutes matter.

This feels a lot like an uninitialized variable in the modem firmware or maybe the certificates are supposed to be null-terminated and mine didn't get stored that way and it's semi-random what happens to be in memory after it...

Bah humbug!

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 22:09:05 GMT

Yep, this NRF9160 is definitely confused. Even with peer_verify=0, my system has been having random lockups all day while doing BSD library calls. I've been developing this application in some form or another for months and haven't had this much bad behavior in one day since I upgraded the original modem firmware.

Would a BSD trace be useful to the Nordic engineers?

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 17:41:48 GMT

Please if you could provide suggestions of how to force my modem credentials to a sane/empty state I would appreciate it. I've already tried "CMNG=1" and nrf_inbuilt_key_delete(), but those are apparently not enough by themselves.

I managed to get my certificates reloaded using the nrf_inbuilt methods after several attempts, but something is still messed up. The call to mqtt_connect now hangs unless I set peer_verify=0. My system had been running properly with peer_verify=2 for months before I started playing with certificates this week. Watching on the server side with "openssl s_server" I can see that the modem is using the cert I am expecting.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 15:24:02 GMT

Yeah, I was finally able to do a CMNG write command... The basic CMNG write is working, but something is hanging in the modem firmware during most writes.

I had loaded the test string "BEEPBEEP" as a CA using nrf_inbuilt_key_write, rebooted into at_client, and was able to read "BEEPBEEP" back using CMNG=2. I then set the CA to "HONKHONK" using CMNG=0 and read *that* back using CMNG=2. (Yay! Proves that it's not an XPMNG issue for that credential at least)

Enthused by that success, I immediately tried to use CMNG=0 to write a real certificate into the same slot, and that command hung. After rebooting the 9160, CMNG=2 still reads back "HONKHONK". I was able to change it to something short again using CMNG=0 and read that back okay. (This final test I also used a multi-line payload to prove that wasn't the issue...)

Is there a size limit on certificates/keys? I don't think mine are excessive in any way. A quick check at the command line puts all three credential files I'm trying to load in the area of 1600-1900 bytes.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 15:01:57 GMT

Yeah, I strongly suspect my modem NVM is in bad shape. Is there a way using nrf9160_mdm_dfu or some other tool that I can check or reset the credential storage?

It is now in a state where if I try to store credentials even using nrf_inbuilt_key_write some writes succeed and others hang. After one pass where it had written the CA cert successfully but then hung during the private key write, I reflashed to at_client and poked around a bit:

The AT host sample started
AT
OK
AT+CFUN=4
OK
AT+CMEE=1
OK
AT%CMNG=1
%CMNG: 64738,0,"0000000000000000000000000000000000000000000000000000000000000000"
%CMNG: 64738,1,"0101010101010101010101010101010101010101010101010101010101010101"
%CMNG: 64738,2,"0202020202020202020202020202020202020202020202020202020202020202"
OK
AT%CMNG=2,64738,0
%CMNG: 64738,0,"0000000000000000000000000000000000000000000000000000000000000000","-----BEGIN CERTIFICATE-----
MIIFLjCCBBagAwIBAgIJANuSwPfghLXlMA0GCSqGSIb3DQEBCwUAMIG7MQswCQYD
VQQGEwJVUzEWMBQGA1UECAwNTWFzc2FjaHVzZXR0czEUMBIGA1UEBwwLTWFybGJv
cm91Z2gxJjAkBgNVBAoMHVNpZ25hbEZpcmUgV2lyZWxlc3MgVGVsZW1ldHJ5MRMw
EQYDVQQLDApNUVRUIENsb3VkMRwwGgYDVQQDDBNTaWduYWxGaXJlIENsb3VkIENB
MSMwIQYJKoZIhvcNAQkBFhRpbmZvQHNpZ25hbC1maXJlLmNvbTAeFw0xOTA0MDMx
NjE1MDhaFw0zOTAzMjkxNjE1MDhaMIG7MQswCQYDVQQGEwJVUzEWMBQGA1UECAwN
TWFzc2FjaHVzZXR0czEUMBIGA1UEBwwLTWFybGJvcm91Z2gxJjAkBgNVBAoMHVNp
Z25hbEZpcmUgV2lyZWxlc3MgVGVsZW1ldHJ5MRMwEQYDVQQLDApNUVRUIENsb3Vk
MRwwGgYDVQQDDBNTaWduYWxGaXJlIENsb3VkIENBMSMwIQYJKoZIhvcNAQkBFhRp
bmZvQHNpZ25hbC1maXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAO9Kf1ufbdkCdIB9hJ5iZ1z9OyUB8Ddaal7uGFUh1j2y/Fk+gbO69YXUAX/8
CbZKiZ4VJmCcbCS9jRU7jNbF85oeJofkoiCpWB0supKtGk+GK6dMBK84TSJL5x29
qdGqu2x/em5UD9PfqSMQtdPpcCraPTekRAWJnUd3BNeYqZ3uixOWVIpytGWDLeZP
PU3UcIJIRSCbunVSzoafTrQI/FNh7azx9RTpJexEhLYRqKB54i8oaKdUQHfve9sD
M4b



AT
OK
AT%CMNG=3,64738,0
OK
AT%CMNG=1
%CMNG: 64738,1,"0101010101010101010101010101010101010101010101010101010101010101"
%CMNG: 64738,2,"0202020202020202020202020202020202020202020202020202020202020202"
OK
AT%CMNG=3,64738,1
OK
AT%CMNG=3,64738,2
OK
AT%CMNG=1
OK
AT%CMNG=0,64738,0,"TEST"

The CMNG=1 listed that the CA, key, and cert all existed, even though it had hung during the key write and the cert write had not been written. When trying to read the CA, it didn't have the whole CA certificate. I then deleted all three certificates and tried to write a super-short string to the CA as a test and that hung and never returned...

(EDIT: The truncated output on CMNG=2 was due to a too-small buffer in the AT driver, which can be remedied by increasing AT_HOST_SOCKET_BUF_SIZE in prj.conf)

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Wed, 19 Jun 2019 13:03:58 GMT

I have previously listed and deleted all keys using CMNG=1 and CMNG=3. I just tried it again and included a CFUN=0 and reboot after deleting the keys but before trying to write new keys with CMNG=0. It exhibited the same behavior and never completed the CMNG write command.

Regarding the XPMNG command, I tried XPMNG=2 and it does report an existing public key. I am quite certain I have never installed one. I happened to have a new dev kit we recently got for GPS testing, so I loaded it up with at_client and XPMNG=2 reports the exact same public key. I assume these must be from manufacturing on Nordic's end. There is no command I can find in the modem manual to delete the XPMNG credential, can you please tell me how to do this?

In case anyone else ever cares, this is what the preinstalled public key looked like:

AT%XPMNG=2
%XPMNG: "-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErrM7SxsE9WStx+6C2TQSsiaCnDww
B6rYZe/xHP7sDuHP8SmB0uauqhWBXXy0e8xoxqAc2bniubZa4HI2Zfz7tQ==
-----END PUBLIC KEY-----
"
OK

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

Martin Lesund — Wed, 19 Jun 2019 11:54:23 GMT

Hi Justin,
Could you try to list the keys (AT%CMNG=1) and delete the ones that you are not using.
It may be that there is a memory issue.

The error 519 is not listed in the documentation, but it means that it already exists something at that place, so you can try to delete the old one first.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Tue, 18 Jun 2019 14:59:32 GMT

I now wonder if I haven't somehow put this modem's NVM into a bad state. Just so I could continue doing other work, I used nrf_inbuilt_key_delete/nrf_inbuilt_key_write to remove all credentials and then reload a good set, and now every attempt to connect to an SSL socket fails with errno=-45. This is the same client/application code I have been using for months without issue.

I further tried using just "openssl s_server" as the listener and disabling peer verify on both sides, and I still get -45 from mqtt_connect. The openssl server just spits out this every time the device hits it:

ERROR
shutting down SSL
CONNECTION CLOSED

They never even begin key exchange. 😞

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Tue, 18 Jun 2019 14:32:11 GMT

I am unable to get XPMNG to work when pushing the public key to the modem. The CME error code is not listed in on the XPMNG documentation page.

***** Booting Zephyr OS v1.14.99-ncs1 *****
The AT host sample started
AT
OK
AT+CFUN=4
OK
AT+CMEE=1
OK
AT%XPMNG=0,"-----BEGIN PUBLIC KEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE0hFbFkKzORGXw5UCyFV6McEsQlH/SA+5OPTg5ff1DwQDj9LQJJ4dkcgdGI46bVho3YU7RQTVc9LojyUFNHNEew==-----END PUBLIC KEY-----"
+CME ERROR: 519

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

Martin Lesund — Tue, 18 Jun 2019 14:04:31 GMT

Please try to follow the Authenticating AT command usage doc, that may be the reason why.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Tue, 18 Jun 2019 13:52:09 GMT

I was working directly via a serial terminal at the moment, just as a proof of concept. I was getting the same behavior with both our application and the at_client sample.

Does CR/LF matter? Are line breaks required in the key material or could the entire thing be concatenated to one line if desired?

The doc page you referenced makes use of XSUDO. Base on the note on this page I had assumed that was not necessary yet, and possibly not fully implemented, so I had not been trying it. Am I likely to have more success if I use XSUDO?

I will try to match my certificate types and formatting more carefully to your reference and see if that changes anything.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

Martin Lesund — Tue, 18 Jun 2019 13:33:02 GMT

Hi Justin,
Are you sending AT-Commands in your application with sockets or are you sending directly via e.g. a serial terminal using the %CMNG command with at_client sample?

Note:

<content> in the read response is exactly what is written, including <CR>, <LF>, and other characters. The characters outside the double quotes are part of the AT response format.

<content> String. Mandatory if <opcode> is write. An empty string is not allowed. A Privacy Enhanced Mail (PEM) file enclosed in double quotes (X.509 PEM entities). Base64-encoded string in double quotes (PSK).

Please look at the examples in the doc page for reference.

RE: Do the AT%CMNG commands function properly for writing credentials in mfw_nrf9160_0.7.0-29.alpha

jbrzozoski — Mon, 17 Jun 2019 20:31:02 GMT

I have the feeling that the %CMNG write command is very unforgiving of certificate bodies that don't match the expected formatting, and it is hanging forever waiting to see a certain pattern it expects... But I can't for the life of me figure out what the pattern should be. The certificate files I'm using are accepted just fine by the nrf_inbuilt_key_write function...

(EDIT: This was definitely not the issue as the CMNG write command stores anything you send w/o doing any validation if things are working properly. Further down in the thread I stored "HONKHONK" as a CA certificate.)