Certificate upload via AT_Client cause freeze

[HoovinSchoovin]

Hello, 

I am trying to upload certificates to my nrf9160 chip on a custom board. I have done the following:

1. Loaded AT_CLIENT from the samples

2. Tested all AT commands are working nominally (AT, CFUN etc)

3. Forced the modem into offline state (AT+CFUN=4) as required

4. In the certificate manager view, copied in all certificates and added a custom security tag (these are not nrf cloud certs)

5. Hit update certificates

This causes the console to say "Updating CA certificate..." and get stuck forever. 

I then wanted to check by manually adding the certs via: AT%CMNG=0, 201, 0,"CERT" but the AT_Client freezes. 

I have been able to upload via CLI before (bypassing the AT commands) and this worked, but hoping to overcome the AT command issue. 

I am interacting with AT commands via UART connected to the board rather than USB with the DKs, could this be an issue? 

Any help in diagnosing would be greatly appreciated. 

Sam

[HoovinSchoovin]

Update:
I managed to bypass the AT Client to push up the certificates via nrfcredstore. 

I did the following:

nrfcredstore /dev/tty.usbserial-B002CZY5 write 201 ROOT_CA_CERT AmazonRootCA1.pem
nrfcredstore /dev/tty.usbserial-B002CZY5 write 201 CLIENT_KEY XXX-private.pem.key
nrfcredstore /dev/tty.usbserial-B002CZY5 write 201 CLIENT_CERT XXX-certificate.pem.crt

And confirmed they are there using AT%CMNG=1 .  

I then changed the following in proj.conf on the AWS sample

CONFIG_AWS_IOT_CLIENT_ID_STATIC="<15-number thingy code>"

CONFIG_AWS_IOT_BROKER_HOST_NAME="<URL>"

CONFIG_MQTT_HELPER_SEC_TAG=201

But the sample stops and freezes here:

*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.289,123] <inf> aws_iot_sample: The AWS IoT sample started, version: v1.0.0
[00:00:00.289,154] <inf> aws_iot_sample: Bringing network interface up and connecting to the network
[00:00:00.289,459] <inf> aws_iot: in aws_iot_init
[00:00:00.289,459] <inf> aws_iot: in about to go mqtt_helper_init
[00:00:00.289,489] <inf> mqtt_helper: in MQTT helper init
[00:00:00.289,489] <inf> mqtt_helper: MQTT helper initialized

Is there anything I can do to check that the certificates worked and there is access to AWS? Im am somewhat confident this may be an issue with how i've gone about setting up authentication. 

Any step to try debug and get the AWS sample working would be super helpful. 

[Hakon]

Hello,

have you tried to debug the code to check where it gets stuck?

[HoovinSchoovin]

Hello, 

Thank you for replying. 

Yes, it seems to be getting to mqtt_helper_init but then failing to initiate l4_event_handler with a  NET_EVENT_L4_CONNECTED event. I found this via putting break points through the code and gauging where it gets caught up. Is this what you mean or is there a more effective way to do it?

Trying the device on the nrf9161DK, i get this:

[00:00:00.568,939] <inf> mqtt_helper: MQTT helper initialized
+CGEV: EXCE STATUS 0
+CEREG: 2,"8910","07A91978",7
+CSCON: 1
+CGEV: ME PDN ACT 0,0
+CNEC_ESM: 50,0
+CEREG: 5,"8910","07A91978",7,,,"11100000","11100000"

<<Then the device connects to AWS>>

Which suggests its not an application or library issue. To rule out internet issues, i checked that the custom boards AT commands and they can connect to internet and the GNSS can successfully download SUPL data. Therefore, there is something specific to the AWS sample on the custom board that is not allowing it to connect to the internet. 

Could this be due to a wrong configuration in the board file? To me its the only differentiator between the DK and the custom board. Yet, because the internet can be connected on the custom board i thought that would be ruled out. 

Any ideas of where to try and debug? 

Thanks!

[Hakon]

HoovinSchoovin said:

Yes, it seems to be getting to mqtt_helper_init but then failing to initiate l4_event_handler with a  NET_EVENT_L4_CONNECTED event.

I believe this is caused by AWS rejecting the connection somehow, L4 should be TLS related issue.

Is it possible for you to capture a modem trace?

HoovinSchoovin said:

To rule out internet issues, i checked that the custom boards AT commands and they can connect to internet and the GNSS can successfully download SUPL data. Therefore, there is something specific to the AWS sample on the custom board that is not allowing it to connect to the internet. 

Yes, the issue is most definitely related to the AWS connection setup, maybe something goes wrong in the TLS handshake or something. Should have modem trace to debug this further.

[HoovinSchoovin]

Your right, the issue is on the AWS end. Its an mqtt -111 issue, suggesting certificates. 

The issue is with the certificates. I made an experiment where on the DK i uploaded certificates via the certificate manager and then via the AT%CMNG=0,201,0/1/2 commands. The AWS connection works when uploading via certificate manager but not the manual approach. 

The issue is with the custom board, the connection manager freezes when attempting to add certificates. 

To check: 
1) Are there normally issues when adding certs through the certificate manager via a UART connection rather than through a direct connection to the board?
2) Is there a certain format required for the certificates? The certificate manager outputs certificates in a different format (with a lot more white space) to simply copying in the certificates into the following:

AT%CMNG=0,201,0,”-----BEGIN CERTIFICATE-----
.....

-----END CERTIFICATE-----”

Ideally, I would like to work out why the custom AT command approach is not working then i can easily add the certificates on the custom board and bypass the issues with the certificate manager. 

Thanks, 
Sam 

[Hakon]

HoovinSchoovin said:

The issue is with the certificates. I made an experiment where on the DK i uploaded certificates via the certificate manager and then via the AT%CMNG=0,201,0/1/2 commands. The AWS connection works when uploading via certificate manager but not the manual approach. 

One thing that can be easy to forget when writing the certificates with AT%CMNG or directly in code is to include newline termination characters. IIRC every line should be ended with '\n' character. Certificate manager automatically includes these, so this is not a problem in that case.

[HoovinSchoovin]

Thanks Hakon ! This is highly likely the issue. I even got to the point of copy and pasting the AT command generated by the certificate manager and it was not working (now i see its not bringing over the /n's)

Ive tried adding /n but cannot replicate the certificate manager (ive tried all the methods i could think of). Is there any cause for cell manager not work through UART? It freezes at at the "Updating CA certificate..." point. It works on the DK but not the custom board (that i interface AT_client with through UART). All other AT commands work. 

Thanks!

[Hakon]

Can you try listing the certs with AT%CMNG=1? Check it's full.

[Hakon]

Also, what modem firmware are you using?