nRF9160, SLM Firmware behaves incorrectly after activating Native TLS (critical problems)

Hi, 
We managed to build and flash SLM (SDK 2.1.2) with Native TLS.
To be able to download about 5Kb of data at one time (from AWS broker).


But not everything works fine, we have several critical problems with this build:
1) First, We can't check certificates, the same we did it before.

Previously, I used commands like this (321, is our internal secure tag):
- AT%CMNG=1,321,0 (For check exists the CA certificate);
- AT%CMNG=1,321,1 (For check exists the Client certificate);
- AT%CMNG=1,321,2  (For check exists the Private Key);

But after moving to Native TLC, and generating the new certificates (via commands like the AT#XCMNG=0,321,0...), We see other certificates by AT%CMNG=1 command:
%CMNG: 3210,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
%CMNG: 3211,0,"EA89F71E2A945D88A7A71C3C35D2F3BA5466B0ACEC750270134F511FB0CB4143"
%CMNG: 3212,0,"EB569D288E10B1D9D5B1BD45642AD4B8D063E0C48584A6C3A7A700987D671B68"

Can I use them to verify the CA, Client and private key?
- AT%CMNG=1,3210,0 (For CA);
- AT%CMNG=1,3211,0 (For Client);
- AT%CMNG=1,3212,0 (For check exist the Private Key).

2) Second: We lost access to nRF Cloud.

All commands for working with CMNG had to be changed (was AT%CMNG became AT#XCMNG), and it helped for access to AWS broker, but not to nRF Cloud.
Our previous certificates don't work more (with Native TLS).
I tried to reinstall them using the #XCMNG AT command, but that didn't work.
AT commands and answers look fine.
Maybe this is due to the generation and installation of a private key, we use:
AT%KEYGEN=16842753,2,0\r\n


But after using the
AT%KEYGEN=16842753,2,0\r\n, followed by saving the CA/Client certificates, we have the next results (every install certificate command returned OK):
[7/12 17:55:04:025] %CMNG: 16842753,2,"7494C435C484599577AB9B04E85DD594C5B5C9676750CA6DA17284E99A93B9C7"
[7/12 17:55:04:025] %CMNG: 168427530,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
[7/12 17:55:04:025] %CMNG: 168427531,0,"D12494EF17B635B10A7511A513CB37B3101D58008ADE0D4BAA381D85A63DA675"

But if we tried to connect to nRF Cloud nothing happened.
We didn't have some ERROR or Event about the nRF Cloud connection.
We received the OK after about 900ms and CSCON: 0 after 6 seconds.
What are we doing wrong with nRF Cloud access now? Is it still because of the  AT%KEYGEN=16842753,2,0  AT command?

3) Third: Something happening with MQTT Disconnect:
After AT#XMQTTCON=0 we received a weird answer (\r\n) and nRF Reset.


AT#XMQTTCON=0
 Ready



Parents
  • Hello, 

    Sorry for the delayed answer here. I am looking into the issue on my side. Will get back to you within tomorrow. 

    Kind regards,
    Øyvind

  • Hello, I've had some issues with my application not allowing to connect to AWS. Trying to find the solution to this first. 

    1) First, We can't check certificates, the same we did it before.

    No, the XCMNG command does not support the list or read functionality. 

    2) Second: We lost access to nRF Cloud.

    What do you get when trying to connect to nRF Cloud? Make sure to disable CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID in prj.conf, i.e. setting 

    CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID=n. 
    3) Third: Something happening with MQTT Disconnect:
    You did not receive #XMQTTEVT: 1,0 ?
    Kind regards,
    Øyvind

  • ,

    Ah, that's somewhat confusing, can you try increasing with 512 from 5120 e.g.

    CONFIG_TFM_CRYPTO_IOVEC_BUFFER_SIZE= 5632

    I do not quite understand where exactly to define this value.
    Because earlier you wrote about:

    Increase TFM_CRYPTO_PARTITION_STACK_SIZE and TFM_CRYPTO_IOVEC_BUFFER_SIZE in nrf\modules\trusted-firmware-m\Kconfig

    But the Kconfig file itself does not define sizes for CONFIG_TFM_CRYPTO_IOVEC_BUFFER_SIZE, it's more like a description file.

    Searching into the project for this configuration did not return any results, can it just be added to prj.conf ? Or does it need to be done elsewhere?

  • Stas, you should be able to set the configuration in the prj.conf of your project (e.g. nrf/applications/serial_lte_modem/prj.conf)

    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    Our developer informs me that the TFM_CRYPTO_IOVEC_BUFFER_SIZE is not important at the moment, and that you should increase the TFM_CRYPTO_PARTITION_STACK_SIZE with

    CONFIG_TFM_CRYPTO_PARTITION_STACK_SIZE=0x2500

    Crash dump is generally suppressed by TF-M. By enabling the symbols above, we should be able to see the crash dump from when your device resets.

    Stas Jis said:
    But the Kconfig file itself does not define sizes for CONFIG_TFM_CRYPTO_IOVEC_BUFFER_SIZE, it's more like a description file.

    Yes, that is true. The developers define these KConfig symbols in the Kconfig file, and this will intiate the default state of this symbols. To reconfigure the value you can use prj.conf. 

    More information can be found in Setting Kconfig configuration values.

    Once you are able to get crash dump, can you also provide the file serial_lte_modem/src/mqtt_c/slm_at_mqtt.c ?

    Thanks. 


    Kind regards,
    Øyvind

  • Hi ,
    I included all configs that you asked.

    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    CONFIG_TFM_CRYPTO_PARTITION_STACK_SIZE=0x2500

    And I reproduced our problem with the DK board.
    So now I see that the AT command of disconnect with broker works, but after that any After AT#XMQTTCON=0 AT command doesn't work.
    nRF9160 DK It looks like something is holding back an internal process and the board is stuck.

    I attached the logs from Link Monitor and Trace Collector V2.

    trace-2023-01-22T10-12-47.087Z.bin

    2023-01-22T10:12:42.597Z DEBUG Application data folder: C:\Users\Stas_Zhiz\AppData\Roaming\nrfconnect\pc-nrfconnect-linkmonitor
    2023-01-22T10:12:42.670Z INFO Using nrf-device-lib-js version: 0.4.13
    2023-01-22T10:12:42.670Z INFO Using nrf-device-lib version: 0.12.8
    2023-01-22T10:12:42.670Z INFO Using nrfjprog DLL version: 10.16.0
    2023-01-22T10:12:42.670Z INFO Using JLink version: JLink_V7.66a
    2023-01-22T10:12:42.689Z DEBUG App pc-nrfconnect-linkmonitor v2.0.2 official
    2023-01-22T10:12:42.690Z DEBUG App path: C:\Users\Stas_Zhiz\.nrfconnect-apps\node_modules\pc-nrfconnect-linkmonitor
    2023-01-22T10:12:42.690Z DEBUG nRFConnect 3.12.0, required by the app is (^3.12.0)
    2023-01-22T10:12:42.690Z DEBUG nRFConnect path: C:\Users\Stas_Zhiz\AppData\Local\Programs\nrfconnect\resources\app.asar
    2023-01-22T10:12:42.690Z DEBUG HomeDir: C:\Users\Stas_Zhiz
    2023-01-22T10:12:42.690Z DEBUG TmpDir: C:\Users\STAS_Z~1\AppData\Local\Temp
    2023-01-22T10:12:45.752Z INFO Modem port is opened
    2023-01-22T10:12:45.760Z DEBUG modem >> AT+CFUN?
    2023-01-22T10:12:45.775Z DEBUG modem << +CFUN: 0
    2023-01-22T10:12:45.778Z DEBUG modem << OK
    2023-01-22T10:12:48.274Z DEBUG modem >> AT
    2023-01-22T10:12:48.287Z DEBUG modem << OK
    2023-01-22T10:12:56.594Z DEBUG modem >> AT%SHORTSWVER
    2023-01-22T10:12:56.611Z DEBUG modem << %SHORTSWVER: nrf9160_1.3.3
    2023-01-22T10:12:56.615Z DEBUG modem << OK
    2023-01-22T10:13:02.041Z DEBUG modem >> AT#XSLMVER
    2023-01-22T10:13:02.050Z DEBUG modem << #XSLMVER: "2.1.2"
    2023-01-22T10:13:02.053Z DEBUG modem << OK
    2023-01-22T10:13:13.266Z DEBUG modem >> AT%CMNG=1
    2023-01-22T10:13:13.380Z DEBUG modem << %CMNG: 0,0,"A86D8BF8D75130765A6453960E0E4E5158E0F3E43F86699EFE36190C463A25CF"
    2023-01-22T10:13:13.388Z DEBUG modem << %CMNG: 0,1,"DB1EC0EBC8F2A25A0982E604DCEEAB9B659AF28C188139AF891E556C4FE08FD3"
    2023-01-22T10:13:13.397Z DEBUG modem << %CMNG: 0,2,"9D8DBD2E213A4A70A99D0EDBD2E253BED39B017C57139D7DD17F933B9D265F2D"
    2023-01-22T10:13:13.404Z DEBUG modem << %CMNG: 0,6,"0606060606060606060606060606060606060606060606060606060606060606"
    2023-01-22T10:13:13.407Z DEBUG modem << %CMNG: 1000,0,"A86D8BF8D75130765A6453960E0E4E5158E0F3E43F86699EFE36190C463A25CF"
    2023-01-22T10:13:13.417Z DEBUG modem << %CMNG: 3210,0,"A86D8BF8D75130765A6453960E0E4E5158E0F3E43F86699EFE36190C463A25CF"
    2023-01-22T10:13:13.421Z DEBUG modem << %CMNG: 3211,0,"197F6580DA27A1E3F8A1B2ADA483031DE844A77C892E8CBF632844C3D68276A6"
    2023-01-22T10:13:13.430Z DEBUG modem << %CMNG: 3212,0,"55565BB27EEAD4957004CF3216E5EEA763C22AF0B230D8EB0968DA2BBEA1B573"
    2023-01-22T10:13:13.436Z DEBUG modem << %CMNG: 16842753,0,"A86D8BF8D75130765A6453960E0E4E5158E0F3E43F86699EFE36190C463A25CF"
    2023-01-22T10:13:13.443Z DEBUG modem << %CMNG: 16842753,1,"33C6012FE08C1DB8C58BCFCD676C88FC05F91D6A9C74E9E6BB568CF813D0FFC8"
    2023-01-22T10:13:13.451Z DEBUG modem << %CMNG: 16842753,2,"20AF836FC3348E3653C702162CB36349419AF3418DD78D992E5B42CBDC4FF715"
    2023-01-22T10:13:13.458Z DEBUG modem << %CMNG: 4294967293,10,"AEE95320E708D6D7F900870D5908C659243E5CF5253996F0E463D6F10FC680C0"
    2023-01-22T10:13:13.466Z DEBUG modem << %CMNG: 4294967294,6,"16050BCDF80936E564911665959605C982FB6B2097CE327FF52043DD3CE90459"
    2023-01-22T10:13:13.474Z DEBUG modem << %CMNG: 4294967292,11,"B2C46C2AE7C81943A8BD6DD4ED2A50B659A225A098A177BACB575459CD57CAEF"
    2023-01-22T10:13:13.477Z DEBUG modem << OK
    2023-01-22T10:13:22.378Z DEBUG modem >> AT+CFUN?
    2023-01-22T10:13:22.392Z DEBUG modem << +CFUN: 0
    2023-01-22T10:13:22.400Z DEBUG modem << OK
    2023-01-22T10:13:26.898Z DEBUG modem >> AT+CEREG=5
    2023-01-22T10:13:26.912Z DEBUG modem << OK
    2023-01-22T10:13:30.386Z DEBUG modem >> AT+CFUN=1
    2023-01-22T10:13:30.433Z DEBUG modem << OK
    2023-01-22T10:13:31.455Z DEBUG modem << +CEREG: 2,"1D6B","001B6A02",7
    2023-01-22T10:13:32.759Z DEBUG modem << +CEREG: 5,"1D6B","001B6A02",7,,,"00011110","11100000"
    2023-01-22T10:13:32.771Z DEBUG modem >> AT+COPS=3,2
    2023-01-22T10:13:32.781Z DEBUG modem << OK
    2023-01-22T10:13:32.784Z DEBUG modem >> AT+COPS?
    2023-01-22T10:13:32.797Z DEBUG modem << +COPS: 0,2,"42503",7
    2023-01-22T10:13:32.801Z DEBUG modem << OK
    2023-01-22T10:13:32.810Z DEBUG modem >> AT%XCBAND
    2023-01-22T10:13:32.819Z DEBUG modem << %XCBAND: 3
    2023-01-22T10:13:32.821Z DEBUG modem << 
    2023-01-22T10:13:32.823Z DEBUG modem << OK
    2023-01-22T10:13:32.832Z DEBUG modem >> AT+CGDCONT?
    2023-01-22T10:13:32.848Z DEBUG modem << +CGDCONT: 0,"IP","ibasis.iot","10.160.209.115",0,0
    2023-01-22T10:13:32.850Z DEBUG modem << OK
    2023-01-22T10:13:32.866Z DEBUG modem >> AT+CGACT?
    2023-01-22T10:13:32.874Z DEBUG modem << +CGACT: 0,1
    2023-01-22T10:13:32.876Z DEBUG modem << 
    2023-01-22T10:13:32.877Z DEBUG modem << OK
    2023-01-22T10:13:37.778Z DEBUG modem >> AT#XMQTTCON=1,"AR999000S33","","","a34k7wa09ujucc-ats.iot.us-east-1.amazonaws.com",8883,321
    2023-01-22T10:13:38.778Z ERROR Error: 'AT#XMQTTCON=1,"AR999000S33","","","a34k7wa09ujucc-ats.iot.us-east-1.amazonaws.com",8883,321
    ' timed out
    2023-01-22T10:13:40.293Z DEBUG modem << OK
    2023-01-22T10:13:40.600Z DEBUG modem << #XMQTTEVT: 0,0
    2023-01-22T10:13:44.801Z DEBUG modem >> AT#XMQTTCON?
    2023-01-22T10:13:44.816Z DEBUG modem << #XMQTTCON: 1,"AR999000S33","a34k7wa09ujucc-ats.iot.us-east-1.amazonaws.com",8883,321
    2023-01-22T10:13:44.818Z DEBUG modem << OK
    2023-01-22T10:13:50.706Z DEBUG modem >> AT#XMQTTCON=0
    2023-01-22T10:13:50.728Z DEBUG modem << #XMQTTEVT: 1,0
    
  • Hi  , 

    Our SLM team provided an answer

    Stas Jis said:
    So now I see that the AT command of disconnect with broker works, but after that any After AT#XMQTTCON=0 AT command doesn't work.
    nRF9160 DK It looks like something is holding back an internal process and the board is stuck.

    Message Ready when MQTT disconnected means that SLM crashed and rebooted.
    CONFIG_TFM_HALT_ON_CORE_PANIC makes application halt when SLM crashes.

    TF-M and lib modem trace use the same UART1 port. As the crash seems to be on the application side, thus we don't need modem trace. Can you please disable modem trace so that UART1 could be used by TF-M?

    If you successfully enable TF-M debug message you should see TF-M banner similar to this:

    [Sec Thread] Secure image initializing!
    TF-M isolation level is:  0x00000001
    TF-M FP mode: Software
    Booting TF-M v1.6.0+8cffe127

    The output of UART1 is what we need to diagnose when SLM crashes.

    Also, can you please try to remove slm_tls_unloadcrdl inside mqtt_thread_fn() to see if this mitigates the issue.

  • Hi ,

    Can you please disable modem trace so that UART1 could be used by TF-M?

    How exactly can we do this?

Reply Children
Related