nRF9160, SLM Firmware behaves incorrectly after activating Native TLS (critical problems)

Hi, 
We managed to build and flash SLM (SDK 2.1.2) with Native TLS.
To be able to download about 5Kb of data at one time (from AWS broker).


But not everything works fine, we have several critical problems with this build:
1) First, We can't check certificates, the same we did it before.

Previously, I used commands like this (321, is our internal secure tag):
- AT%CMNG=1,321,0 (For check exists the CA certificate);
- AT%CMNG=1,321,1 (For check exists the Client certificate);
- AT%CMNG=1,321,2  (For check exists the Private Key);

But after moving to Native TLC, and generating the new certificates (via commands like the AT#XCMNG=0,321,0...), We see other certificates by AT%CMNG=1 command:
%CMNG: 3210,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
%CMNG: 3211,0,"EA89F71E2A945D88A7A71C3C35D2F3BA5466B0ACEC750270134F511FB0CB4143"
%CMNG: 3212,0,"EB569D288E10B1D9D5B1BD45642AD4B8D063E0C48584A6C3A7A700987D671B68"

Can I use them to verify the CA, Client and private key?
- AT%CMNG=1,3210,0 (For CA);
- AT%CMNG=1,3211,0 (For Client);
- AT%CMNG=1,3212,0 (For check exist the Private Key).

2) Second: We lost access to nRF Cloud.

All commands for working with CMNG had to be changed (was AT%CMNG became AT#XCMNG), and it helped for access to AWS broker, but not to nRF Cloud.
Our previous certificates don't work more (with Native TLS).
I tried to reinstall them using the #XCMNG AT command, but that didn't work.
AT commands and answers look fine.
Maybe this is due to the generation and installation of a private key, we use:
AT%KEYGEN=16842753,2,0\r\n


But after using the
AT%KEYGEN=16842753,2,0\r\n, followed by saving the CA/Client certificates, we have the next results (every install certificate command returned OK):
[7/12 17:55:04:025] %CMNG: 16842753,2,"7494C435C484599577AB9B04E85DD594C5B5C9676750CA6DA17284E99A93B9C7"
[7/12 17:55:04:025] %CMNG: 168427530,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
[7/12 17:55:04:025] %CMNG: 168427531,0,"D12494EF17B635B10A7511A513CB37B3101D58008ADE0D4BAA381D85A63DA675"

But if we tried to connect to nRF Cloud nothing happened.
We didn't have some ERROR or Event about the nRF Cloud connection.
We received the OK after about 900ms and CSCON: 0 after 6 seconds.
What are we doing wrong with nRF Cloud access now? Is it still because of the  AT%KEYGEN=16842753,2,0  AT command?

3) Third: Something happening with MQTT Disconnect:
After AT#XMQTTCON=0 we received a weird answer (\r\n) and nRF Reset.


AT#XMQTTCON=0
 Ready



Parents Reply
  • I'm currently looking into this and trying to find correct people to help me with your issue. In order to do so, I need to verify your issues.

    1) First, We can't check certificates, the same we did it before.

    Have you been able to solve this? To write certificates to modem when using native TLS you will need to use AT#XCMNG, afterwards they can be read using AT%CMNG but in your case it will be from sec_tag 3210, 3211 and 3212 due to how certificates are written with AT#XCMNG.

    2) Second: We lost access to nRF Cloud.

    Were you able to fix this issue?

    3) Third: Something happening with MQTT Disconnect:

    This seems to be reset of some sort based on the description. This is not visible in the logs you have provided, other than "Ready" is returned after MQTT disconnect. 



    Also, from your email:


    check if it is possible to use Native TLS without disconnecting to the AWS broker when we use only go to idle mode (xsleep=2)?

    Will look into this as well.

Children
  • Hi ,
    For the first two points, you have already given the answers earlier:
    1)

    Stas Jis said:
    But then why after generating new certificates (via commands like the AT#XCMNG=0,321,0...) I see new certificate entries with our security tag ?

    Sorry my answer was unclear. To clarify, you can use %CMNG to read 3210, 3211 and 3212

    2) 

    Thanks for providing these logs. Unfortunately, they do no provide much information. Can you please provide a modem trace? This should indicate why the device is resetting. 

    Stas Jis said:
    And now I can connect to nRF Cloud without some problems.
    But, obviously, we have different logic for connecting to AWS Broker (with Native TLS logic) and nRF Cloud.

    Happy to hear you were able to find a work around for connecting to nRF Cloud. Yes, it seems that for AWS you will need to use AT#XCMNG and for nRF Cloud you must use AT%CMNG. This is supported by Native TLS.

    Now is actually only the third point interest us where we have a problem with disconnecting to the AWS broker.

  • Stas, I've forwarded this case to our Serial LTE Modem developers to get more information. 

    Their initial feedback is: 

    XSLEEP=2 only power off UART, all LTE and IP connections not affected

    Will push for more answers.

  • Hi ,

    This seems to be reset of some sort based on the description. This is not visible in the logs you have provided, other than "Ready" is returned after MQTT disconnect. 

    If we use the logic analyzer to view TX/RX from UART, sometimes we saw different results after executing the broker disconnect command.
    Sometimes it's just line breaks (\r\n), and sometimes it's an illogical set of characters before getting "Ready".



  • Hello,

    i will forward your findings regarding UART response.  

    Our R&D developers working with the SLM application have looked at this issue. Following are steps they would like from you.


    1. In order to get more information, they are asking if you can provide more information from exception dump. This can be enabled by adding:

    CONFIG_TFM_LOG_LEVEL_SILENCE=n
    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    Observe log from UART1 when crash happen.
    They should see something like this when SLM crashed.

    [Sec Thread] Secure image initializing!
    TF-M isolation level is:  0x00000001
    TF-M FP mode: Software
    Booting TF-M v1.6.0+8cffe127
    FATAL ERROR:  SecureFault
    Here is some context for the exception:
         EXC_RETURN (LR):  0xFFFFFFB5
        Exception came from  non-secure FW in  handler mode.
         xPSR:     0x60000007
        MSP:      0x200007F8
        PSP:      0x20006E18
        MSP_NS:   0x20027CE0
        PSP_NS:   0x20027508
        Exception frame at:  0x20027CE0
            R0:    0x20019428
            R1:    0x00000000
            R2:    0x20027D38
            R3:    0x00000000
            R12:   0x00000000
            LR:    0x0004A7FB
            PC:    0x0004A7EA
            xPSR:  0x2100002C
        CFSR:   0x00000000
        BFSR:   0x00000000
        BFAR:  Not Valid
         MMFSR:  0x00000000
        MMFAR: Not Valid
         UFSR:   0x00000000
        HFSR:   0x00000000
        SFSR:   0x00000048
        SFAR:  0x00000004

    Customer can learn where fault come from. Secure FW or non-secure FW.
    They can also learn from PC and LR register where program crash by looking up the address inside zephyr.lst 


    2. Increase TF-M crpyto stack size.

    Increase TFM_CRYPTO_PARTITION_STACK_SIZE and TFM_CRYPTO_IOVEC_BUFFER_SIZE in nrf\modules\trusted-firmware-m\Kconfig

     


    3. The SLM use native TLS for the Secure socket/TLS Proxy server/HTTPS client. MQTT is not supported. Customer might make some modification themselves.  We will need to look into the settings of the project.

  • Hi ,
    Regarding:
    1)

    In order to get more information, they are asking if you can provide more information from exception dump. This can be enabled by adding:

    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    As far as I concerned you wanna see the log from nrf9160 DK (with these configs)?

    2) 

    Increase TF-M crpyto stack size.

    Increase TFM_CRYPTO_PARTITION_STACK_SIZE and TFM_CRYPTO_IOVEC_BUFFER_SIZE in nrf\modules\trusted-firmware-m\Kconfig

    The Kconfig file from SDK\v2.1.2\nrf\modules\trusted-firmware-m not include TFM_CRYPTO_PARTITION_STACK_SIZE and the TFM_CRYPTO_IOVEC_BUFFER_SIZE config look like:

    config TFM_CRYPTO_IOVEC_BUFFER_SIZE
    int
    prompt "TF-M Crypto - PSA FF IO vector buffer size" if !TFM_PROFILE_TYPE_MINIMAL
    depends on TFM_IPC
    default 1024 if TFM_PROFILE_TYPE_MINIMAL
    default 16384 if TFM_REGRESSION_S || TFM_REGRESSION_NS
    default 5120
    help
    This parameter applies only to IPC model builds. In IPC model,
    during a Service call, input and outputs are allocated
    temporarily in an internal scratch buffer whose size is
    determined by this parameter. To be configured based on the
    desired use case and application requirements.

    What exactly do we need to change here?

    3) 

    The SLM use native TLS for the Secure socket/TLS Proxy server/HTTPS client. MQTT is not supported. Customer might make some modification themselves.  We will need to look into the settings of the project.

    What namely (and where) you wanna see?

Related