nRF9160, SLM Firmware behaves incorrectly after activating Native TLS (critical problems)

Hi, 
We managed to build and flash SLM (SDK 2.1.2) with Native TLS.
To be able to download about 5Kb of data at one time (from AWS broker).


But not everything works fine, we have several critical problems with this build:
1) First, We can't check certificates, the same we did it before.

Previously, I used commands like this (321, is our internal secure tag):
- AT%CMNG=1,321,0 (For check exists the CA certificate);
- AT%CMNG=1,321,1 (For check exists the Client certificate);
- AT%CMNG=1,321,2  (For check exists the Private Key);

But after moving to Native TLC, and generating the new certificates (via commands like the AT#XCMNG=0,321,0...), We see other certificates by AT%CMNG=1 command:
%CMNG: 3210,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
%CMNG: 3211,0,"EA89F71E2A945D88A7A71C3C35D2F3BA5466B0ACEC750270134F511FB0CB4143"
%CMNG: 3212,0,"EB569D288E10B1D9D5B1BD45642AD4B8D063E0C48584A6C3A7A700987D671B68"

Can I use them to verify the CA, Client and private key?
- AT%CMNG=1,3210,0 (For CA);
- AT%CMNG=1,3211,0 (For Client);
- AT%CMNG=1,3212,0 (For check exist the Private Key).

2) Second: We lost access to nRF Cloud.

All commands for working with CMNG had to be changed (was AT%CMNG became AT#XCMNG), and it helped for access to AWS broker, but not to nRF Cloud.
Our previous certificates don't work more (with Native TLS).
I tried to reinstall them using the #XCMNG AT command, but that didn't work.
AT commands and answers look fine.
Maybe this is due to the generation and installation of a private key, we use:
AT%KEYGEN=16842753,2,0\r\n


But after using the
AT%KEYGEN=16842753,2,0\r\n, followed by saving the CA/Client certificates, we have the next results (every install certificate command returned OK):
[7/12 17:55:04:025] %CMNG: 16842753,2,"7494C435C484599577AB9B04E85DD594C5B5C9676750CA6DA17284E99A93B9C7"
[7/12 17:55:04:025] %CMNG: 168427530,0,"AD6FB002E6B34C0559FA8F93A3794FF12C4E3F119BD77290C52525123FB9EA74"
[7/12 17:55:04:025] %CMNG: 168427531,0,"D12494EF17B635B10A7511A513CB37B3101D58008ADE0D4BAA381D85A63DA675"

But if we tried to connect to nRF Cloud nothing happened.
We didn't have some ERROR or Event about the nRF Cloud connection.
We received the OK after about 900ms and CSCON: 0 after 6 seconds.
What are we doing wrong with nRF Cloud access now? Is it still because of the  AT%KEYGEN=16842753,2,0  AT command?

3) Third: Something happening with MQTT Disconnect:
After AT#XMQTTCON=0 we received a weird answer (\r\n) and nRF Reset.


AT#XMQTTCON=0
 Ready



Parents
  • Hello, 

    Sorry for the delayed answer here. I am looking into the issue on my side. Will get back to you within tomorrow. 

    Kind regards,
    Øyvind

  • Hello, I've had some issues with my application not allowing to connect to AWS. Trying to find the solution to this first. 

    1) First, We can't check certificates, the same we did it before.

    No, the XCMNG command does not support the list or read functionality. 

    2) Second: We lost access to nRF Cloud.

    What do you get when trying to connect to nRF Cloud? Make sure to disable CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID in prj.conf, i.e. setting 

    CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID=n. 
    3) Third: Something happening with MQTT Disconnect:
    You did not receive #XMQTTEVT: 1,0 ?
    Kind regards,
    Øyvind
  • Hi ,
    We commented out the slm_tls_unloadcrdl the inside mqtt_thread_fn() function and recheck in with 9160DK board.
    And for nrf9160DK now connection/disconnection through a broker looks correct. nRF no longer reboots, and AT commands continue to work after being  disconnected from the broker.

    What side effects will we get if we include these changes in our firmware?

    Regarding:

    Can you please disable modem trace so that UART1 could be used by TF-M?

    Is it still required?

     

  • ,

    I still can't understand how I'm supposed to work with UART1 (what PINs should I use TX/RX and where ?) I defined the definitions you mentioned:

    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    but nothing happens.
    What exactly do we need to do to work with UART1?

    The uart1 is disable in project into the nrf9160dk_nrf9160_ns.dts file:

    /* Disable UART1, because it is used by default in TF-M */
    &uart1 {
        status = "disabled";
    };
    After trying to activate uart1, the project crashes.




  • Stas, my apologies I was out of office due to illness in my family. 

    I will reach out to our developers and get an answer in regards to UART1 and a work-around.

    As the thread is getting quite long, can you please explain how your log output is setup today? What UART are you using in your project, and what is it used for? Where do you get normal application log? 

  • Hi ,

    I was out of office due to illness in my family. 

    I hope you and your family are doing well.

    Regarding:

    What UART are you using in your project, and what is it used for? Where do you get normal application log? 

    Into the "nrf9160dk_nrf9160_ns.conf" file we manage UART_0 and UART_2 for work with internal either external MCU.

    &uart0 {
        status = "disabled";
    };
    
    &uart2 {
        compatible = "nordic,nrf-uarte";
        current-speed = <115200>;
        status = "okay";
        hw-flow-control;
    
        pinctrl-0 = <&uart2_default_alt>;
        pinctrl-1 = <&uart2_sleep_alt>;
        pinctrl-names = "default", "sleep";
    };


    And, into the "nrf9160dk_nrf9160_ns.dts" exist configuration for uart1, but we never use it before.

    /* Disable UART1, because it is used by default in TF-M */
    &uart1 {
        status = "disabled";
    };


    And for any logs (if we talk about of work with internal MCU) types we always using LTE Link Monitor or Trace Collector V2.
    That is All. Maybe we don't know about some additional way to get the logs?

Reply
  • Hi ,

    I was out of office due to illness in my family. 

    I hope you and your family are doing well.

    Regarding:

    What UART are you using in your project, and what is it used for? Where do you get normal application log? 

    Into the "nrf9160dk_nrf9160_ns.conf" file we manage UART_0 and UART_2 for work with internal either external MCU.

    &uart0 {
        status = "disabled";
    };
    
    &uart2 {
        compatible = "nordic,nrf-uarte";
        current-speed = <115200>;
        status = "okay";
        hw-flow-control;
    
        pinctrl-0 = <&uart2_default_alt>;
        pinctrl-1 = <&uart2_sleep_alt>;
        pinctrl-names = "default", "sleep";
    };


    And, into the "nrf9160dk_nrf9160_ns.dts" exist configuration for uart1, but we never use it before.

    /* Disable UART1, because it is used by default in TF-M */
    &uart1 {
        status = "disabled";
    };


    And for any logs (if we talk about of work with internal MCU) types we always using LTE Link Monitor or Trace Collector V2.
    That is All. Maybe we don't know about some additional way to get the logs?

Children
  • Stas Jis said:
    I hope you and your family are doing well.

    Thank you!

    Stas Jis said:
    Into the "nrf9160dk_nrf9160_ns.conf" file we manage UART_0 and UART_2 for work with internal either external MCU.

    Does this apply to you custom HW as well? 


    Stas Jis said:
    how I'm supposed to work with UART1

    UART1 is shared by modem trace and TF-M logs. In order to receive TF-M logs you must disable the modem trace again.

    In prj.conf you can ensure that you have the following settings:

    CONFIG_NRF_MODEM_LIB_TRACE=n
    CONFIG_TFM_SECURE_UART1=y
    CONFIG_TFM_LOG_LEVEL_SILENCE=n
    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    Are you using the nRF9160DK during your tests? If so, you can use any terminal to open VCOM2 of DK board with baudrate 115200. 
    From the nRF9160 DK Virtual COM port documentation:

    The virtual COM ports are the following:

    • VCOM0 – Connected to nRF9160 (default)
    • VCOM1 – Connected to nRF52840 (nonconfigurable)
    • VCOM2 – Connected to nRF9160 (default)

    In the VS Code extension your board should be visible and you should be able to connect to it.


    In regards to :

    Stas Jis said:

    We commented out the slm_tls_unloadcrdl the inside mqtt_thread_fn() function and recheck in with 9160DK board.
    And for nrf9160DK now connection/disconnection through a broker looks correct. nRF no longer reboots, and AT commands continue to work after being  disconnected from the broker.

    What side effects will we get if we include these changes in our firmware?

    Our developers point out that the credentials seem to be unloaded twice. In the following .diff file they have added "graceful unloading of credentials" (missing from our first .diff file).

    This means e.g. 

    #if defined(CONFIG_SLM_NATIVE_TLS)
    		(void)slm_tls_unloadcrdl(ctx.sec_tag);
    #endif
    
    

    should be 

    #if defined(CONFIG_SLM_NATIVE_TLS)
    	if (ctx.sec_tag != INVALID_SEC_TAG) {
    		(void)slm_tls_unloadcrdl(ctx.sec_tag);
    		ctx.sec_tag = INVALID_SEC_TAG;
    	}
    #endif

    diff --git a/applications/serial_lte_modem/overlay-native_tls.conf b/applications/serial_lte_modem/overlay-native_tls.conf
    index c312b9bb2..4004d8007 100644
    --- a/applications/serial_lte_modem/overlay-native_tls.conf
    +++ b/applications/serial_lte_modem/overlay-native_tls.conf
    @@ -24,7 +24,7 @@ CONFIG_MBEDTLS_ENABLE_HEAP=y
     # and CONFIG_MBEDTLS_HEAP_SIZE to 36864
     CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=5120
     CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096
    -CONFIG_MBEDTLS_HEAP_SIZE=36864
    +CONFIG_MBEDTLS_HEAP_SIZE=38656
     CONFIG_MBEDTLS_SSL_SERVER_NAME_INDICATION=y
     #Cipher Suite disable
     CONFIG_MBEDTLS_CTR_DRBG_C=n
    diff --git a/applications/serial_lte_modem/src/mqtt_c/slm_at_mqtt.c b/applications/serial_lte_modem/src/mqtt_c/slm_at_mqtt.c
    index aa7c91f3d..e83b9fbea 100644
    --- a/applications/serial_lte_modem/src/mqtt_c/slm_at_mqtt.c
    +++ b/applications/serial_lte_modem/src/mqtt_c/slm_at_mqtt.c
    @@ -261,6 +261,13 @@ static void mqtt_thread_fn(void *arg1, void *arg2, void *arg3)
     		}
     	}
     
    +#if defined(CONFIG_SLM_NATIVE_TLS)
    +	if (ctx.sec_tag != INVALID_SEC_TAG) {
    +		(void)slm_tls_unloadcrdl(ctx.sec_tag);
    +		ctx.sec_tag = INVALID_SEC_TAG;
    +	}
    +#endif
    +
     	LOG_INF("MQTT thread terminated");
     }
     
    @@ -328,6 +335,9 @@ static void client_init(void)
     		struct mqtt_sec_config *tls_config;
     
     		tls_config = &(client.transport).tls.config;
    +#if defined(CONFIG_SLM_NATIVE_TLS)
    +		tls_config->set_native_tls = true;
    +#endif
     		tls_config->peer_verify   = TLS_PEER_VERIFY_REQUIRED;
     		tls_config->cipher_list   = NULL;
     		tls_config->cipher_count  = 0;
    @@ -359,9 +369,25 @@ static int do_mqtt_connect(void)
     
     	/* Connect to MQTT broker */
     	client_init();
    +
    +#if defined(CONFIG_SLM_NATIVE_TLS)
    +	err = slm_tls_loadcrdl(ctx.sec_tag);
    +	if (err < 0) {
    +		LOG_ERR("Fail to load credential: %d", err);
    +		return -EAGAIN;
    +	}
    +#endif
    +
     	err = mqtt_connect(&client);
     	if (err != 0) {
     		LOG_ERR("ERROR: mqtt_connect %d", err);
    +#if defined(CONFIG_SLM_NATIVE_TLS)
    +		if (ctx.sec_tag != INVALID_SEC_TAG) {
    +			(void)slm_tls_unloadcrdl(ctx.sec_tag);
    +			ctx.sec_tag = INVALID_SEC_TAG;
    +		}
    +#endif
    +
     		return err;
     	}
     
    @@ -388,6 +414,13 @@ static int do_mqtt_disconnect(void)
     		return err;
     	}
     
    +#if defined(CONFIG_SLM_NATIVE_TLS)
    +	if (ctx.sec_tag != INVALID_SEC_TAG) {
    +		(void)slm_tls_unloadcrdl(ctx.sec_tag);
    +		ctx.sec_tag = INVALID_SEC_TAG;
    +	}
    +#endif
    +
     	if (k_thread_join(&mqtt_thread, K_SECONDS(CONFIG_MQTT_KEEPALIVE)) != 0) {
     		LOG_WRN("Wait for thread terminate failed");
     	}
    diff --git a/applications/serial_lte_modem/src/slm_native_tls.c b/applications/serial_lte_modem/src/slm_native_tls.c
    index b14a68d2f..3dc7a5573 100644
    --- a/applications/serial_lte_modem/src/slm_native_tls.c
    +++ b/applications/serial_lte_modem/src/slm_native_tls.c
    @@ -10,7 +10,7 @@
     LOG_MODULE_REGISTER(slm_tls, CONFIG_SLM_LOG_LEVEL);
     
     /* max buffer length to load credential */
    -#define MAX_CRDL_LEN		4096
    +#define MAX_CRDL_LEN		4352
     
     /* pointer to credential buffer */
     static uint8_t *crdl;
    

  • Hi ,
    I apologize for wait. We have tested your solution.
    We modified everything as specified in the file native_tls_mqtt_aws_unload.diff.
    And we see that the connection/disconnection process with the broker works correctly, without any problems.

    But we still cannot get the TF-M logs (from UART1) when we using nRF9160DK.

    We included all definitions in prj.conf, except for one (CONFIG_TFM_LOG_LEVEL_SILENCE=n) the project did not build with it (we work with SDK 2.1.2):

    CONFIG_NRF_MODEM_LIB_TRACE=n
    CONFIG_TFM_SECURE_UART1=y
    #CONFIG_TFM_LOG_LEVEL_SILENCE=n
    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y

    And if we test the connection / disconnection process with a broker via LTE Link Monitor (COM31).
    And in parallel polling another terminal (COM27), we didn't see anything.
    Not a bit of logs.

    I have attached our prj.conf, maybe something is wrong?

    CONFIG_LOG=y
    CONFIG_LOG_DEFAULT_LEVEL=3
    CONFIG_STACK_SENTINEL=y
    CONFIG_NEWLIB_LIBC=y
    CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
    CONFIG_RING_BUFFER=y
    
    # Segger RTT
    CONFIG_USE_SEGGER_RTT=y
    CONFIG_RTT_CONSOLE=y
    CONFIG_UART_CONSOLE=n
    CONFIG_LOG_BACKEND_RTT=y
    CONFIG_LOG_BACKEND_UART=n
    
    # Network
    CONFIG_NETWORKING=y
    CONFIG_NET_SOCKETS=y
    CONFIG_NET_NATIVE=n
    
    # Modem library
    CONFIG_NRF_MODEM_LIB=y
    # Align the max FD entry to NRF_MODEM_MAX_SOCKET_COUNT(8)
    CONFIG_POSIX_MAX_FDS=8
    # Enable below for modem trace
    CONFIG_NRF_MODEM_LIB_TRACE=n
    CONFIG_TFM_SECURE_UART1=y
    #CONFIG_TFM_LOG_LEVEL_SILENCE=n
    # Handle modem fault
    CONFIG_NRF_MODEM_LIB_ON_FAULT_APPLICATION_SPECIFIC=y
    
    # Use GPIO
    CONFIG_GPIO=y
    CONFIG_GPIO_NRFX=y
    
    # UART interface
    CONFIG_SERIAL=y
    CONFIG_UART_ASYNC_API=y
    CONFIG_NRFX_TIMER2=y
    
    # Stacks and heaps
    CONFIG_MAIN_STACK_SIZE=4096
    CONFIG_HEAP_MEM_POOL_SIZE=16384
    CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
    
    # Device power management
    CONFIG_PM_DEVICE=y
    
    # FOTA
    CONFIG_HTTP_PARSER_URL=y
    CONFIG_FOTA_DOWNLOAD=y
    CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
    CONFIG_DFU_TARGET=y
    CONFIG_DOWNLOAD_CLIENT=y
    CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
    CONFIG_BOOTLOADER_MCUBOOT=y
    CONFIG_IMG_MANAGER=y
    
    # FLASH
    CONFIG_FLASH=y
    CONFIG_FLASH_PAGE_LAYOUT=y
    CONFIG_FLASH_MAP=y
    CONFIG_MPU_ALLOW_FLASH_WRITE=y
    CONFIG_IMG_ERASE_PROGRESSIVELY=y
    
    # Settings
    CONFIG_SETTINGS=y
    CONFIG_SETTINGS_FCB=y
    CONFIG_FCB=y
    
    # nRF Cloud
    #CONFIG_NRF_CLOUD_IPV6=y
    CONFIG_NRF_CLOUD_MQTT=y
    CONFIG_NRF_CLOUD_FOTA=n
    CONFIG_NRF_CLOUD_AGPS=y
    CONFIG_NRF_CLOUD_AGPS_FILTERED=n
    CONFIG_NRF_CLOUD_PGPS=y
    CONFIG_NRF_CLOUD_CELL_POS=y
    CONFIG_NRF_CLOUD_LOG_LEVEL_INF=y
    CONFIG_NRF_CLOUD_GPS_LOG_LEVEL_INF=y
    CONFIG_DATE_TIME=y
    CONFIG_MODEM_INFO=y
    CONFIG_MODEM_INFO_ADD_DATE_TIME=n
    CONFIG_AT_MONITOR=y
    CONFIG_MODEM_JWT=y
    CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID=y
    
    #
    # SLM-specific configurations
    #
    CONFIG_SLM_CUSTOMIZED=n
    CONFIG_SLM_LOG_LEVEL_INF=y
    CONFIG_SLM_EXTERNAL_XTAL=n
    CONFIG_SLM_START_SLEEP=n
    #CONFIG_SLM_DATAMODE_URC=y
    
    # nRF Cloud based location services
    CONFIG_SLM_AGPS=y
    CONFIG_SLM_PGPS=y
    CONFIG_SLM_CELL_POS=y
    CONFIG_SLM_LOG_LEVEL_DBG=y
    CONFIG_NRF_CLOUD_LOG_LEVEL_DBG=y
    CONFIG_MQTT_CLEAN_SESSION=y
    
    CONFIG_TFM_SPM_LOG_LEVEL_DEBUG=y
    CONFIG_TFM_EXCEPTION_INFO_DUMP=y
    CONFIG_THREAD_NAME=y
    CONFIG_THREAD_ANALYZER=y
    CONFIG_TFM_HALT_ON_CORE_PANIC=y
    
    CONFIG_TFM_CRYPTO_PARTITION_STACK_SIZE=0x2500


  • Hi Stas, 

    Just to confirm after our meeting yesterday. The work-around works for you, and the reset issue we saw earlier is gone. This means that there is no use for the TF-M logs.

    As mentioned, please test the solution thoroughly and let us know if there are any further issues with the native TLS solution.

    Kind regards,
    Øyvind

Related