No connection to nRF cloud (Attempts to use AGPS)

[Stas Jis]

We are using nRF9160 (Development Kit), SDK 1.9.1 board with "serial lte modem" firmware.
We are trying to get AGPS coordinates (using an external antenna), but for some reason we stopped receiving confirmations about the connection to the nRF cloud.
In response to the command "AT#XNRFCLOUD=1" we always get only "OK" without (#XNRFCLOUD: 1,0).
Checking the connection to the cloud shows that there is no connection:

AT#XNRFCLOUD?

#XNRFCLOUD: 0,0,16842753,"nrf-352656108600480"

The last time during development we checked the command sequence for getting coordinates using AGPS about a month ago and everything worked fine (was tested on SDK 1.8). But now even on SDK 1.8 we cannot get coordinates using AGPS, only regular GPS works.
AT commands: 

2022-03-31T12:15:16.966Z DEBUG modem << Ready
2022-03-31T12:15:18.043Z DEBUG modem << [00:00:00.467,926] [0m<inf> slm: Serial LTE Modem[0m
2022-03-31T12:15:18.050Z DEBUG modem << [00:00:00.500,030] [0m<inf> slm_at_host: at_host init done[0m
2022-03-31T12:15:25.049Z DEBUG modem >> AT
2022-03-31T12:15:25.059Z DEBUG modem << OK
2022-03-31T12:15:25.873Z DEBUG modem >> AT+CFUN?
2022-03-31T12:15:25.886Z DEBUG modem << +CFUN: 0
2022-03-31T12:15:25.894Z DEBUG modem << OK
2022-03-31T12:15:45.149Z DEBUG modem << *** Booting Zephyr OS build v2.7.99-ncs1-1  ***
2022-03-31T12:15:46.145Z DEBUG modem << Ready
2022-03-31T12:15:47.126Z DEBUG modem << [00:00:01.176,116] [0m<inf> slm: Serial LTE Modem[0m
2022-03-31T12:15:47.135Z DEBUG modem << [00:00:01.208,160] [0m<inf> slm_at_host: at_host init done[0m
2022-03-31T12:16:27.274Z DEBUG modem >> AT
2022-03-31T12:16:27.285Z DEBUG modem << OK
2022-03-31T12:16:28.249Z DEBUG modem >> AT+CFUN?
2022-03-31T12:16:28.265Z DEBUG modem << +CFUN: 0
2022-03-31T12:16:28.272Z DEBUG modem << OK
2022-03-31T12:16:34.122Z DEBUG modem << *** Booting Zephyr OS build v2.7.99-ncs1-1  ***
2022-03-31T12:16:34.439Z DEBUG modem << Ready
2022-03-31T12:16:35.420Z DEBUG modem << [00:00:00.498,748] [0m<inf> slm: Serial LTE Modem[0m
2022-03-31T12:16:35.429Z DEBUG modem << [00:00:00.530,822] [0m<inf> slm_at_host: at_host init done[0m
2022-03-31T12:16:41.915Z DEBUG modem << *** Booting Zephyr OS build v2.7.99-ncs1-1  ***
2022-03-31T12:16:42.651Z DEBUG modem << Ready
2022-03-31T12:16:43.632Z DEBUG modem << [00:00:00.915,985] [0m<inf> slm: Serial LTE Modem[0m
2022-03-31T12:16:43.639Z DEBUG modem << [00:00:00.948,059] [0m<inf> slm_at_host: at_host init done[0m
2022-03-31T12:16:56.482Z DEBUG modem >> AT%XSYSTEMMODE=1,0,1,0
2022-03-31T12:16:56.498Z DEBUG modem << OK
2022-03-31T12:17:01.946Z DEBUG modem >> AT%XMAGPIO=1,0,0,1,1,1574,1577
2022-03-31T12:17:01.960Z DEBUG modem << OK
2022-03-31T12:17:06.921Z DEBUG modem >> AT%XCOEX0
2022-03-31T12:17:06.932Z DEBUG modem << OK
2022-03-31T12:17:11.401Z DEBUG modem >> AT+CEREG=5
2022-03-31T12:17:11.414Z DEBUG modem << OK
2022-03-31T12:17:16.289Z DEBUG modem >> AT+CPSMS=1
2022-03-31T12:17:16.302Z DEBUG modem << OK
2022-03-31T12:17:21.488Z DEBUG modem >> AT+CFUN=1
2022-03-31T12:17:21.534Z DEBUG modem << OK
2022-03-31T12:17:23.561Z DEBUG modem << +CEREG: 2,"026F","00332520",7
2022-03-31T12:17:25.178Z DEBUG modem << +CEREG: 5,"026F","00332520",7,,,"00011110","11100000"
2022-03-31T12:17:28.768Z DEBUG modem >> AT#XNRFCLOUD=1
2022-03-31T12:17:28.776Z DEBUG modem << OK
2022-03-31T12:17:36.432Z DEBUG modem >> AT#XAGPS=1,1
2022-03-31T12:17:36.439Z DEBUG modem << ERROR
2022-03-31T12:17:36.440Z ERROR Error: AT#XAGPS=1,1
 failed
2022-03-31T12:17:42.194Z DEBUG modem >> AT#XNRFCLOUD?
2022-03-31T12:17:42.206Z DEBUG modem << #XNRFCLOUD: 0,0,16842753,"nrf-352656108600480"
2022-03-31T12:17:42.208Z DEBUG modem << OK

prj.conf:

#
# Copyright (c) 2020 Nordic Semiconductor ASA
#
# SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
#
# General config
CONFIG_LOG=y
CONFIG_LOG_DEFAULT_LEVEL=3
CONFIG_STACK_SENTINEL=y
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
CONFIG_RING_BUFFER=y

# Segger RTT
CONFIG_USE_SEGGER_RTT=y
CONFIG_RTT_CONSOLE=y
CONFIG_UART_CONSOLE=y
CONFIG_LOG_BACKEND_RTT=n
CONFIG_LOG_BACKEND_UART=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_NATIVE=n

# Modem library
CONFIG_NRF_MODEM_LIB=y
# Align the max FD entry to NRF_MODEM_MAX_SOCKET_COUNT(8)
CONFIG_POSIX_MAX_FDS=8
# Enable below for modem trace
#CONFIG_NRF_MODEM_LIB_TRACE_ENABLED=y

# Use GPIO
CONFIG_GPIO=y
CONFIG_GPIO_NRFX=y

# UART interface
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y
CONFIG_NRFX_TIMER2=y

# Stacks and heaps
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_HEAP_MEM_POOL_SIZE=16384
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

# Device power management
CONFIG_PM_DEVICE=y

# FOTA
CONFIG_HTTP_PARSER_URL=y
CONFIG_FOTA_DOWNLOAD=y
CONFIG_FOTA_DOWNLOAD_PROGRESS_EVT=y
CONFIG_DFU_TARGET=y
CONFIG_DOWNLOAD_CLIENT=y
CONFIG_DOWNLOAD_CLIENT_STACK_SIZE=4096
CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_IMG_MANAGER=y

# FLASH
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

# Settings
CONFIG_SETTINGS=y
CONFIG_SETTINGS_FCB=y
CONFIG_FCB=y

# nRF Cloud
CONFIG_CLOUD_API=y
CONFIG_NRF_CLOUD_MQTT=y
CONFIG_NRF_CLOUD_FOTA=n
CONFIG_NRF_CLOUD_AGPS=y
CONFIG_NRF_CLOUD_PGPS=y
CONFIG_NRF_CLOUD_CELL_POS=y
CONFIG_NRF_CLOUD_LOG_LEVEL_INF=y
CONFIG_DATE_TIME=y
CONFIG_MODEM_INFO=y
CONFIG_MODEM_INFO_ADD_DATE_TIME=n
CONFIG_AT_MONITOR=y
CONFIG_MODEM_JWT=y
# Use "nrf-<IMEI>" as the default device_id format
# Use "<device_uuid>" as the device_id format by below
#CONFIG_NRF_CLOUD_CLIENT_ID_SRC_INTERNAL_UUID=y

#
# SLM-specific configurations
#
CONFIG_SLM_CUSTOMIZED=n
CONFIG_SLM_LOG_LEVEL_INF=y
CONFIG_SLM_EXTERNAL_XTAL=n
CONFIG_SLM_START_SLEEP=n
#\nUse\nUART_0\n(when\nworking\nwith\nPC\nterminal)
CONFIG_UART_0_NRF_HW_ASYNC_TIMER=2
#\nUse\nUART_2\n(when\nworking\nwith\nexternal\nMCU)
#CONFIG_SLM_CONNECT_UART_2=y
#CONFIG_UART_2_NRF_HW_ASYNC_TIMER=2
#\nnRF\nCloud\nbased\nlocation\nservices
CONFIG_SLM_AGPS=y
CONFIG_SLM_PGPS=y
CONFIG_SLM_CELL_POS=y
#CONFIG_SLM_LOG_LEVEL_DBG=y

[justin]

I see two CLOUD_EVT_CONNECTING events, but no CLOUD_EVT_CONNECTED event. 

In the file slm_at_gnss.c, could you add a value to the log message:

case CLOUD_EVT_CONNECTING:
		LOG_DBG("CLOUD_EVT_CONNECTING: status %d", evt->data.err);

The value in evt->data.err is of type enum cloud_connect_result. 

Setting CONFIG_NRF_CLOUD_LOG_LEVEL_DBG=y might be helpful too.

[Stas Jis]

justin,

Have you changed or deleted the credentials in the modem sec_tag 16842753?

The problem may be related to the fact that before that we installed our certificates to communicate with the Amazon MQTT Broker (in this evaluation board)?
Used commands:
AT%CMNG=0,321,0,-----BEGIN CERTIFICATE-----\n"....
AT%CMNG=0,321,1,-----BEGIN CERTIFICATE-----\n" ...
AT%CMNG=0,321,2,-----BEGIN EC PRIVATE KEY-----\n" ...

[justin]

the modem sec tags allow you to have credentials for multiple servers. 
so if you used 321 for AWS IoT and kept 16842753 for nrf cloud, there shouldn't be a problem. 

did you use sec tag 321 when you added your device to nrf cloud?  if so, rebuild your serial LTE modem application with:
CONFIG_NRF_CLOUD_SEC_TAG=321

If that isn't the solution, deleting the device from nrf cloud and re-provisioning with the correct credentials should work.

[Stas Jis]

justin,
did you use sec tag 321 when you added your device to nrf cloud?  if so, rebuild your serial LTE modem application with:

No, I'm having problems registering the device (gives error 40101).
This board was previously registered by our former employee, and we were unable to access his account. Can we somehow transfer the device to my profile?

[Stas Jis]

justin:

How can we return the default certificates for nRF Cloud with the tag (16842753 )?

[justin]

Ah, ok, that is expected.  A device can only belong to one account at a time.
Your device has been removed from the old account.
You should be able to add the device to a new account now.

If the device has JITP (the default) certs in 16842753, nRF Cloud (AWS) will disconnect the device on the first connection attempt so it can do provisioning.  The device will need to then reconnect and you can associate the device with your account.

You can also look into using the nRF Cloud "Teams" feature: https://docs.nrfcloud.com/Reference/Teams
It might be useful if you have multiple users.

[Stas Jis]

justin_nordic:
Your device has been removed from the old account.
I still can't add device to nRF cloud, error - Code 40412.

[justin]

i see you have another active ticket for this provisioning issue, so i won't continue that here.
but i will explain briefly how JITP works and why you are getting 40412.
here are the steps:

1. First, the device must be provisioned with AWS/nRF Cloud.  This means that there is a record of your device and its certificate on the cloud.  For a device with JITP credentials, the provisioning step happens automatically when the device first connects to the cloud.  The cloud will immediately disconnect the device while it performs provisioning.

2.  Next, the device must be associated with your account (AWS tenant).  This step cannot be done unless the device is provisioned with the cloud.  If you attempt to associate a device that is not provisioned, you will get error 40412.  Once associated, the device will successfully connect to the cloud.

note: the sec tag, credentials, and device id must be consistent throughout these steps.

[Stas Jis]

Dear justin_nordic,
You said:
Next, the device must be associated with your account (AWS tenant).  This step cannot be done unless the device is provisioned with the cloud.  If you attempt to associate a device that is not provisioned, you will get error 40412.  Once associated, the device will successfully connect to the cloud.

What exactly is required to prepare a device for adding to the nRF cloud?
I've been trying to get at least one of the two devices (nRF9160 DK, Custom Board) to no avail for two weeks now.

[justin]

Please review this guide:
https://docs.nrfcloud.com/Guides/GettingStarted/Devices

There are two type of provisioning: JITP and "pre-connect".  If you are downloading credentials from nrfcloud.com then you are doing JITP.  I have described JITP in my previous reply.  Pre-connect provisioning is what custom/manufactured devices would use. 

Your device must have credentials installed into a sec tag in the modem.  This includes: the amazon root CA, device certificate(s) and device private key.  Your FW application (CONFIG_NRF_CLOUD_SEC_TAG) must use the sec tag containing those credentials.  The sec tag is only for your device, the cloud does not know or care about the sec tag.

For JITP credentials, using LTE Link Monitor's Certificate Manager is the easiest way to do this.  It will parse the credentials from the downloaded JSON file and send them to the device in the correct format.  Before updating the credentials, you must send the AT command: AT+CFUN=4 to deactivate the modem.