When connecting to AWS IoT with low signal it gets stuck

We have devices that have low signal so they have to reconnect to AWS a lot. We have a watchdog timer with a 100s timeout that we feed every 10 seconds in the control loop.

Occasionally we see the device hit the 100s watchdog timer and reset when connecting to AWS.

Environment:

Only see with devices with low signal

nRF Connect SDK v3.5.99-ncs1-1

SDK and tool chain: v2.6.1

Tf-M v2.0.0

MODEM_INFO_FW_VERSION: mfw_nrf9160_1.3.6
LTE-M cellular connected in the U.S.
Configs:
CONFIG_AWS_IOT_CONNECT_TIMEOUT_SECONDS=30
Code:
	LOG_INF("Connecting to AWS IoT broker");
	const struct aws_iot_config config = {
			.client_id = client_id_buf,
			.host_name = CONFIG_AWS_IOT_BROKER_HOST_NAME,
	};

	int start_time = k_uptime_get();
	// This is synchronous and cant take a while (30+ seconds)
	err = aws_iot_connect(&config);
Logs:
[00:06:04.668,487] <inf> cloud: Connecting to AWS IoT broker
[00:06:04.674,621] <dbg> mqtt_helper: broker_init: Resolving IP address for [deleted]
[00:06:06.815,155] <dbg> mqtt_helper: broker_init: IPv4 Address found [deleted] (AF_INET)
[00:06:06.823,944] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING
// WDT hit and reconnects
�[Sec Thread] Secure image initializing!
Booting TF-M v2.0.0
*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
[00:00:00.258,026] <inf> main: main started
[00:00:00.262,512] <wrn> main: reset cause 0x10


Another example:
[00:24:59.793,365] <inf> cloud: Connecting to AWS IoT broker
[00:24:59.799,530] <dbg> mqtt_helper: broker_init: Resolving IP address for 
[00:25:01.423,126] <dbg> mqtt_helper: broker_init: IPv4 Address found(AF_INET)
[00:25:01.431,762] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING
[00:26:03.737,792] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_TRANSPORT_CONNECTED
[00:26:03.750,396] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTED --> MQTT_STATE_CONNECTING
[00:26:03.762,176] <dbg> mqtt_helper: client_connect: Using send socket timeout of 60 seconds
[00:26:03.770,935] <dbg> mqtt_helper: mqtt_helper_connect: MQTT connection request sent
[00:26:03.779,235] <dbg> mqtt_helper: mqtt_helper_poll_loop: Took connection_poll_sem
[00:26:03.787,322] <dbg> mqtt_helper: mqtt_helper_poll_loop: Starting to poll on socket, fd: 0
[00:26:03.796,173] <dbg> mqtt_helper: mqtt_helper_poll_loop: Polling on socket fd: 0
�[Sec Thread] Secure image initializing!
Booting TF-M v2.0.0
*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
[00:00:00.258,087] <inf> main: main started
[00:00:00.262,573] <wrn> main: reset cause 0x10


// Another Example
[00:16:36.075,561] <inf> cloud: Connecting to AWS IoT broker
[00:16:36.081,726] <dbg> mqtt_helper: broker_init: Resolving IP address for [0m
[00:16:37.178,619] <dbg> mqtt_helper: broker_init: IPv4 Address found(AF_INET)
[00:16:37.187,408] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING
[00:17:50.167,114] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_TRANSPORT_CONNECTED
[00:17:50.179,687] <dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTED --> MQTT_STATE_CONNECTING
[00:17:50.191,497] <dbg> mqtt_helper: client_connect: Using send socket timeout of 60 seconds
[00:17:50.200,225] <dbg> mqtt_helper: mqtt_helper_connect: MQTT connection request sent
[00:17:50.208,557] <dbg> mqtt_helper: mqtt_helper_poll_loop: Took connection_poll_sem
[00:17:50.216,613] <dbg> mqtt_helper: mqtt_helper_poll_loop: Starting to poll on socket, fd: 0
[00:17:50.225,494] <dbg> mqtt_helper: mqtt_helper_poll_loop: Polling on socket fd: 0
�[Sec Thread] Secure image initializing!
Booting TF-M v2.0.0
*** Booting nRF Connect SDK v3.5.99-ncs1-1 ***
[00:00:00.258,117] <inf> main: main started
[00:00:00.262,603] <wrn> main: reset cause 0x10
Sometimes I also see that connecting took longer than the 30 second config we have set.
[00:09:30.122,406] [0m<inf> cloud: aws_iot_connect() took 52963 ms
Here's where it took so long:
[00:08:38.476,684] [0m<dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_DISCONNECTED --> MQTT_STATE_TRANSPORT_CONNECTING[0m
[00:09:27.934,753] [0m<dbg> mqtt_helper: mqtt_state_set: State transition: MQTT_STATE_TRANSPORT_CONNECTING --> MQTT_STATE_TRANSPORT_CONNECTED[0m
Related