DNS Issue with nrf9160 and 1NCE Network and AWS IoT

Hi,

we are using the NRF9160 with AWS IoT MQTT. SDK is 1.8.0 and Modem is 1.3.1

Our devices stopped working one day to another (first realized 28.04.2022). The device connects to 1nce network but can not resolve to the aws cloud via mqtt.

Some other strange behaviour came up: The same SIM Card works in another device with the same firmware. The not working device also works with a different sim card.

We checked SIM Lock and did a sim reset at 1nce. Non of them worked. After more investigation I could reproduce the non working devices responde to an IP but not to a URL (AWS IoT is URL). We used to check: 

err = getaddrinfo("google.de", NULL, &hints, &res);
err = getaddrinfo("8.8.8.8", NULL, &hints, &res);

In the non working device-sim combination only second works. In a working device-sim combination both work.

1nce can see we connect but they see 0Bytes of data traffic for non working devices. 
I attached a log and a trace (trace v1) from a non connecting device


Thanks for your help here. We need to find a solution here, because this kind of scenario would be a big issue in production. In a conclusion our devices stopped working without our implact. This should not happen again.

I am also in contact with 1nce with the same case. I think aws can be excluded as the source of the issue

PS: I also added some information in another thread (https://devzone.nordicsemi.com/f/nordic-q-a/87467/nrf9160-dk-aws-iot-cloud-connection-issue) but I think our case is a different issue thats why I opened new here

best regards

Daniel

00> *** Booting Zephyr OS build v2.7.0-ncs1  ***
00> <inf> <mflt>: Reset Reason, RESETREAS=0x0
00> <inf> <mflt>: Reset Causes: 
00> <inf> <mflt>:  Power on Reset
00> <inf> <mflt>: GNU Build ID: 8de9aee36569f6939ca966c34abdf52b064fd7fe
00> I: [WD] No Dump
00> I: [WD] Watchdog Init to 1000 ms timeout
00> W: [WD] NOT Ready - Keep Active
00> I: 3 Sectors of 4096 bytes
00> I: alloc wra: 0, f78
00> I: data wra: 0, 38
00> E: [NVS] Settings read from MEM failed SYS:0 MEM:0 CLOUD:0
00> I: [MAIN] MEMO Init V: vX.X.X
00> I: [MAIN] Software is in Test
00> I: [MAIN] Reset Reason 0, Button:0 Software:0 WD:0
00> I: [MAIN] Reboot Counter: 1
00> W: [INIT] UART0 is Disabled
00> I: [MODEM] Modem Band Lock Set 
00> I: [MODEM] Version: 1.3.1
00> I: [CHARGER] Bat Power OK: 1
00> I: [CHARGER] Battery is Connected
00> I: [MAIN] Start First Init
00> I: [BUTTON] Silent Mode: 1
00> I: [PROXY] Load Cal from Mem
00> I: [INIT] Set Tray to Unknown
00> I: [INIT] Hardware OK
00> W: [MAIN] Startup Check - Bat: 3993
00> I: [USB] DISCONNECTED
00> E: [AWS] Send Message: AWS is not Init
00> I: [MODEM] sim_id: 901405102318251
00> I: [MODEM] client_id: nrf-352656106701W: [AWS] iot_reconnect - wait 30 s connect try: 2/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -11
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> I: [MODEM] Network registration status: Connected - roaming
00> I: [MODEM] PSM parameter update: TAU: 1116000, Active time: -1
00> I: [AWS] iot_reconnect - Restart Once after network back
00> W: [AWS] iot_reconnect - wait 30 s connect try: 3/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -2
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> W: [AWS] iot_reconnect - wait 30 s connect try: 4/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -2
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> I: [AWS] iot_reconnect Restart Modem
00> I: [AWS] CFUN4: 
00> I: [AWS] CFUN1: 
00> W: [AWS] iot_reconnect - wait 30 s connect try: 5/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -11
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> I: [AWS] iot_reconnect - Restart Once after network back
00> W: [AWS] iot_reconnect - wait 30 s connect try: 6/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -11
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> I: [AWS] iot_reconnect - Restart Once after network back
00> W: [AWS] iot_reconnect - wait 30 s connect try: 7/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -11
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
00> I: [MODEM] Network registration status: Connected - roaming
00> W: [AWS] iot_reconnect - wait 30 s connect try: 8/30
00> I: [AWS] IOT_EVT_CONNECTING
00> E: getaddrinfo, error -2
00> E: client_broker_init, error: -10
00> I: [AWS] IOT_EVT_CONNECTING
(Connection lost)

5852.trace-2022-05-02T14-14-04.176Z.bin

Parents
  • Here are the latest response from 1NCE to the DNS issue:

    Dear Mr. Daniel Böber,

    thank you for your feedback regarding your DNS resolution.

    As already suspected by you, the results of the performed tests confirm a possible challenge with DNS resolution by your devices.

    Please note that the DNS settings are not specified or stored by the SIM card itself. Instead, the DNS server settings are the responsibility of your device.

    Please let us know how your Nordic devices retrieve and/or process DNS server information (e.g. manual DNS configuration, ePCO or PCO) and, if necessary, find out by contacting Nordic.

    We have checked our network and can confirm that the DNS settings are transmitted correctly and as intended to your device as soon as the data connection is accepted. You can see this in the attached trace file for e.g. your SIM card with the IMSI 901405102318267 under the PCO values.

    However, we cannot verify whether these values are actually still contained in the Deutsche Telekom signaling at the location of your device. Therefore, we ask you to connect the device to the mobile network of Vodafone on a test basis, to test the DNS resolution for a connection to your server afterwards and to inform us about the result of this test.

    If this test is successful, our technical specialists will need the following information for an internal analysis with Deutsche Telekom:

    How your Nordic device retrieves the DNS server dataLog files of your device showing the connection setup including the exact times of the operations.
    We thank you in advance for your cooperation

    imsi_901405102318267_2022-05-03T09-44Z_2022-05-03T09-47Z.pcapng

Reply
  • Here are the latest response from 1NCE to the DNS issue:

    Dear Mr. Daniel Böber,

    thank you for your feedback regarding your DNS resolution.

    As already suspected by you, the results of the performed tests confirm a possible challenge with DNS resolution by your devices.

    Please note that the DNS settings are not specified or stored by the SIM card itself. Instead, the DNS server settings are the responsibility of your device.

    Please let us know how your Nordic devices retrieve and/or process DNS server information (e.g. manual DNS configuration, ePCO or PCO) and, if necessary, find out by contacting Nordic.

    We have checked our network and can confirm that the DNS settings are transmitted correctly and as intended to your device as soon as the data connection is accepted. You can see this in the attached trace file for e.g. your SIM card with the IMSI 901405102318267 under the PCO values.

    However, we cannot verify whether these values are actually still contained in the Deutsche Telekom signaling at the location of your device. Therefore, we ask you to connect the device to the mobile network of Vodafone on a test basis, to test the DNS resolution for a connection to your server afterwards and to inform us about the result of this test.

    If this test is successful, our technical specialists will need the following information for an internal analysis with Deutsche Telekom:

    How your Nordic device retrieves the DNS server dataLog files of your device showing the connection setup including the exact times of the operations.
    We thank you in advance for your cooperation

    imsi_901405102318267_2022-05-03T09-44Z_2022-05-03T09-47Z.pcapng

Children
No Data
Related