mqtt_simple sample gets error -2 in getaddrinfo

[tomtomtom2401]

Hello,

We have developed our own module based on the nrf9160 DK and want to use it to send data to a MQTT broker. A few days ago on the dev-kit everything worked fine, now the new PCBs have arrived and we get this error in the mqtt_simple sample (as well as in our own code, but in a little different format):

 *** Booting Zephyr OS build v3.2.99-ncs1 ***                                      
[00:00:00.504,638] <inf> mqtt_simple: The MQTT simple sample started               
[00:00:00.504,699] <inf> mqtt_simple: Disabling PSM and eDRX                              
[00:00:00.505,401] <inf> mqtt_simple: LTE Link Connecting...                              
[00:00:09.277,618] <inf> mqtt_simple: LTE Link Connected!                                 
[00:00:09.277,862] <err> mqtt_simple: getaddrinfo failed: -2                              
[00:00:09.277,893] <err> mqtt_simple: Failed to initialize broker connection              
[00:00:09.277,893] <err> mqtt_simple: client_init: -10   

We just added the following config lines to use NB-IoT in the sample:

CONFIG_LTE_NETWORK_MODE_NBIOT=y
CONFIG_LTE_MODE_PREFERENCE_NBIOT=y

When using the IP address of the moquitto test server, it works perfectly fine.

*** Booting Zephyr OS build v3.2.99-ncs1 ***                                      
[00:00:00.568,603] <inf> mqtt_simple: The MQTT simple sample started                      
[00:00:00.568,664] <inf> mqtt_simple: Disabling PSM and eDRX                              
[00:00:00.569,366] <inf> mqtt_simple: LTE Link Connecting...                              
[00:00:11.186,523] <inf> mqtt_simple: LTE Link Connected!                                 
[00:00:11.186,950] <inf> mqtt_simple: IPv4 Address found 91.121.93.94                     
[00:00:11.187,438] <dbg> mqtt_simple: client_id_get: client_id = nrf-2EB11F7FD373E99C     
[00:00:11.598,114] <inf> mqtt_simple: MQTT client connected                               
[00:00:11.598,144] <inf> mqtt_simple: Subscribing to: my/subscribe/topic len 22       
[00:00:11.746,185] <inf> mqtt_simple: SUBACK packet id: 1234                              
[00:06:48.615,112] <inf> mqtt_simple: MQTT PUBLISH result=0 len=15                        
[00:06:48.615,203] <inf> mqtt_simple: Received: test with ip address                           
[00:06:48.615,783] <inf> mqtt_simple: Publishing: test with ip address                         
[00:06:48.615,814] <inf> mqtt_simple: to topic: my/publish/topic len: 20              
[00:06:48.761,352] <inf> mqtt_simple: PUBACK packet id: 39802 

Has anybody an idea? 

We are using the NCS v2.2.0 with the toolchain 2.2.0 (but tried the 2.1.0 as well, as we used with the dev-kit), modem firmware is 1.2.3

Edit: I forgot to add: when using the same code on the dev-kit, it works also fine with the IP Address and the URL. Modem firmware is here also 1.2.3 - thats the reason we choose 1.2.3 instead of 1.3.3. We also use the same SIM card to prevent any errors from the provider side (bad DNS, APN, whatever...)

Thank you!

BR Thomas

[Hakon]

Hello,

which SIM card are you using?

[tomtomtom2401]

Hello,

We are using a m2m SIM from company "drei" from Austria. I've used the very same SIM card in the dev kit and in our module to exclude any errors from this side.
In the meantime, I tried to set the DNS manually (like in thread), but it only works randomly and only once after flashing the modem firmware. Then it looks like its running into a timeout and a watchdog restarts the module.

BR Thomas

[Achim Kraus]

I don't think, this is related to the modem hardware revision, and the issue is even with the "current modem hw release".

> Our hardware is basically the devkit but with minor changes to the GPIOs and the debugger part has been removed - HF part, etc. is the same...

> (small ext. antenna, ~ -90dBm to ~ -100dBm).

FMPOV, the ext. antenna belongs to the "HF part" and is changed.

Some experts told me, that for the most antennas, you need a matching ground-plane. Not sure, if that applies for your antenna. Also not sure, what the "signal strength" value really refer to.

I'm still not clear, why the capture "trace_with_dns.bin" contains the changing ip-addresses, but your log trace_with_dns.bin the ip-address is stable.

Anyway, once you have some trace of your dev, without "nrf_setdnsaddr", it will be very interesting, what that shows.

[Achim Kraus]

And maybe you check, if the SIM card is connected proper, not that the connection to the SIM card is instable.

I would guess, that this is all in the part of the trace, which is not included in the wireshark capture. So that may require the analysis of Nordic modem's experts.

[tomtomtom2401]

> FMPOV, the ext. antenna belongs to the "HF part" and is changed.

> Some experts told me, that for the most antennas, you need a matching ground-plane. Not sure, if that applies for your antenna. Also not sure, what the "signal strength" value really refer to.

Thats right, but 50 Ohm matching, single ended, and so on, HF part is working in my opinion (tested fine with IP address of the broker and the CoAP server, signal levels, ...). What I wanted to say is, that there is no circuit with COEX pins controlling an external RF switch for antenna matching which will require some software modifications to the basic sample (like on the Thingy:91).

> I'm still not clear, why the capture "trace_with_dns.bin" contains the changing ip-addresses, but your log trace_with_dns.bin the ip-address is stable.

You linked the same two files ;)

For the new trace from the devkit, please see the mqtt_simple_dns_request_redo_devkit.pcapng (or the bin file) from two posts above.
The trace_with_dns.bin is with the nrf_setdnsaddr, on the "redo" traces it is without setting the secondary dns. I will try to set it manually to the DNS server from drei and do the retries.

> I would guess, that this is all in the part of the trace, which is not included in the wireshark capture. So that may require the analysis of Nordic modem's experts.

I hope someone can check it - but as we sadly see, there is not that much response from their side...

BR Thomas

[Achim Kraus]

> You linked the same two files ;)

Yes, the 2. one should be mqtt_simple_dns_request_redo.bin

> from their side...

Maybe because it's also still wired.

I would try to make captures from the real device (though the devkit doesn't show the issue), without calling nrf_setdnsaddr, but reset the config to factory, and let that run for a while.

[tomtomtom2401]

> I would try to make captures from the real device (though the devkit doesn't show the issue), without calling nrf_setdnsaddr, but reset the config to factory, and let that run for a while.

Those are the "mqtt_simple_dns_request_redo.bin" ;)
I've stopped testing with the nrf_setdnsaddr because the device is frozen after a minute or so and then rebooted.

For better clearance:
From devkit is: mqtt_simple_dns_request_redo_devkit
From our hardware: mqtt_simple_dns_request_redo

both without setting the dns, just with several retries with a delay of 10s and the additional AT commands mentioned earlier.

But... I got an update! (big smile on my face included):

I added the secondary dns via nrf_setdnsaddr - but not the drei dns server, the google dns 8.8.8.8 as tried earlier and it worked now. I have no idea why suddenly it works, or what has changed to earlier tries besides the AT commands mentioned (okay, maybe the modem needs a factory reset everytime? nah...)

Here you go with the traces:

mqtt_simple_dns_request_redo_setdnsaddr.bin

mqtt_simple_dns_request_redo_setdnsaddr.pcapng

[tomtomtom2401]

complete history for recreation:

1) disabled the factory reset command -> works fine

2) disabled all other AT commands -> works fine

3) flashed the "old" mqtt_simple sample with the nrf_setdnsaddr as used now -> doesn't work (log stops at LTE connecting / AT+CFUN=1, and device reboots after ~20s)

4) flashed version from (2) without AT commands -> doesn't work, similar behaviour like (3) but without rebooting, waited 3min+ but with no further response

5) enabled the both AT Commands %HWVERSION and %SHORTSWVER -> LTE connected after ~58s, getaddrinfo worked fine and mqtt connected

6) to verify, i flashed the old mqtt_simple sample from (3) and then the new without AT commands from (4), same behaviour, no reaction for like more than 3 minutes

7) flashed new mqtt_simple sample with the two AT cmds and then nrf_setdnsaddr -> no reaction for >3 min

8) enabled factory reset -> connection established after 20s, getaddrinfo worked and mqtt connected

9) disabled factory reset again -> workes, sometimes connection is established in under 10s, sometimes it needs up to a minute (cycles by pressing the reset button)

So in my opinion: the modem writes something to an internal NVM, this is cleared by a factory reset via the AT command %XFACTORYRESET=0

And to reproduce from a new mqtt_simple sample at our custom board with MFW 1.3.3:

1) created a new mqtt_simple sample

2) created the build configuration with enabled debug options

3) added in in prj.conf

#LTE link control
CONFIG_LTE_NETWORK_MODE_NBIOT=y
CONFIG_LTE_MODE_PREFERENCE_NBIOT=y

# modem trace debugging
CONFIG_NRF_MODEM_LIB_TRACE=y

### build and program: lte connects, getaddrinfo failes with -2

mqtt_simple_1.bin / mqtt_simple_1.pcapng

4) added to modem_configure() after disabling PSM and eDRX:

#include <nrf_socket.h>


int err;
char response[128];
static struct nrf_in_addr dns_google;
dns_google.s_addr = 134744072; // Google DNS, 8.8.8.8
err = nrf_setdnsaddr(1, &dns_google, sizeof(dns_google));
LOG_INF("DNS set result %d - %d", err, errno);

### build and program: lte connects, getaddrinfo works, mqtt connects

mqtt_simple_2_setdnsaddr.bin  / mqtt_simple_2_setdnsaddr.pcapng

conclusio: something must have written something to the modem's NVM to make this strange behaviour plus we dont get a primary DNS from our provider on our hardware (why? I dont know). but with a "clean" board after doing the factory reset AT command it works - even without doing the factory reset everytime. (I dont know what I added to the mqtt_simple sample before trying to add the DNS server via nrf_setdnsaddr at the first try)

Added this small snippet to our code and flashed it to a "clean" hardware - works fine - also with our sever.

Looking forward to hear something from a modem expert, we are trying to work with this workaround at the moment. If needed, i can upload our 3 mqtt_simple samples.

Thank you Achim for your big help!

BR Thomas

[Achim Kraus]

> %XFACTORYRESET=0

The AT commands are all documented, that contains also the info, if the config is stored in the NVM.

I ment to use that factory-reset only once, I do so, if I have the feeling, that a device doesn' behave proper. That mainly happens, if I use a device for a lot testing, changing the configuration again and again. The sometimes, a factory-reset helps.

From your latest capture (mqtt_simple_2_setdnsaddr.pcapng):

The first DNS request shows 10.160.202.185 as source-ip. That "timesout".

The second shows 10.84.42.235 and that succeeds. The device is then using that address for further requests.

I'm still worried about the address change. Lets look forward, what Nordic finds in that trace.