mqtt_client_tls_write hangs on writing to TLS socket send(client->transport.tls.sock....)

[Johan]

Hi,

We are using the modem firmware V1.2 and source code ncs v1.2.0.

Problem: we are using the TLS connected socket to amazon cloud and after about 12 minutes of doing:

```

waiting 2 min
connecting to MQTT
sending a few packets
disconnecting from MQTT

```

we get a hang on a specific call out of my reach:
File: mqtt_transport_socket_tls.c

Function: int mqtt_client_tls_write(struct mqtt_client *client, const u8_t *data, u32_t datalen)

Inside the function is a loop and a call here - which the source code isn't revealed for.
ret = send(client->transport.tls.sock, data + offset, datalen - offset, 0);

The last argument - flags - is set to 0 by Nordic, so if there is any problem sending, this call will hang forever.
Linux supports having MSG_DONTWAIT and returncodes to handle situations if they occur but now it's just not working.

We need a fix for this bug.
If the transmit fails, there needs to be a timeout so we can recover and not get hung, which kills the application calling.
If I can produce a trace or whatever needed for this let me know. We are running our custom code so it only works on our custom hw.

/Johan

[Heidi]

Hi!

It looks like bsdlib 0.7.0 added support for send timeout on TCP, including secure sockets.  So upgrading to NCS tag 1.3.0, which uses bsdlib version 0.7.3 might solve this issue for you. 

Best regards,

Heidi

[Johan]

All right, I will try that approach.

But If I use timeout, what is your recommended solution handling when the send function fails?

Do you have a sample for correctly handling errors like this? I have no idea how to error handle this case your code produces if Im using a timeout as a "break" operation when it otherwise would hang.

[Heidi]

Hi, a lot of people are finally back from vacation this week so we can really start looking into this.

It seems like this is an issue we've seen previously so I've escalated it to the developers. I'll let you know once I hear back. 

Best regards,

Heidi

[Johan]

Great. Im back on the 10th myself

Johan

[Johan]

Heidi
Any update from your team on this issue since 7 days back?

I'm back in my office again!

/Johan

[Heidi]

Hi, the case has been assigned to a developer but no updates yet. Could you take a modem trace, maybe that will help debug the issue? 

[Johan]

I'm not sure how to record a modem trace with custom hardware.
Can you guide how that is done?

I know you want to have some serial output from the modem over some serial channel, becoming a usb-serial port into the PC - so the software can pick it up.

In our design, i'm not sure which serial port that needs to be pinned out from the modem - to supply such data outward to the pc going through an usb-converter - also named so your software can recognize it as the modem trace serial channel.

See the problem with custom hardware vs your sdk kit hw?

[Johan]

Ping

[EvenFL]

Hi,

Have you tried MSG_DONTWAIT in the flag section when calling send()? It seems like we have an issue with timeout on send() if the modem is out of memory for some reason. MSG_DONTWAIT should be supported by NCS 1.3.0.

[Johan]

Thanks, I'll try soon Heidi! 

To help you reproduce:
When we were sending intensely before, this error came quite fast like 12 minutes or a bit more. I was sending alot of maximized (in size) MQTT-packets. We were close to the limit, to maximize the 2KB restriction in your packets over the radio every time. And we got a MQTT_PUBACK response before sending the next, so we were waiting properly, and got it confirmed transmission was ok by it as well.

[Heidi]

Hi Johan! How is it going with this issue? Can I close this case? 

[Johan]

Hi Heidi,

It seems we don't see the problem with ncs 1.3.0 så du kan stänga caset! :-)