Sending A large binary payload via MQTT

RE: Sending A large binary payload via MQTT

Albrecht Markus Schellenberger — Fri, 11 Feb 2022 08:34:25 GMT

Hello Randall,

[quote user="Philrt"]So I'm not sure what else I'm supposed to do to get a trace.[/quote]

it should be enough to set:

CONFIG_NRF_MODEM_LIB_TRACE_ENABLED=y

I have tested this with the Serial LTE modem application on my nRF9160 DK an it works as expected. Which COM port did you select for recording?

Regards,

Markus

RE: Sending A large binary payload via MQTT

Randall — Thu, 10 Feb 2022 21:49:40 GMT

Well, I recompiled my serial_lte_modem project with CONFIG_NRF_MODEM_LIB_TRACE_ENABLED=y, then I used the AT%XMODEMTRACE=1,1 command to enable tracing on the application and AT+CFUN=0 which is my understanding that it should save it to NV RAM. In the Trace Collector V2 app I chose the debug UART, but nothing seemed to come over. All the .bin files generated have 0 bytes in them. So I'm not sure what else I'm supposed to do to get a trace.

RE: Sending A large binary payload via MQTT

Albrecht Markus Schellenberger — Tue, 08 Feb 2022 09:23:24 GMT

Hello again Randall,

I have gotten some feedback from our developers as well:

“Please be aware that the modem IP stack has a buffer limit of 4248 byte (MSS 708, max 6 packets). To publish large payload, MQTT Publish should have QoS 1 at least so as to get PUBACK, i.e. some basic uplink flow control. Otherwise there is possibility to overflow the TCP transmission buffer on modem side. My recommendation is to add QoS 1 in MQTT Pub request to monitor the traffic status first. TCP connection could vary for each uplink packet based on network condition.”

I hope this will help you.

Regards,

//Markus

RE: Sending A large binary payload via MQTT

Albrecht Markus Schellenberger — Tue, 08 Feb 2022 08:10:07 GMT

Hello Randall,

could you provide me with a modem trace + full application log from device boot, showing the described issue you are facing? You can preferably use the Trace Collector V2 preview for that.

Thanks & regards,

Markus

RE: Sending A large binary payload via MQTT

Albrecht Markus Schellenberger — Mon, 07 Feb 2022 16:49:17 GMT

Hello Randall,

[quote user="Philrt"]Is that possible? Or is there any way to retain the reliability but have it send faster?[/quote]

Thanks for the updates on your progress!

I’m not aware of any way to speed this up. Please note that TLS transmission is limited by hardware to 2kB, so you are already operating with a workaround.

But I’ll check with our developers if they have any input or comments to share.

Regards,

Markus

RE: Sending A large binary payload via MQTT

Randall — Mon, 07 Feb 2022 16:18:15 GMT

OK, I should probably make it clear what I'm asking for here. With the extra 500 MSec at the end of the loop, the system seems to work pretty reliably, but is slow (takes about 15 seconds up upload 64k). Without the 500 Msec but only the retires, it sometimes works faster but fails about 50% of the time, so that isn't acceptable. I was hoping there was some way to check the socket buffer, and wait until it's empty, then only to the SendTo when it's completely empty. Is that possible? Or is there any way to retain the reliability but have it send faster?

RE: Sending A large binary payload via MQTT

Randall — Mon, 07 Feb 2022 14:25:18 GMT

OK. Made a small mistake. It turns out that the flag change didn't help anything. I forgot I still had the retries in there. So it was actually them making it work, with the 2 failure conditions.

RE: Sending A large binary payload via MQTT

Randall — Mon, 07 Feb 2022 14:11:33 GMT

Update:

I've discovered that if I add the commented out k_sleep shown below;

offset = 0;
flags |= NRF_MSG_WAITALL;
while (offset < msg->msg_iov[i].iov_len) {
for( retries = 0; retries < 5; retries++ )
{
ret = nrf91_socket_offload_sendto(obj,
(((uint8_t *) msg->msg_iov[i].iov_base) + offset),
(msg->msg_iov[i].iov_len - offset), flags,
msg->msg_name, msg->msg_namelen);
if ( ret >= 0 ) break;
k_sleep(K_MSEC( 500 + (retries * 500) ) );
}
if (ret < 0) {
printf( "Transmission Failed\n" );
return ret;
}
printf( "offset = %lu, len = %lu, ret = %lu\n", offset, len, ret );
offset += ret;
len += ret;
//if ( offset < msg->msg_iov[i].iov_len ) k_sleep(K_MSEC(500) );
}

it works. But this seems rather inefficient. So I added the flags |= NRF_MSG_WAITALL. This is much more efficient but occasionally fails in 2 different ways. 1 returns failed with errno set to EOPNOTSUPP or it simply locks up and seems to never return (at least not during the time period I was willing to wait). Now I could make this work with some extrra higher level error handling. But I was hoping that there was a better way.