Sending A large binary payload via MQTT

[Randall]

Hi, I'm using the serial_lte_modem sample application and I've modified the AT#XMQTTPUB command so that I can specify a slot number from which the device can grab a bank of binary data and send it up to an aws server.  For testing, I just have a predefined bank of binary data and I use the slot number to determine how much of the buffer is sent;

AT#XMQTTPUB=<topic>,<msg>[,<qos>[,<retain>]].  So in my implementation, msg is required, it can be a string, then that message is published, it can be "" or 0, in which case the unit goes into datamode, or it can be a number, which refers to a slot.  In the future, I'm going to be adding AT# functions to populate these slots with binary data, but for now, I have a fixed set of binary data and I use the slot number to indicate how many bytes of that data I will send (just for testing purposes);

if (strlen(pub_msg) == 0)
{
if ( binSlot == 0 )
{
/* Publish payload in data mode */
err = enter_datamode(mqtt_datamode_callback);
}
else
{
err = do_mqtt_publish(pub_bin, binSlot );
}
} else {
err = do_mqtt_publish(pub_msg, msg_sz);
}

OK, I've enabled CONFIG_NRF91_SOCKET_SEND_SPLIT_LARGE_BLOCKS so that I get around the 2k byte limit of TLS.  but I'm still having problems going over 6047 bytes.  Up to that point, the data goes over fine.  Once I got 6048 and beyond I get;

AT#XMQTTPUB="<topic obscured>",6048,0

#XMQTTEVT: 1,-95

out the debug port i get; [00:00:48.454,010] <err> slm_mqtt: POLLNVAL, which I assume because it's still trying to do transmissions on a closed socket.

To facilitate the large transmission size I've added 

#define MQTT_TX_BUFFER_LEN (64*1024)
#define MQTT_MESSAGE_BUFFER_LEN NET_IPV4_MTU

to create a special buffer for the MQTT Transmission;

static uint8_t tx_buffer[MQTT_TX_BUFFER_LEN];

I feel like there are other buffers I need to increase to avoid the EOPNOTSUPP, but I can't seem to find it.

I'm using SDK 1.8.0

[Randall]

Update:

I've discovered that if I add the commented out k_sleep shown below;

offset = 0;
flags |= NRF_MSG_WAITALL;
while (offset < msg->msg_iov[i].iov_len) {
for( retries = 0; retries < 5; retries++ )
{
ret = nrf91_socket_offload_sendto(obj,
(((uint8_t *) msg->msg_iov[i].iov_base) + offset),
(msg->msg_iov[i].iov_len - offset), flags,
msg->msg_name, msg->msg_namelen);
if ( ret >= 0 ) break;
k_sleep(K_MSEC( 500 + (retries * 500) ) );
}
if (ret < 0) {
printf( "Transmission Failed\n" );
return ret;
}
printf( "offset = %lu, len = %lu, ret = %lu\n", offset, len, ret );
offset += ret;
len += ret;
//if ( offset < msg->msg_iov[i].iov_len ) k_sleep(K_MSEC(500) );
}

it works.  But this seems rather inefficient.  So I added the flags |= NRF_MSG_WAITALL.  This is much more efficient but occasionally fails in 2 different ways.  1 returns failed with errno set to EOPNOTSUPP or it simply locks up and seems to never return (at least not during the time period I was willing to wait).  Now I could make this work with some extrra higher level error handling.   But I was hoping that there was a better way.

[Randall]

OK.  Made a small mistake.  It turns out that the flag change didn't help anything.  I forgot I still had the retries in there.  So it was actually them making it work, with the 2 failure conditions.

[Randall]

Well, I recompiled my serial_lte_modem project with CONFIG_NRF_MODEM_LIB_TRACE_ENABLED=y, then I used the AT%XMODEMTRACE=1,1 command to enable tracing on the application and AT+CFUN=0 which is my understanding that it should save it to NV RAM.  In the Trace Collector V2 app I chose the debug UART, but nothing seemed to come over.  All the .bin files generated have 0 bytes in them.  So I'm not sure what else I'm supposed to do to get a trace.