Sending A large binary payload via MQTT

Hi, I'm using the serial_lte_modem sample application and I've modified the AT#XMQTTPUB command so that I can specify a slot number from which the device can grab a bank of binary data and send it up to an aws server.  For testing, I just have a predefined bank of binary data and I use the slot number to determine how much of the buffer is sent;

AT#XMQTTPUB=<topic>,<msg>[,<qos>[,<retain>]].  So in my implementation, msg is required, it can be a string, then that message is published, it can be "" or 0, in which case the unit goes into datamode, or it can be a number, which refers to a slot.  In the future, I'm going to be adding AT# functions to populate these slots with binary data, but for now, I have a fixed set of binary data and I use the slot number to indicate how many bytes of that data I will send (just for testing purposes);

if (strlen(pub_msg) == 0)
{
if ( binSlot == 0 )
{
/* Publish payload in data mode */
err = enter_datamode(mqtt_datamode_callback);
}
else
{
err = do_mqtt_publish(pub_bin, binSlot );
}
} else {
err = do_mqtt_publish(pub_msg, msg_sz);
}

OK, I've enabled CONFIG_NRF91_SOCKET_SEND_SPLIT_LARGE_BLOCKS so that I get around the 2k byte limit of TLS.  but I'm still having problems going over 6047 bytes.  Up to that point, the data goes over fine.  Once I got 6048 and beyond I get;

AT#XMQTTPUB="<topic obscured>",6048,0

#XMQTTEVT: 1,-95

out the debug port i get; [00:00:48.454,010] <err> slm_mqtt: POLLNVAL, which I assume because it's still trying to do transmissions on a closed socket.

To facilitate the large transmission size I've added 

#define MQTT_TX_BUFFER_LEN (64*1024)
#define MQTT_MESSAGE_BUFFER_LEN NET_IPV4_MTU

to create a special buffer for the MQTT Transmission;

static uint8_t tx_buffer[MQTT_TX_BUFFER_LEN];

I feel like there are other buffers I need to increase to avoid the EOPNOTSUPP, but I can't seem to find it.

I'm using SDK 1.8.0

Parents
  • Hello again Randall,

    I have gotten some feedback from our developers as well:

    Please be aware that the modem IP stack has a buffer limit of 4248 byte (MSS 708, max 6 packets). To publish large payload, MQTT Publish should have QoS 1 at least so as to get PUBACK, i.e. some basic uplink flow control. Otherwise there is possibility to overflow the TCP transmission buffer on modem side. My recommendation is to add QoS 1 in MQTT Pub request to monitor the traffic status first. TCP connection could vary for each uplink packet based on network condition.”

    I hope this will help you.

    Regards,

    //Markus

Reply
  • Hello again Randall,

    I have gotten some feedback from our developers as well:

    Please be aware that the modem IP stack has a buffer limit of 4248 byte (MSS 708, max 6 packets). To publish large payload, MQTT Publish should have QoS 1 at least so as to get PUBACK, i.e. some basic uplink flow control. Otherwise there is possibility to overflow the TCP transmission buffer on modem side. My recommendation is to add QoS 1 in MQTT Pub request to monitor the traffic status first. TCP connection could vary for each uplink packet based on network condition.”

    I hope this will help you.

    Regards,

    //Markus

Children
No Data
Related