The setup is that there is a remote server that has a small TCP receive window (in this case 100 bytes) and a local nRF9160 device trying to send 4000 bytes to that server.
Code for nRF9160:
#include <zephyr/kernel.h>
#include <stdio.h>
#include <modem/lte_lc.h>
#include <zephyr/net/socket.h>
#include <fcntl.h>
void my_assert(bool b) {
if (!b) {
printk("failed %d\r\n", errno);
exit(1);
}
}
void main(void)
{
int err;
printk("Waiting for network..\n");
err = lte_lc_init_and_connect();
if (err) {
printk("Failed to connect to the LTE network, err %d\n", err);
return;
}
printk("Connected to network\n");
int sk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
my_assert(sk >= 0);
struct sockaddr_in sa;
sa.sin_family = AF_INET;
sa.sin_port = htons(9632);
sa.sin_addr.s_addr = ...; // INSERT IP ADDRESS
my_assert(connect(sk, (const struct sockaddr *)&sa, sizeof(sa)) == 0);
printk("Connected to server\n");
static const uint8_t buf[4000] = {0};
ssize_t nsent = send(sk, buf, sizeof(buf), 0);
printk("send returned %d\n", nsent);
if (nsent < 0) {
printk("errno: %d\n", errno);
}
close(sk);
}
Server code below. Works on a Windows computer, compile using gcc on MSYS2 or Cygwin (not mingw). Linux does not work since it cannot use such small TCP receive windows.
#include <assert.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <poll.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/ip.h>
int main(int argc, char *argv[]) {
struct sockaddr_in addr;
int res;
int sk;
int client_sk;
int pid;
sk = socket(AF_INET, SOCK_STREAM, 0);
assert(sk >= 0);
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(9632);
addr.sin_addr.s_addr = INADDR_ANY;
res = bind(sk, (struct sockaddr *)&addr, sizeof(addr));
assert(res == 0);
res = listen(sk, 5);
assert(res == 0);
int winsize = 100;
res = setsockopt(sk, SOL_SOCKET, SO_RCVBUF, &winsize, sizeof(int));
assert(res == 0);
for (;;) {
client_sk = accept(sk, NULL, NULL);
if (client_sk < 0) {
continue;
}
pid = fork();
assert(pid >= 0);
if (pid == 0) {
// client process
char buf[1000];
struct pollfd pfd = {client_sk, POLLIN, 0};
close(sk); // close server accept socket
res = poll(&pfd, 1, 30000);
assert(res >= 0);
if (res == 0) {
puts("Receive timeout");
} else {
puts("Client sent some data");
sleep(60);
}
close(client_sk);
return 0;
}
// server process
close(client_sk);
}
}
Output on nRF9160:
*** Booting Zephyr OS build v3.2.99-ncs2 *** Waiting for network.. Connected to network Connected to server
It permanently hangs in the send call. Wireshark on the server shows that only the TCP three-way handshake is performed and not a single byte of data is sent. The server then closes the connection (sends FIN) after a while and the nRF9160 modem successfully ACKs this packet. Finally, the server force-closes the connection using a RST packet. Not even at this point does the send call return. Please ignore the retransmission and dup ack in the image.
The expected outcome is that the send function should accept between 1 and 4000 bytes, put that amount of bytes in its send buffer and return the number of bytes processed, for example 100 since the receiver's receive window is 100 (alternatively, for a socket in blocking mode, it is ok to block and return first when all bytes have actually been sent). Also when the remote endpoint sends RST, I expect a blocking send call to return (with an error). Even if I put MSG_DONTWAIT in the flags to enable non-blocking mode, the function hangs in the exact same way.
If I try to send a smaller buffer, like 1000 bytes instead, then the function does not hang anymore. If I try to send the same 4000 bytes big buffer, but increase the receive window size on the remote server, everything also works fine.
Obviously there is some bug in the implementation that happens with small receive windows when the user tries to send a large packet.
nRF Connect SDK version: 2.3.0.
Modem firmware version: 1.3.4.
Mobile operator: Telenor SE.
