nRF9160 goes into cpu idle after sending data

RE: nRF9160 goes into cpu idle after sending data

nWre — Wed, 23 Mar 2022 14:00:22 GMT

Well Einarh, you are a certified genius. It works very well. I tip my hat and thank you for the help.

RE: nRF9160 goes into cpu idle after sending data

Einarh — Wed, 23 Mar 2022 13:21:57 GMT

Ok I have a suggestion.

I assume it's the main "send" function that prevents you from sending more than 4kB at a time.

If you find "do_ftp_send_data" in ftp_client.c, and you modify the code to always limit the input to "send" to 4kB, would that work?

Something like

[...]
while(offset < length) {
    if ((length - offset) > {4kB} { 
        sendlen = {4kB}
    } else { 
        sendlen = (length - offset)
    }
    ret = send(client.data_sock, message + offset, sendlen, 0);
[...]

RE: nRF9160 goes into cpu idle after sending data

nWre — Wed, 23 Mar 2022 12:45:37 GMT

Correct! And not just a connection timed out but the cpu idles so it doesn't continue executing. I've had timed out before where it just kept executing from where it was.

RE: nRF9160 goes into cpu idle after sending data

Einarh — Wed, 23 Mar 2022 12:44:32 GMT

hm yes it does look like this is what you're doing in your example.

So after you run ftp_put 109 times in this example you get the connection timed out error?

RE: nRF9160 goes into cpu idle after sending data

nWre — Wed, 23 Mar 2022 10:31:46 GMT

I see, that's what I'm already doing in the example code, no?

Regards

nWre

RE: nRF9160 goes into cpu idle after sending data

Einarh — Wed, 23 Mar 2022 10:13:04 GMT

I don't think you should run the k_work functions manually, if you use ftp_put then the FTP client will make sure the sequencing is done properly.

What I mean is you should try running ftp_put for each 4kB segment without running uninit etc in between.

-Einar

RE: nRF9160 goes into cpu idle after sending data

nWre — Wed, 23 Mar 2022 09:37:50 GMT

Hi again!

Yes, I've tried that but unless it follows the flow described in ftp_put it doesn't work. You either get an error saying bad sequence of commands or nothing at all happens.

as described, the flow is:

/** Typical sequence:
   * FTP       51   Request: PASV
   * FTP       96   Response: 227 Entering Passive Mode (90,130,70,73,105,177).
   * FTP       63   Request: STOR upload2.txt
   * FTP-DATA   53   FTP Data: 8 bytes (PASV) (STOR upload2.txt)
   * FTP       67   Response: 150 Ok to send data.
   * FTP       69   Response: 226 Transfer complete.
   */

but if I do:

        for (int i = 0; i < 10; i++) {
           k_work_submit_to_queue(&ftp_work_q, &data_task_param.work);
           ret = poll_data_task_done();
       }

or even

        for (int i = 0; i < 10; i++) {
           k_work_submit_to_queue(&ftp_work_q, &data_task_param.work);
       }

it doesn't work.

In another ftp thread a user said he managed to avoid socked disconnecting on each put and that he transferred a 10mb file so this should be possible, I just can't get it to work.

RE: nRF9160 goes into cpu idle after sending data

Einarh — Wed, 23 Mar 2022 09:30:40 GMT

Yes it's correct that the modem can't send more than 4kB at a time, I thought the "send" function might be able to figure that out but I guess not.

It does seem like the FTP put function automatically closes the data connection when the transfer is completed though, so it might be the case that you're not supposed to uninit and open between each consecutive transfer.

It could be that you're clogging the available connections with control message connections since you're opening many connections quickly?

Remember that when you issue a put command, the transfer is placed in a queue, and isn't executed immediately.

I would try just queueing your transfers without disconnecting between each one if you haven't already.

-Einar

RE: nRF9160 goes into cpu idle after sending data

nWre — Tue, 22 Mar 2022 15:04:07 GMT

If I try to send a buffer larger than 4kB I get "E: send data failed: -122" and 122 error code is "Message too long".

Edit: And yes, I slice it myself. I read 4kB from my file into a buffer, send via ftp, read next 4kB, send via ftp etc.

Edit2: I should mention that a previous thread about FTP on this forum said that 4kB was a restriction without a workaround, something with the modem firmware.

RE: nRF9160 goes into cpu idle after sending data

Einarh — Tue, 22 Mar 2022 14:57:23 GMT

Are you slicing the file into 4kB parts yourself before sending?

Because it looks like the ftp code will do this for you, have you tried just sending the whole file?

And when you send the "PASV" command I believe you are telling the server to enter passive mode to prepare to receive data. "Pass" is related to sending password credentials.

You can read more about how FTP works here.

-Einar

RE: nRF9160 goes into cpu idle after sending data

nWre — Tue, 22 Mar 2022 14:54:06 GMT

Here is a screenshot of the output where it fails:

so it fails after 109 sends, as you can see, the port number is incremented each segment and I think this might be the cause of the problem.

RE: nRF9160 goes into cpu idle after sending data

nWre — Tue, 22 Mar 2022 14:06:52 GMT

Hi!

No it doesn't matter. If I send a "file" large enough it will halt. What I'm doing is reading from a bin-file that I have stored on a micro-sd card and then sending 4kB at a time from that with the append command.

Each time the "PASV" command is sent, a new port on the server is opened and my thinking was that maybe this is a limitation, that if you open enough ports, eventually the server will "lock".

If there is a way to use the same port, maybe by sending "PASS" instead of "PASV" or manually saving the port that was previously used this wouldn't happen. I've tried doing this but I get error bad sequence of commands.

After each segment sent, poll_task_done(); is called and I imagine the thread is deconstructed.

The error occurs after about 100 segments, I will check the exact amount. It is not based on time but in segments and I think opening 100 ports for passive connection might be the problem but I see no way around it.

Regards
nWre

RE: nRF9160 goes into cpu idle after sending data

Einarh — Tue, 22 Mar 2022 13:18:35 GMT

I'm looking through the program flow in ftp_client.c, and it looks like the file you send gets added to a workqueue as a single workitem.

I could imagine that the work thread might starve the other threads, including the heartbeat timer, if the workitem is sufficiently large, causing your issue.

Do you see the same behavior if you send the 70 packets right away without sending the 40 packets first?

How long does it take before you get the error when you start sending your big file? Is it a matter of seconds?

Also, from what I understand FTP keeps two separate connections for control messages and data transfer, which is why the connection would still be up after a timeout in the data connection.

Best regards,

Einar

RE: nRF9160 goes into cpu idle after sending data

nWre — Mon, 21 Mar 2022 14:03:48 GMT

Hi! Here is a test version:

Main file

#include "ftptest.c"

void main(void)
{
    tfirstinit();
	tftpinit();
	for (int i = 0;i<120;i++){
		tftptest();
	}
}

ftptest.c

#include <zephyr.h>
#include <net/socket.h>
#include <stdio.h>
#include <drivers/uart.h>
#include <string.h>
#include <logging/log.h>
#include <ctype.h>
#include <net/tls_credentials.h>
#include <net/ftp_client.h>
#include <fs/fs.h>
#include <ff.h>
#include <device.h>
#include <drivers/spi.h>
#include <sys/printk.h>
#include <disk/disk_access.h>
#include <logging/log.h>

int treturnValue;

K_SEM_DEFINE(tmy_sem, 0, 1);
uint64_t testData[500] = {0};

void tftp_ctrl_callback(const uint8_t *msg, uint16_t len)
{
    printk("%s", msg);
    /* Release semaphore when receiving answer from server */
    k_sem_give(&my_sem);
}

void tftp_data_callback(const uint8_t *msg, uint16_t len)
{
    printk("%s", msg);
}

int tfirstinit(void){
    treturnValue = ftp_init(tftp_ctrl_callback, tftp_data_callback);
    return treturnValue;
}

int tftpinit(void)
{
    treturnValue = ftp_open("MYSERVER", 21, -1);
    treturnValue = ftp_login("ACCNAME", "ACCPASS");
    return treturnValue;
}

int tftptest()
{
    for (int i = 0;i<500;i++){
        testData[i] = 0;
    } 
    int retVal = 0;
    retVal = ftp_put("test.bin", testData, sizeof(testData), FTP_PUT_APPEND);
    printk("put procedure says: %i \n", retVal);
    return retVal;
}

int tftpuninit(void){
    treturnValue = ftp_uninit();
    return treturnValue;
}

and prjconf

CONFIG_SPI=y                            # SPI hardware bus
CONFIG_SPI_SLAVE=y                      # SPI slave support [EXP]

CONFIG_LTE_AUTO_INIT_AND_CONNECT=y
CONFIG_NCS_SAMPLES_DEFAULTS=y
# LTE link control
CONFIG_LTE_LINK_CONTROL=y
# Modem library
CONFIG_NRF_MODEM_LIB=y
# Modem library
CONFIG_NRF_MODEM_LIB_SYS_INIT=y
# FTP Client
CONFIG_FTP_CLIENT=y

CONFIG_DISK_DRIVER_SDMMC=y
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_SPI_NRFX_RAM_BUFFER_SIZE=8
CONFIG_FILE_SYSTEM=y
CONFIG_FAT_FILESYSTEM_ELM=y

# General config
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

# Network
CONFIG_PRINTK=y
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=n
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_OFFLOAD=y

CONFIG_HEAP_MEM_POOL_SIZE=4096
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
CONFIG_PRINTK=y
CONFIG_ASSERT=y
CONFIG_STDOUT_CONSOLE=y
CONFIG_UART_INTERRUPT_DRIVEN=n
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_NET_BUF_USER_DATA_SIZE=1000
CONFIG_LOG=y
CONFIG_SDMMC_LOG_LEVEL_DBG=n

As each 4kB segment sent increments the portnumber(?) as shown by the output, I'm thinking I might reach a maximum amounts of connections to the server. I'm not that familiar with the FTP protocol but it feels like I shouldn't be entering passive mode for each segment but only once and then just send data.

Thanks again!

RE: nRF9160 goes into cpu idle after sending data

Einarh — Mon, 21 Mar 2022 09:35:53 GMT

Hello again!

Could you please share the most basic setup that causes the issue, so I can try to replicate?

Also, have you looked into whether the issue could be server side, since your error message is connection related?

Best regards,

Einar