nRF9160 PPP stability

[Bernhard]

Hello,

we are trying to use the nRF9160 as a serial modem for a Raspberry Pi running on Debian.

The application is basically the nRF sample "nrf\samples\cellular\modem_shell" without major changes (compiled without CMUX as described for Linux target systems),
modem firmware is v1.3.6, board is nrf9160dk_nrf9160_ns.

We are using the PPP channel only (without a second UART), the application is configured for 'autostart' and no additional commands (including AT) are sent from the Raspberry Pi.

This is working in principle, but we are facing problems with long term stability, the main application is running for a while but after a few hours the connection may get lost.

1. Is the application even tested for longer runtime or just for e.g. single data transmissions?

2. Should this basically be working as we are trying it now or is a more complex control (e.g. restart routine) via AT command channel required in the concept ?

Also we had the problem that the application is not starting up with RTT logging, only after RTT Viewer was connected.  (?)
We therefore for now had to configure the logging to a non existing UART channel (pins are floating), which is why I don't have any logs of the events.

Thank you!

Best regards,
Bernhard

[dejans]

Hi,

Also we had the problem that the application is not starting up with RTT logging, only after RTT Viewer was connected.  (?)

If you want to use RTT backend instead of UART, you need to enable RTT support by adding overlay-rtt.conf file to your build command. 

This is working in principle, but we are facing problems with long term stability, the main application is running for a while but after a few hours the connection may get lost.

What happens after connection gets lost? Could you describe stability issue in more detail?
Do you connect to LTE-M or NB-IoT network?

Best regards,
Dejan

[Bernhard]

Hello Dejans,

dejans said:

If you want to use RTT backend instead of UART, you need to enable RTT support by adding overlay-rtt.conf file to your build command.

I have done that and RTT logging is working, but only after the 'Connect' in RTT Viewer, the application is not starting up on it's own.

dejans said:

Do you connect to LTE-M or NB-IoT network?

LTE-M

dejans said:

What happens after connection gets lost? Could you describe stability issue in more detail?

For example I got following error messages:

1. longer time without an error

2. a lot of these:

00> ppp_modem_dl_data_thread_handler: recv() from modem failed, -1, errno: -115
00> ppp_modem_dl_data_thread_handler: recv() from modem failed, -1, errno: -115
00> ppp_modem_dl_data_thread_handler: recv() from modem failed, -1, errno: -115

3. until:

00> [01:16:49.348,419] <wrn> net_ppp: 66 written to RX buf, but after that only 498 space left. Disabling RX for now.
00> . Disabling RX for now.
00>  leftrtt:~$ [01:47:14.208,435] <wrn> at_monitor: No heap space for incoming notification: %CESQ: 23,1,0,0
00> 
00> rtt:~$ [01:47:14.220,275] <err> os: r0/a1:  0x00000004  r1/a2:  0x0000004f  r2/a3:  0x00000008
00> rtt:~$ [01:47:14.220,306] <err> os: r3/a4:  0x0003187d r12/ip:  0x00000000 r14/lr:  0x000387cf
00> rtt:~$ [01:47:14.220,336] <err> os:  xpsr:  0x4100003a
00> rtt:~$ [01:47:14.220,336] <err> os: s[ 0]:  0x000388b1  s[ 1]:  0x2000dea8  s[ 2]:  0x2000dd74  s[ 3]:  0x2000dd6c
00> rtt:~$ [01:47:14.220,367] <err> os: s[ 4]:  0x00000002  s[ 5]:  0x2001c8a6  s[ 6]:  0x00000000  s[ 7]:  0x00042bc1
00> rtt:~$ [01:47:14.220,367] <err> os: s[ 8]:  0x00042b51  s[ 9]:  0x00000001  s[10]:  0x2000dd74  s[11]:  0x000421e1
00> rtt:~$ [01:47:14.220,397] <err> os: s[12]:  0x2001c41c  s[13]:  0x2001c33c  s[14]:  0x00000001  s[15]:  0x00000000
00> rtt:~$ [01:47:14.220,428] <err> os: fpscr:  0x20011e08
00> rtt:~$ [01:47:14.220,458] <err> os: r4/v1:  0x2000dec0  r5/v2:  0x00000000  r6/v3:  0x20027f28
00> rtt:~$ [01:47:14.220,458] <err> os: r7/v4:  0x20027f28  r8/v5:  0x00000000  r9/v6:  0x20011500
00> rtt:~$ [01:47:14.220,489] <err> os: r10/v7: 0x00000000  r11/v8: 0x00000000    psp:  0x20022588
00> rtt:~$ [01:47:14.220,489] <err> os: EXC_RETURN: 0x0
00> rtt:~$ [01:47:14.220,489] <err> os: Faulting instruction address (r15/pc): 0x0004e2e2
00> rtt:~$ [01:47:14.220,550] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
00> rtt:~$ [01:47:14.220,550] <err> os: Fault during interrupt handling
00> 
00> rtt:~$ [01:47:14.220,611] <err> os: Current thread: 0x20011e08 (ppp_modem_dl_data_thread)
00> rtt:~$ [01:47:14.314,971] <err> os: Halting system

[dejans]

Hi,

You could try to run addr2line tool on the faulting instruction address to get the line in the code which corresponds to the faulting address.
On Linux, addr2line can be run as
 /toolchains/<toolchain_version>/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line -e build/zephyr/zephyr.elf 0x0004e2e2

Best regards,
Dejan

[Bernhard]

Hello Dejan,

output of the tool is:

C:/ncs/v2.6.0/zephyr/lib/os/printk.c:227

which is the first 'if' statement in:

static int str_out(int c, struct str_context *ctx)
{
	if (ctx->str == NULL || ctx->count >= ctx->max) {
		ctx->count++;
		return c;
	}

	if (ctx->count == ctx->max - 1) {
		ctx->str[ctx->count++] = '\0';
	} else {
		ctx->str[ctx->count++] = c;
	}

	return c;
}

[dejans]

Hi Bernhard,

Thank you for additional information.

I am looking at one of your previous comments.

Bernhard said:

1. longer time without an error

2. a lot of these:

How long does your application work as expected, without errors?
When do the errors 

00> ppp_modem_dl_data_thread_handler: recv() from modem failed, -1, errno: -115

appear? How often do they appear? How did the device recover when they started appearing? 

Bernhard said:

3. until:

What happens after you receive FATAL ERROR? How did the device recover?

How often do errors mentioned in 2. and 3. occur? Do you see recurrent behavior?

Which LTE network do you connect to?

Best regards,
Dejan