Why is my PPP connection failing using the modem_shell on the nRF9160dk to talk to a Dart-6UL over uart? Also what are these errors I'm seeing on the MoSH even when a PPP connection is successful?

[MichelleV]

I'm trying to connect the nrf9160dk from its uart0 running the modem_shell sample code from SDK v2.2.0 to our Dart6ul on its tty4 uart running at 921600 baud using a PPP connection. I have been able to use the modem_shell on the nrf9160 and ubuntu laptop successfully. But I'm getting errors in both configurations that I don't understand completely. When trying to replace the ubuntu laptop (ttyACM0) with the Dart6UL (ttymxc4), I use the same command (changing the port) to initiate the PPP connection from the unix device:

 pppd -detach /dev/ttymxc4 921600 noauth crtscts noccp novj nodeflate nobsdcomp local debug +ipv6 ipv6cp-use-ipaddr usepeerdns noipdefault defaultroute ipv6cp-restart 5 ipcp-restart 5 lcp-echo-interval 0

I have been able to successfully run the Serial LTE Modem example on my 9160 and Dart-6ul with these same devicetree settings to use the ttymxc4 and uart0.

When running the mosh, I see these same errors on the MoSH terminal (regardless of ubuntu or dart-6ul):

[00:00:04.657,104] <wrn> net_if: iface 0x200127c4 is down

[00:00:34.670,959] <err> net_sock: invalid access on sock -1 by thread 0x20012ad0

[00:00:36.702,850] <err> net_ppp: uart_rx_enable() failed, err -16

And the bottom two repeat. I'm attaching photos of the ubuntu and the dart-6ul outputs from MoSH prompt. In addition the Dart-6ul is not completing the setup of the PPP connection. I see the PPP start but then it fails out and eventually gives me a fork error. I'm attaching my net-connect and net-chat file running on the Dart-6ul. And a log file that the net-connect kicks off.

Any help getting these understood and me pointed in the right direction would be greatly appreciated!

9160dk_mosh_errors.jpg shows the startup of the LTE connection and the issues identified above.

dart_6ul_fail1.jpg shows that the PPP did initiate but it doesn't appear as a network interface and then displays the fork error.

Fullscreen net-connect.txt Download

#!/bin/bash
exec > ppp_connect_log.txt 2>&1

/usr/sbin/pppd -detach /dev/ttymxc4 921600 noauth crtscts noccp novj nodeflate nobsdcomp local debug +ipv6 ipv6cp-use-ipaddr usepeerdns noipdefault defaultroute ipv6cp-restart 5 ipcp-restart 5 lcp-echo-interval 0

Fullscreen net-chat.txt Download

ABORT 'NO CARRIER'
ABORT 'NO DIALTONE'
ABORT 'ERROR'
'' ATZ
OK 'AT+CGDCONT=1,"IP","hologram"'
OK 'ATDT*99***1#'
CONNECT \d\c

Fullscreen ppp_connect_log.txt Download

Script /etc/ppp/net-connect finished (pid 511), status = 0x8
Connect script failed

ce temporarily unavailable
Connect script failed

[Didrik Rokhaug]

Hi,

When running the mosh, I see these same errors on the MoSH terminal (regardless of ubuntu or dart-6ul):

I'll check with the developers, but I think you can ignore them.

see the PPP start but then it fails out and eventually gives me a fork error.

This seems to be a problem with your Linux setup, not the nRF9160 or the modem_shell.

One thing you should check is how many child processes ppp can create, and consider increasing the limit.

Best regards,

Didrik

[MichelleV]

I had also started a ticket with variscite on the linux side. Of course, they thought it was more of a nRF9160 problem with the uart setup. And sent me the following link: https://devzone.nordicsemi.com/f/nordic-q-a/101569/error-when-using-date_time_ntp-in-ncs-2-4-0

Both you and variscite have suggested increasing the child processes ppp can create. I'm a little confused because I have no idea how many would be expected and they way that they were popping up looks to me like they would just continue until the new limit is reached. If you don't have a number of max processes I should expect, can you point me to somewhere with this kind of information?

[Didrik Rokhaug]

MichelleV said:

An additional question, I was just looking into enabling modem trace but there is a line in the overlay-ppp.conf (line 118) that states "UART traced cannot be enabled". Does that mean I can't get a modem trace while using the PPP configuration?

At the moment (this will change in the next NCS release), the UART backend for modem traces is hard-coded to use UART1. It can be changed to use any UART, you just have to manually edit the source file, rather than doing it with the device tree.

In ppp.overlay, UART0 is chosen for PPP, and UART1 is chosen for the shell. As this conflicts with the UART used for modem traces, modem tracing is disabled.

However, you can use other UARTs than the ones chosen in the overlay file, or you can change which UART is used for modem traces. Another limitation is that on the nRF9160 DK, only two UARTs are routed to the interface chip, so you would have to use an external UART-to-USB converter to get the trace.

A third option is to send the modem trace over RTT instead, by enabling CONFIG_NRF_MODEM_LIB_TRACE_BACKEND_RTT. It isn't as stable as the UART backend, and we don't have any tools for receiving the trace (so you would have to use the tools from Segger), which is why it isn't our preferred option, but it is a possible solution.

MichelleV said:

Is it required to have two individual uart interfaces for the 9160 to dart-6ul interface?

If you need to control the modem? Yes, I believe so.

The modem will try to re-establish network connection if the connection is lost, and the modem shell can be configured to restart if it crashes, so you might be able to get by without any extra control of the modem in some use cases.

However, in general, yes, you would need to use two UARTs.

If the limit is in the number of UARTs on the host processor, you might also be able to work around it with a multiplexer or switches.

[MichelleV]

Didrik Rokhaug said:

MichelleV said:

Is it required to have two individual uart interfaces for the 9160 to dart-6ul interface?

If you need to control the modem? Yes, I believe so.

What kind of control do mean here? The modem_shell should handle the modem operation, correct?

[Didrik Rokhaug]

Turing the LTE link on and off, enabling/disabling PSM, etc. through the modem_shell.

I.e. use of the shell interface itself, not just PPP.

[MichelleV]

Maybe I'm missing something here but wouldn't the application be sending commands and data from the dart-6ul via the ppp0 connection over the uart and receiving responses and data directly to the ppp uart on the nrf9160? Isn't that why the net-connect and net-chat files are needed on the dart-6ul side to interpret the data from the ppp uart and send the correct responses? I guess I'm unsure why we would need the additional uart for the console/mosh uart in our end application where we want the debian network interface to use the 9160 as a radio co-processor.

[Didrik Rokhaug]

No, it is not possible to send AT commands over the PPP interface.

The PPP implementation is quite basic, so authentication, etc. which is what you typically use net-connect and net-chat for isn't supported.

There is (at least on Ubuntu, which is what we test against) no need for net-connect or net-chat.

Instead, you will have to use the shell interface if you want any manual control over the modem.

The modem shell sample should try to connect to the network and start the PPP interface automatically, so you might not need any extra control over the modem. It is also possible to add some commands that are run on startup, if needed.

[MichelleV]

One thing I don't understand, I've been able to sucessfully get a ppp connection running on a variscite dart 6ul via usb but still can't via uart. I have verified that I'm able to send and receive data from the uart but the ppp request continues to fail as above. Do you happen to know the linux requirements for the ppp connection on the nrf9160?

[Didrik Rokhaug]

MichelleV said:

I've been able to sucessfully get a ppp connection running on a variscite dart 6ul via usb but still can't via uart.

How did you connect the dart 6ul to the nRF9160 with USB?

It might be some flow control issue?

Another thing you could try is to use a logic analyzer to capture the activity on the UART lines in the cases where it works vs. when it doesn't. There might be some differences there that can indicate what is wrong.

MichelleV said:

I was also told that since the modem_shell is a "test" application, I really can't get any more assistance. We are trying to run this in the configuration we will need for the future product but we are still very much in the test / evaluation phase of the project. Can I continue to reach out to you to get this figured out?

We will still try our best to help you use the nRF9160 and the modem shell, but "our best" is a bit more limited in this case, compared to more common use cases.

Also, because the modem shell is primarily intended to be used when testing the nRF9160, the testing has been limited to desktop Linux (Ubuntu), and not single board computers with custom Linux kernels, or other similar devices.

And unfortunately, our knowledge falls a bit short on that front.

You can also try to reach out to the Zephyr community. The PPP implementation comes from Zephyr, and is not something we have done ourselves. So if you ask there, they might be able to provide more details.

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/2.4.0/zephyr/introduction/index.html#community-support

[MichelleV]

I do have a couple of logic captures...I'll dig through those here in a bit and see if there is anything that jumps out.

I took my 9160DK and plugged the usb cable from it into the top usb jack on my variscite dart-6ul dk. I was then able to get the PPP connection to the 9160 via the ttyACM0 device. 

I'll check out the Zephyr community as well. Thank you!