[Regression ncs v1.8.0] nRF9160-DK , Modem_shell demo , HARD FAULT after connecting over ppp

[adam_h]

Hello,

I was testing modem_shell demo in terms of using ppp. I'm using nrf9160dk board. 

I found issue present in ncs v1.7.1 https://devzone.nordicsemi.com/f/nordic-q-a/80776/nrf9160-dk-modem_shell-demo-ppp-unable-to-access-the-internet

When I'm using  v1.7.1 with applied changes from PR as advised, I'm able to connect with ppp client (from linux I'm executing: sudo pppd -detach /dev/ttyACM0 115200 noauth nocrtscts local ) without any issues

When I'm using ncs v1.8.0 which seems to have fix for this issue merged in (I verified that changes are present in cs/zephyr/subsys/net/ip/packet_socket.c), the problem with hard fault is still present.

[Håkon Alseth]

Hi,

 

Thank you for reporting this back to us. I can confirm that I see the same fault occuring from the "ppp_workq" thread.

I will report this internally.

 

Kind regards,

Håkon

[Håkon Alseth]

I got help from R&D to debug this issue, and the fault you're seeing is highly likely a side-effect to the ipv6 support having been added to the modem_shell.

If you add the flag "noccp" to the pppd options, it should work as intended:

$ sudo pppd -detach /dev/ttyACM0 115200 noauth nocrtscts local noccp
Using interface ppp0
Connect: ppp0 <--> /dev/ttyACM0
local  LL address fe80::xyz
remote LL address fe80::xyzz
local  IP address x.y.z.zz
remote IP address x.y.z.zz

 

Could you try this and report back?

 

Kind regards,

Håkon

[adam_h]

Hi ,
In case of hard fault that helped.
Right now I'm getting:

Using interface ppp0
Connect: ppp0 <--> /dev/ttyACM0
LCP: Rcvd Code-Reject for code 192, id 33
local IP address x.y.z.zz
remote IP address x.y.z.zz
LCP: Rcvd Code-Reject for code 192, id 33
LCP: Rcvd Code-Reject for code 192, id 33
LCP: Rcvd Code-Reject for code 192, id 33
LCP: Rcvd Code-Reject for code 192, id 33
LCP: Rcvd Code-Reject for code 192, id 33




[Håkon Alseth]

Hi,

 

Does the shell (uart1) output show anything specific?

You can also add "debug" to the pppd options to get a more verbose output on the PC side.

 

Are you able to for instance ping from the PC side?

ping -I ppp0 google.com

 

Kind regards,

Håkon

[adam_h]

There is nothing interesting on shell uart:

*** Booting Zephyr OS build v2.7.0-ncs1  ***

MOSH version:       v1.8.0
MOSH build id:      custom
MOSH build variant: dev

Initializing modemlib...


mosh:~$ Initialized modemlib


Network registration status: searching
Currently active system mode: LTE-M
LTE cell changed: ID: 61565196, Tracking area: 54003
RRC mode: Connected
PDN event: PDP context 0 activated
Network registration status: Connected - roaming
PSM parameter update: TAU: 10800, Active time: -1 seconds
Default PDN is active: starting PPP automatically.
modem data socket 4 created for modem data
PPP Zephyr data socket 5 created
Modem config for system mode: LTE-M
Modem config for LTE preference: No preference, automatically selected by the modem
Currently active system mode: LTE-M
Modem FW version:      mfw_nrf9160_1.2.3
Operator full name:   "Orange"
Operator short name:  "Orange"
Operator PLMN:        "26003"
Current cell id:       61565196 (0x03AB690C)
Current phy cell id:   421
Current band:          20
Current TAC:           54003 (0xD2F3)
Current rsrp:          26: -115dBm
Current snr:           27: 3dB
Mobile network time and date: 22/01/05,12:15:27+04
PDP context info 1:
  CID:                0
  PDN ID:             0
  PDP context active: yes
  PDP type:           IP
  APN:                xxx
  IPv4 MTU:           1500
  IPv4 address:       x.y.z.zz
  IPv6 address:       ::
  IPv4 DNS address:   x.y.z.zz, x.y.z.zz
  IPv6 DNS address:   ::, ::
PPP carrier ON
RRC mode: Idle
Dial up (IPv4) connection up
RRC mode: Connected
RRC mode: Idle
RRC mode: Connected
RRC mode: Idle
RRC mode: Connected

From PC pppd cal with debug I can see:

~ sudo pppd -detach /dev/ttyACM0 115200 noauth nocrtscts local noccp debug
using channel 6
Using interface ppp0
Connect: ppp0 <--> /dev/ttyACM0
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0xf62fbb22> <pcomp> <accomp>]
rcvd [LCP ConfRej id=0x1 <asyncmap 0x0> <magic 0xf62fbb22> <pcomp> <accomp>]
sent [LCP ConfReq id=0x2]
rcvd [LCP ConfAck id=0x2]
rcvd [LCP ConfReq id=0x1 <mru 1500>]
sent [LCP ConfAck id=0x1 <mru 1500>]
sent [LCP EchoReq id=0x0 magic=0x0]
sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x1 <addr x.y.z.zz> <ms-dns1 10.105.16.254> <ms-dns2 10.105.144.254>]
sent [IPCP ConfRej id=0x1 <ms-dns1 10.105.16.254> <ms-dns2 10.105.144.254>]
rcvd [IPV6CP ConfReq id=0x1 <addr fe80::0000:5eff:fe00:5371>]
Unsupported protocol 'IPv6 Control Protocol' (0x8057) received
sent [LCP ProtRej id=0x3 80 57 01 01 00 0e 01 0a 00 00 5e ff fe 00 53 71]
rcvd [LCP EchoRep id=0x0 magic=0x0]
rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>]
sent [IPCP ConfReq id=0x2 <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x2 <addr x.y.z.zz>]
sent [IPCP ConfAck id=0x2 <addr x.y.z.zz>]
rcvd [LCP CodeRej id=0x3 c0 21 08 03 00 14 80 57 01 01 00 0e 01 0a 00 00 5e ff]
LCP: Rcvd Code-Reject for code 192, id 33
rcvd [IPCP ConfNak id=0x2 <addr x.y.z.zz>]
sent [IPCP ConfReq id=0x3 <addr x.y.z.zz>]
rcvd [IPCP ConfAck id=0x3 <addr x.y.z.zz>]
local  IP address x.y.z.zz
remote IP address x.y.z.zz
Script /etc/ppp/ip-up started (pid 14054)
Script /etc/ppp/ip-up finished (pid 14054), status = 0x0
rcvd [IPV6CP ConfReq id=0x1 <addr fe80::0000:5eff:fe00:5371>]
Unsupported protocol 'IPv6 Control Protocol' (0x8057) received
sent [LCP ProtRej id=0x4 80 57 01 01 00 0e 01 0a 00 00 5e ff fe 00 53 71]
rcvd [LCP CodeRej id=0x4 c0 21 08 04 00 14 80 57 01 01 00 0e 01 0a 00 00 5e ff]
LCP: Rcvd Code-Reject for code 192, id 33

I tried to ping time.google.com (ip: 216.239.35.0) as I know it is on white list of my sim card but also without success.
Strange thing is that I'm also not able to ping it from modem shell. I need to make some more tries as I think I was able to do it in the past

[Håkon Alseth]

Hi,

 

Everything looks to be up and running, there's no errors or "bad" warnings, other than ipv6 not being supported.

However, the RSRP shows that your signal strength is quite bad. Are you able to move to an area in your building where the signal strength is a bit better and retry the ping procedure from the PC side?

 

You mention whitelisting, do you have another sim card you can try that has no restrictions in terms of DNS/ip-range?

 

Kind regards,

Håkon

[adam_h]

Hi,

I moved from my office to my home and made tests once again.

I was able to execute ping with success, but the way to achieve this seems strange for me. Below are the steps that I made

1. Device has connected

*** Booting Zephyr OS build v2.7.0-ncs1  ***

MOSH version:       v1.8.0
MOSH build id:      custom
MOSH build variant: dev

Initializing modemlib...


mosh:~$ Initialized modemlib


Network registration status: searching
Currently active system mode: LTE-M
LTE cell changed: ID: 61677067, Tracking area: 54011
RRC mode: Connected
PDN event: PDP context 0 activated
Network registration status: Connected - roaming
PSM parameter update: TAU: 10800, Active time: -1 seconds
Default PDN is active: starting PPP automatically.
modem data socket 4 created for modem data
PPP Zephyr data socket 5 created
Modem config for system mode: LTE-M
Modem config for LTE preference: No preference, automatically selected by the modem
Currently active system mode: LTE-M
Modem FW version:      mfw_nrf9160_1.2.3
Operator full name:   "Orange"
Operator short name:  "Orange"
Operator PLMN:        "26003"
Current cell id:       61677067 (0x03AD1E0B)
Current phy cell id:   336
Current band:          20
Current TAC:           54011 (0xD2FB)
Current rsrp:          66: -75dBm
Current snr:           43: 19dB
Mobile network time and date: 22/01/05,17:46:04+04
PDP context info 1:
  CID:                0
  PDN ID:             0
  PDP context active: yes
  PDP type:           IP
  APN:                cellular.firmwave.gdsp
  IPv4 MTU:           1500
  IPv4 address:       x.y.z.zz
  IPv6 address:       ::
  IPv4 DNS address:   a.b.c.cc, a.b.c.cc
  IPv6 DNS address:   ::, ::
PPP carrier ON
RRC mode: Idle
Dial up (IPv4) connection up
RRC mode: Connected

From my PC:

➜  ~ sudo pppd -detach /dev/ttyACM0 115200 noauth nocrtscts local noccp debug
using channel 5
Using interface ppp0
Connect: ppp0 <--> /dev/ttyACM0
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x14e69c84> <pcomp> <accomp>]
rcvd [LCP ConfRej id=0x1 <asyncmap 0x0> <magic 0x14e69c84> <pcomp> <accomp>]
sent [LCP ConfReq id=0x2]
rcvd [LCP ConfAck id=0x2]
rcvd [LCP ConfReq id=0x1 <mru 1500>]
sent [LCP ConfAck id=0x1 <mru 1500>]
sent [LCP EchoReq id=0x0 magic=0x0]
sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x1 <addr x.y.z.zz> <ms-dns1 a.b.c.cc> <ms-dns2 a.b.c.cc>]
sent [IPCP ConfRej id=0x1 <ms-dns1 a.b.c.cc> <ms-dns2 a.b.c.cc>]
rcvd [IPV6CP ConfReq id=0x1 <addr fe80::0000:5eff:fe00:5381>]
Unsupported protocol 'IPv6 Control Protocol' (0x8057) received
sent [LCP ProtRej id=0x3 80 57 01 01 00 0e 01 0a 00 00 5e ff fe 00 53 81]
rcvd [LCP EchoRep id=0x0 magic=0x0]
rcvd [IPCP ConfRej id=0x1 <compress VJ 0f 01>]
sent [IPCP ConfReq id=0x2 <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x2 <addr x.y.z.zz>]
sent [IPCP ConfAck id=0x2 <addr x.y.z.zz>]
rcvd [LCP CodeRej id=0x3 c0 21 08 03 00 14 80 57 01 01 00 0e 01 0a 00 00 5e ff]
LCP: Rcvd Code-Reject for code 192, id 33
rcvd [IPCP ConfNak id=0x2 <addr x.y.z.zz>]
sent [IPCP ConfReq id=0x3 <addr x.y.z.zz>]
rcvd [IPCP ConfAck id=0x3 <addr x.y.z.zz>]
local  IP address x.y.z.zz
remote IP address x.y.z.zz
Script /etc/ppp/ip-up started (pid 8511)
Script /etc/ppp/ip-up finished (pid 8511), status = 0x0
rcvd [IPV6CP ConfReq id=0x1 <addr fe80::0000:5eff:fe00:5381>]
Unsupported protocol 'IPv6 Control Protocol' (0x8057) received
sent [LCP ProtRej id=0x4 80 57 01 01 00 0e 01 0a 00 00 5e ff fe 00 53 81]
rcvd [LCP CodeRej id=0x4 c0 21 08 04 00 14 80 57 01 01 00 0e 01 0a 00 00 5e ff]
LCP: Rcvd Code-Reject for code 192, id 33
rcvd [IPV6CP ConfReq id=0x1 <addr fe80::0000:5eff:fe00:5381>]
Unsupported protocol 'IPv6 Control Protocol' (0x8057) received
sent [LCP ProtRej id=0x5 80 57 01 01 00 0e 01 0a 00 00 5e ff fe 00 53 81]
rcvd [LCP CodeRej id=0x5 c0 21 08 05 00 14 80 57 01 01 00 0e 01 0a 00 00 5e ff]
LCP: Rcvd Code-Reject for code 192, id 33

2. I made ping to time.google.com without success:

~ ping -I ppp0 time.google.com
PING  (216.239.35.4) from x.y.z.zz ppp0: 56(84) bytes of data.
^C
---  ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9224ms

3. From modem shell I executed ping cmd:

mosh:~$ ping
Usage: ping [optional options] -d destination

mandatory options:
  -d, --destination, [str] Name or IP address
optional options:
  -t, --timeout, [int]     Ping timeout in milliseconds
  -c, --count, [int]       The number of times to send the ping request
  -i, --interval, [int]    Interval between successive packet transmissions
                           in milliseconds
  -l, --length, [int]      Payload length to be sent
  -I, --cid, [int]         Use this option to bind pinging to specific CID.
                           See link cmd for interfaces
  -6, --ipv6,              Force IPv6 usage with the dual stack interfaces
  -r, --rai                Set RAI options for ping socket. In order to use RAI,
                           it must be enabled with 'link rai' command.
  -h, --help,              Shows this help information

RRC mode: Idle
RRC mode: Connected
RRC mode: Idle

and then once again I made pink from my P

➜  ~ ping -I ppp0 time.google.com
PING  (216.239.35.4) from x.y.z.zz ppp0: 56(84) bytes of data.
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
From 198.18.210.132 (198.18.210.132) icmp_seq=0 Destination Host Unreachable
^C
---  ping statistics ---
8 packets transmitted, 0 received, +8 errors, 100% packet loss, time 7011ms

4. I executed from modem shell ping time.google.com 

mosh:~$ ping -d time.google.com
Initiating ping to: time.google.com
RRC mode: Connected

after that step I was able to ping from PC with success

➜  ~ ping -I ppp0 time.google.com
PING  (216.239.35.4) from x.y.z.zz ppp0: 56(84) bytes of data.
64 bytes from time2.google.com (216.239.35.4): icmp_seq=1 ttl=253 time=257 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=2 ttl=253 time=871 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=3 ttl=253 time=183 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=4 ttl=253 time=302 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=5 ttl=253 time=139 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=6 ttl=253 time=1121 ms
64 bytes from time2.google.com (216.239.35.4): icmp_seq=7 ttl=253 time=158 ms
^C
---  ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6025ms

Are modem shell ping cmd and ppp correlated somehow or am I doing something wrong?

Thanks,

Adam

[Håkon Alseth]

Hi Adam,

 

adam_h said:

I executed from modem shell ping time.google.com 

While the ppp connection is up, pinging from the shell isn't supported.

That being said, it's strange that this made the PC side ping work.

 

Is this behavior consistent on your end? If you reset the board, re-enable the ppp connection on the PC side, and do a "ping -I ppp0 time.google.com", does it still reply with destination unreachable?

 

Kind regards,

Håkon

[adam_h]

Hi,

sorry for late response. I've made tests.
1. Ping from shell is not working for me.

mosh:~$ ping -d time.google.com
Initiating ping to: time.google.com
RRC mode: Connected
getaddrinfo(dest) error: -11
Cannot resolve remote host

mosh:~$ ping -d 216.239.35.0
Initiating ping to: 216.239.35.0
Source IP addr: x.y.z.zz
Destination IP addr: 216.239.35.0
send() failed: (-22)
send() failed: (-22)
send() failed: (-22)
send() failed: (-22)
Ping statistics for 216.239.35.0:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)
Pinging DONE

I made resets and sent cmd again without succes.

2. Reset of the board doesn't help for ping from PC. In both cases to get ping from PC working I need to execute ping cmd from mosh shell

When I issue cmd ping -I ppp0 216.239.35.0 it hangs after printing
PING 216.239.35.0 (216.239.35.0) from 100.67.125.106 ppp0: 56(84) bytes of data.

till I issue ping on mosh shell. After executing this cmd it (PC shell) starts printing

64 bytes from 216.239.35.0: icmp_seq=17 ttl=253 time=169 ms
64 bytes from 216.239.35.0: icmp_seq=18 ttl=253 time=167 ms
64 bytes from 216.239.35.0: icmp_seq=19 ttl=253 time=227 ms

Kind regards,

Adam

[Håkon Alseth]

Hi Adam,

 

Thank you for the detailed log and description.

I tried to recreate this on my end, but I am unfortunately not able to do so. I will still report this back to the modem_shell team, so that they are aware of this.

I have been talking to the team, and they mentioned that some instabilities have been observed while using the usb-uart bridge on the nRF DK, so they recommended that one uses a dedicated usb/uart bridge for this. However, the behavior that you're seeing seems to be related towards network rather than communication issues, so not likely going to help your specific scenario.

 

Kind regards,

Håkon