Unable to connect to AT&T after upgrading MFW to v1.1.0

[jbrzozoski]

We had been using an AT&T M1-enabled SIM in one of our development units successfully for quite some time.  After we upgraded that node to MFW v1.1.0, it is no longer able to connect to the cell network.  We tried swapping SIMs around between devices with different firmware versions confirmed that any node still running MFW v1.0.1 is able to connect to the AT&T network using this SIM, and any node running MFW v1.1.0 is not.  I put the SIM in one of our v1.1.0 nodes, loaded up at_client, and turned on as much output as I could.  The relevant error when trying to connect to the network is this:

+CNEC_ESM: 33,0
+CNEC_EMM: 19
+CEREG: 2,"FFFE","FFFFFFFF",7,0,19,"11100000","11100000"
+CEREG: 2,"0411","00F63F11",7,0,19,"11100000","11100000"

It gets the CEREG failure almost immediately and then just sits there doing nothing.  (We waited 15+ minutes)

Again, this failure is using an official AT&T SIM that returns 2 to the AT%XOPERID command.

We are able to connect to the same AT&T cells using v1.1.0 if we put in an MVNO SIM (hologram).

What is the modem doing differently with and without the official AT&T SIM?  Whatever it is doing, it's breaking something in our testing.

[Hakon]

Hello,

Please perform modem trace and post the trace file here. https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/how-to-get-modem-trace-using-trace-collector-in-nrf-connect

[chester314]

Hello.  I have additional data again. 

I found that I can connect with modem 1.1.0 if I use a hologram SIM but not with an ATT SIM.  It may be useful to note that I've seen the link monitor say the hologram sim roamed successfully on both verizon and ATT.

Below are modem traces and console output of modem 1.0.1 working with the ATT SIM, 1.1.0 not working with the ATT SIM and 1.1.0 working with the hologram SIM.

modem101_att_sim_connects_in_10_secs.bin

modem110_att_sim_no_connect.bin

modem110_hologram_sim_connects_in_15secs.bin

Fullscreen modem110_att_sim_no_connect_console_log.txt Download

SPM: NS image at 0xc000
SPM: NS MSP at 0x2002beb8
SPM: NS reset vector at 0x13155
SPM: prepare to jump to Non-Secure image.
[00:00:00.267,974]
[1;31m<err> i2c_nrfx_twim: Error 195952642 occurred for message 0
[0m
***** Booting Zephyr OS build v2.0.99-ncs1 *****
*****************
CX1751 Demo App Started
*****************
BME280 bound successfully
APDS9960 bound successfully
ACC DT LBL:=MMA8451QFXOS8700 bound successfully
ACC DT LBL:=VL53L0XVL53L0X bound successfully
GPS initialized
Setting up the Battery Monitor succeeded.
[00:00:00.650,268]
[0m<dbg> lte_lc.w_lte_lc_connect: Network mode: AT%XSYSTEMMODE=1,0,1,0
[0m
%CESQ: 38,1,17,2
+CEREG: 2,"3319","0ABA3210",7,0,0,"11100000","11100000"
[00:00:02.294,189]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
+CEREG: 2,"3319","0ABA3210",7,0,19,"11100000","11100000"
[00:00:03.624,267]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 255,0,255,0
+CEREG: 2,"FFFE","FFFFFFFF",7,0,19,"11100000","11100000"
[00:00:04.897,949]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 25,1,0,0
+CEREG: 2,"3319","0ABB1011",7,0,19,"11100000","11100000"
[00:00:06.126,983]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 12,0,20,2
%CESQ: 26,1,3,0
%CESQ: 10,0,22,3
AT+CFUN?
+CFUN: 1
OK
AT+CGSN=1
+CGSN: "352656100816191"
OK
AT+CGMI
Nordic Semiconductor ASA
OK
AT+CGMM
nRF9160-SICA
OK
AT+CGMR
mfw_nrf9160_1.1.0
OK
AT+CEMODE?
+CEMODE: 2
OK
AT%XCBAND=?
%XCBAND: (1,2,3,4,5,8,12,13,14,17,18,19,20,25,26,28,66)
OK
AT+CMEE?
+CMEE: 0
OK
AT+CMEE=1
OK
AT+CNEC?
+CNEC: 0
OK
AT+CNEC=24
%CESQ: 24,1,4,0
OK
AT+CGEREP?
+CGEREP: 0,0
OK
AT+CGDCONT?
+CGDCONT: 0,"IP","","",0,0
OK
AT+CGACT?
+CGACT: 0,0
OK
AT+CGEREP=1
OK
AT+CIND=1,1,1
OK
AT+CEREG=2
OK
AT+CEREG?
+CEREG: 2,2,"3319","0ABB1011",7
OK
AT%CESQ=1
OK
AT+CESQ
+CESQ: 99,99,255,255,4,24
OK
AT%XSIM=1
OK
AT%XSIM?
%XSIM: 1
OK
AT+CPIN?
+CPIN: READY
OK
AT+CPINR="SIM PIN"
+CPINR: "SIM PIN",3
OK
AT+CIMI
310170833560137
OK
%CESQ: 29,1,4,0
%CESQ: 12,0,20,2
%CESQ: 25,1,3,0
%CESQ: 12,0,20,2
%CESQ: 25,1,3,0
%CESQ: 10,0,22,3
%CESQ: 23,1,4,0

Fullscreen modem_110_hologram_sim_connects_console_log.txt Download

SPM: NS MSP at 0x2002beb8
SPM: NS reset vector at 0x13155
SPM: prepare to jump to Non-Secure image.
[00:00:01.638,458]
[1;31m<err> i2c_nrfx_twim: Error 195952642 occurred for message 0
[0m
***** Booting Zephyr OS build v2.0.99-ncs1 *****
*****************
CX1751 Demo App Started
*****************
BME280 bound successfully
APDS9960 bound successfully
ACC DT LBL:=MMA8451QFXOS8700 bound successfully
ACC DT LBL:=VL53L0XVL53L0X bound successfully
GPS initialized
Setting up the Battery Monitor succeeded.
[00:00:02.020,690]
[0m<dbg> lte_lc.w_lte_lc_connect: Network mode: AT%XSYSTEMMODE=1,0,1,0
[0m
%CESQ: 38,1,14,2
+CEREG: 2,"3319","0ABA3210",7,0,0,"11100000","11100000"
[00:01:33.780,731]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
+CEREG: 2,"3319","0ABA3210",7,0,11,"11100000","11100000"
[00:01:33.886,535]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 38,1,9,1
%CESQ: 255,0,255,0
+CEREG: 2,"FFFE","FFFFFFFF",7,0,11,"11100000","11100000"
[00:01:35.160,308]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 43,2,15,2
+CEREG: 2,"9103","0237AC02",7,0,11,"11100000","11100000"
[00:01:36.318,298]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 255,0,255,0
+CEREG: 2,"FFFE","FFFFFFFF",7,0,11,"11100000","11100000"
[00:01:38.919,555]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 38,1,15,2
+CEREG: 2,"3319","0ABA3210",7,0,11,"11100000","11100000"
[00:01:39.518,463]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
%CESQ: 38,1,10,1
%CESQ: 33,1,10,1
+CEREG: 5,"3319","0ABA3210",7,,,"11100000","11100000"
[00:01:42.517,456]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 5
[0m
>>APP EVT: LTE Link Connection succeeded!
AT+COPS=3,2
OK
AT+COPS?
+COPS: 0,2,"310410",7
OK
AT%XCBAND
%XCBAND: 17
OK
AT+CGDCONT?
%CESQ: 33,1,5,0
+CGDCONT: 0,"IP","hologram","10.184.142.60",0,0
OK
AT+CGACT?
+CGACT: 0,1
OK
IPv4 Address found 3.220.217.199

AT+CFUN?
+CFUN: 1
OK
AT+CGSN=1
+CGSN: "352656100816191"
OK
AT+CGMI
Nordic Semiconductor ASA
OK
AT+CGMM
nRF9160-SICA
OK
AT+CGMR
mfw_nrf9160_1.1.0
OK
AT+CEMODE?
+CEMODE: 2
OK
AT%XCBAND=?
%XCBAND: (1,2,3,4,5,8,12,13,14,17,18,19,20,25,26,28,66)
OK
AT+CMEE?
+CMEE: 1
OK
AT+CNEC?
+CNEC: 24
OK
AT+CGEREP?
+CGEREP: 1,0
OK
AT+CIND=1,1,1
OK
AT+CEREG=2
OK
AT+CEREG?
+CEREG: 2,5,"3319","0ABA3210",7
OK
AT+COPS=3,2
OK
AT+COPS?
+COPS: 0,2,"310410",7
OK
AT%XCBAND
%XCBAND: 17
OK
AT+CGDCONT?
+CGDCONT: 0,"IP","hologram","10.184.142.60",0,0
OK
AT+CGACT?
+CGACT: 0,1
OK
AT%CESQ=1
OK
AT+CESQ
+CESQ: 99,99,255,255,15,33

Fullscreen modem101_att_sim_console_log.txt Download

SPM: NS MSP at 0x2002beb8
SPM: NS reset vector at 0x13155
SPM: prepare to jump to Non-Secure image.
[00:00:00.263,793]
[1;31m<err> i2c_nrfx_twim: Error 195952642 occurred for message 0
[0m
***** Booting Zephyr OS build v2.0.99-ncs1 *****
*****************
CX1751 Demo App Started
*****************
BME280 bound successfully
APDS9960 bound successfully
ACC DT LBL:=MMA8451QFXOS8700 bound successfully
ACC DT LBL:=VL53L0XVL53L0X bound successfully
GPS initialized
Setting up the Battery Monitor succeeded.
[00:00:00.618,408]
[0m<dbg> lte_lc.w_lte_lc_connect: Network mode: AT%XSYSTEMMODE=1,0,1,0
[0m
%CESQ: 26,1
+CEREG: 2,"3319","0ABB1011",7,0,0,"11100000","11100000"
[00:00:02.532,379]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 2
[0m
+CEREG: 1,"3319","0ABB1011",7,,,"11100000","11100000"
[00:00:04.841,522]
[0m<dbg> lte_lc.parse_nw_reg_status: Network registration status: 1
[0m
>>APP EVT: LTE Link Connection succeeded!
IPv4 Address found 3.220.217.199
client_username: iH3iQ1aPOLlUizn1Zb6C, size: 21
>>APP EVT: MQTT Broker Connection Succeeded!
+CEREG: 1,"3319","0ABA3210",7,,,"11100000","11100000"

[Hakon]

Thanks, I will relay this information to the developers, and report back when they answer.

[Hakon]

It will be fixed in the 1.1.1 release. In the mean time you need to use a different SIM.

[chester314]

Thank you.  Do you know approximately when 1.1.1 will be released?  (days?  weeks?)

[Hakon]

I can't give you a specific date for this release.

Edit; New information is available. Ignore the above, 1.1.1 will be released in January.

Also, if you have used nRF Connect version 3.3 and Programmer 1.3.1, then go back to release v1.0.1.

[Hakon]

Hello again,

This should now be fixed in the 1.1.1 release. Has it worked out? Do you have any feedback?

[jbrzozoski]

Seems to work fine now.  No issues in our first few days of running with 1.1.1. 

[chester314]

Sorry for the delayed reply but I'm please to report that 1.1.1 fixes our AT&T issue as well.  Thank you for the support.