nRF9160 DK stopped responding to AT commands

[stevenminnick]

I have 2 nRF9160 DK's.  I've been using 1 for most of my testing and the second one was still in the box. 

I loaded the secure boot and the AT_CLIENT demo project.  Opened up the nRF Link Monitor tool.  Hit the reset button and Zephyr stuff flies on the screen.  Ends up with:

Secure Boot: prepare to jump to Non-Secure image

***** Booting Zephyr OS v1.13.99-ncs2 *****

The AT host sample started.  err=0

(I added the err=%d) in the printf statement to prove that I was doing stuff. 

The problem is when I send it an AT command, I get: Error: 'AT ' timed out

I reloaded the modem firmware with: nrf9160_mdm_dfu --update 

That worked.  But still did not respond to AT commands, just time out.

Took my other dev kit out of the box.  Moved my sim card to DevKit #2.  Loaded the same code on it.  With this: nrfjprog.exe --program zephyr.hex --sectorerase -r -f nrf91 --verify

Reboot DevKit #2, and get the same Zephyr messages and everything works.  AT commands work.  Connects to the network. 

Hook back up to DevKit #1, AT commands time out. 

Now I did solder on a 1 ohm resistor on R60 to measure current.  And if I program the asset tracker software to it, it connects and works perfectly.  But why does the AT_CLIENT time out now? 

Anyone have any thoughts?  Thanks!

Steve

[stevenminnick]

Basically I have 2 dev kits.  The asset tracker application works on both of them.  But the at_client software only works on 1 of them. 

The 1 where the at_client doesn't work I was trying to use the IDE to debug, trying to step through.  But I ended up just using the programmer and erasing the whole memory and cloning the second dev kit and the at_client still doesn't work. AT commands just time out.  But it works perfectly on the second dev kit. 

[Bjørn Kvaale]

Have you tried to reset the board multiple times by pressing the RESET button. Erase the flash via nrfjprog --eraseall -f nrf91, then program the secure_boot example via Segger Embedded Studio & finally program the at_client example. Then, open up the Link Monitor application from nRF Connect for Desktop & connect to the board. Then, press the reset button. If the AT commands time out, try to press the reset button again. For some reason, this sometimes works!

Also, once you see the text "The AT host sample started", wait a few seconds & then try pressing the AT+CFUN? button. That should then make the at client example work.

[stevenminnick]

Totally missed that.  Thanks!  I think loading the SecureBoot/SecurePartitionMananger might help Pratik to get his at_client to run and respond.

[buono]

I'm having the exact same issue. I'm trying to follow your suggestions, but I'm a bit confused. Are you saying you need to load 2 hex files? The SPM first and then the AT_client? Is there a special way of doing this?

[stevenminnick]

I believe that, yes you have to load 2 different hex files. 

nrfjprog.exe --program at_client.hex --sectorerase -r -f nrf91 --verify

nrfjprog.exe --program Secure_Partition_Manager.hex --sectorerase -r -f nrf91 --verify

You can also use the programmer too in the nRF Connect GUI that lets you load multiple hex files.  You can have your bootloader loaded and then just run out of the IDE as well.

That all being said, I'm still having trouble with the new 0.4.0 version of the AT client, was going to make a new thread about that. But this is what I did with the 0.3.0 version. 

[buono]

Thanks for the reply! I tried using the nRF Connect GUI first, but it said the HEX files have overlapping data. I tired using the command line with the command you gave above, but  still dont think its working.  When I reset the DK the output in the terminal window is 

***** Booting Zephyr OS v1.14.99-ncs1 *****

The AT host sample started

Shouldn't there be a bunch of other lines related to the SPM first before the AT host starts? (Still getting the AT timeout error if I send one after the output above)

[stevenminnick]

yes there should be.  And for some reason the 0.4.0 version of the AT Client writes itself to address 0 instead of 40000 (or whatever, I'm pulling that from memory).  If you take the secure boot I posted above, or take the SPM from 0.4.0 and load that.  And then take AT Client from 0.3.0 (which I can post here), it writes itself to address 40000.  so if you flash both it will give you the bunch of stuff from the SPM and then the AT client and that works.  If you load the old radio firmware 0.6.8 on there, the whole thing works (at least for me). 

But if you load the new 0.4.0 AT Client it overwrites the SPM and all you get is the AT Client, which I don't think works anyway.  At least I haven't figured out how it works. 

at_client_snarkyPrintf.hex

WARNING, I get a bit strange when I've been working on things like this and to make sure I was really flashing the code I was compiling I added a snarky/rude line.  I don't feel this way, or am angry, but if you see this line you know 100% for sure it's not Nordic's stock code.  I was thinking, what could I write that they would never put in code.  And so it exists.

[buono]

LOL, ok thanks! I'll try this out tomorrow. 

[Ruth Fuchss]

For 0.4.0, the nRF9160 samples include the SPM in a multi-image build. So instead of programming the two HEX files separately, you should program the merged.hex file - follow the instructions for a multi-image project here.

[stevenminnick]

I believe the solution is the "ns".  If you open a project with the board directory as nrf9160_pca10090 but the board name as nrf9160_pca10090ns (and the build directory but that doesn't matter).    The trick is the NS.

Without the NS:

To build and flash a multi-image project:

1. Select your project in the Project Explorer.
2. From the menu, select Build -> Build Solution.
3. When the multi-image build completes, you can flash the sample to a connected board. To do this, select Target -> Download File -> Download Intel Hex File. Navigate to the zephyr folder in your build directory and choose merged.hex.

I don't get a merged.hex file.  I get a zephyr.hex.  And when I flash that, it's not the SPM and AT Client.  Just the AT Client.  And it doesn't respond to AT commands. 

With the NS you get a merged.hex file.

If I try to follow the directions exactly and compile the asset tracker:

If I build for nrf9160_pca10090

4> In file included from C:/git/nRF/ncs/zephyr/include/toolchain.h:21:0,
4>                  from zephyr/misc/generated/configs.c:7:
4> C:/git/nRF/ncs/zephyr/include/toolchain/gcc.h:325:51: error: expected expression before ')' token
4> zephyr/misc/generated/configs.c:22:1: note: in expansion of macro 'GEN_ABSOLUTE_SYM'
4> C:/git/nRF/ncs/zephyr/include/toolchain/gcc.h:325:51: error: expected expression before ')' token
4> zephyr/misc/generated/configs.c:29:1: note: in expansion of macro 'GEN_ABSOLUTE_SYM'
Build failed

But if I build for nrf9160_pca10090ns

It's fine.  And I get a merged.hex file

What's the NS stand for?  I was assuming Not Secure.  But I also assumed that not secure was if you didn't want the SPM or secure boot.  Maybe it's Nowloading Securepartitionmanager. 

[buono]

Wow, that worked! Thanks so much for your help. Strange that the new AT client overlaps with the SPM app if they are both required in order to work. 

[stevenminnick]

Actually I figured out a few more things today.  And I'm sure someone smarter than I will come in and correct my description/terminology. 

The new AT Client and 0.4.0 stuff is meant to combine both the SPM and the application together in 1 hex download.  So when you compile it "wrong", you get the application set to install at address 0. 

But if you go get the latest radio code, 0.7.0, flash that. 

Then go get the latest GIT repos.  And get the latest Segar stuff and use all of that it all works.  They seemed to have changed the AT client from running in main to responding to interrupts or running in it's own thread.  So if you don't run the SPM first it's not setting up the processor right and all you get is the new AT clinet printing "host started".

I documented this info here as well: https://devzone.nordicsemi.com/f/nordic-q-a/44816/nrf9160dk-can-not-find-the-cellular-network

But if you check out the 0.4.0 versions.  Personally I have:
fw-nrfconnect-nrf: v0.4.0
fw-nrfconnect-zephyr:  v1.14.99-ncs1
fw-nrfconnect-mcuboot: v1.3.99-ncs1
nrfxlib: v0.4.0 

So far I have tested asset tracker and the at client, but so far so good. 

Things that caused me issues:

1. make sure you go in and delete the CMakeCache.txt files.  I know it's an option in the "open project" but do it before you open the project, sometimes it made a difference. 

2. Make sure you open the project and pick the board directory as nrf9160_pca10090 and the board name to be nrf9160_pca10090ns.  Notice the NS vs non.  See: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/0.4.0/nrf/gs_programming.html 

3. Make sure you update the radio firmware.  0.7.0 works with this.  0.6.8 seems to only work with the older git tagged repos that ebran posted. 

4. When flashing your board flash the merged.hex file. Just this one file, it will write both the SPM and the application together. 

You can update to the latest of everything and it appears to be working.

Your directories might be different than mine, but if you work out of a c:\git\nRF directory.  You can:

cd C:\git\nRF\mfwnrf916007015alpha
nrf9160_mdm_dfu --update

cd C:\git\nRF\ncs\zephyr
git reset --hard
git checkout
cd C:\git\nRF\ncs\mcuboot
git reset --hard
git checkout
cd C:\git\nRF\ncs\nrf
git reset --hard
git checkout
cd C:\git\nRF\ncs\nrfxlib
git reset --hard
git checkout

pip3 install -r C:\git\nRF\ncs\zephyr\scripts\requirements.txt
pip3 install -r C:\git\nRF\ncs\nrf\scripts\requirements.txt
pip3 install -r C:\git\nRF\ncs\mcuboot\scripts\requirements.txt