Problems with "Enable debug options"

[Lars M]

Hi,

I am facing a problem with the selection "Enable debug options" when creating a build configuration. I use nRF connect SDK 1.9.1 with VSCode. I create an application from sample aws_iot. Then I make two different build configurations, one with "Enable debug options" checked, and one with the option unchecked. The one that has debug option enabled is not possible to debug (it may work once, but all other debug sessions do not work). But the one that doesn't have the option enabled works fine to debug.

How come there is a difference?

What does "Enable debug options" provide?

Best regards,
Lars

[dejans]

Hi,

I have replicated the scenario you described myself using aws_iot sample with VS Code and NCS v1.9.1. I did not observe any issue, no matter if "Enable debug options" was checked or unchecked, .

Best regards,
Dejan

[Lars M]

Hi Dejan,

Strange. I have staerted over, and the first time I debug the build configuration that has "Enable debug options" checked, it works fine. But all other debug sessions don't. I have attached three files with the printouts from "Debug console". One file is how it looks in the build configuration that doesn't have "Enable debug options" checked. The other two has the option checked. The one where debugging is working is the first debug session, the other one is how it looks for all other debug sessions. Is it possible to find any interesting information from this?

Has any other of your team tested to see if you can repeat this?

What does "Enable debug options" provide?

Best regards,
Lars

Fullscreen build debug options not enabled, debugging ok.txt Download

Reading symbols from c:/projekt\nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-objdump.exe --syms -C -h -w c:\Projekt\nordic\aws_iot\build\zephyr\merged.hex
Reading symbols from c:/projekt/nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-nm.exe --defined-only -S -l -C -p c:\Projekt\nordic\aws_iot\build\zephyr\merged.hex
Launching GDB: "c:\\Projekt\\NordicSemi\\v1.9.1\\toolchain\\opt\\bin\\arm-none-eabi-gdb.exe" -q --interpreter=mi2 "c:\\Projekt\\nordic\\aws_iot\\build\\zephyr\\merged.hex"
    Set "showDevDebugOutput": true in your "launch.json" to see verbose GDB transactions here. Helpful to debug issues or report problems
Launching gdb-server: "C:\\Program Files (x86)\\SEGGER\\JLink\\JLinkGDBServerCL.exe" -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device nRF9160_xxAA -select usb=960008799 -rtos "C:\\Program Files (x86)\\SEGGER\\JLink\\GDBServer\\RTOSPlugin_Zephyr.dll"
    Please check TERMINAL tab (gdb-server) for output from C:\Program Files (x86)\SEGGER\JLink\JLinkGDBServerCL.exe
Finished reading symbols from objdump: Time: 96 ms
Finished reading symbols from nm: Time: 95 ms
Reading symbols from c:\Projekt\nordic\aws_iot\build\zephyr\merged.hex...
(No debugging symbols found in c:\Projekt\nordic\aws_iot\build\zephyr\merged.hex)
WARNING: Cortex-Debug will deprecate use of GDB version 8 after July 2022. Please upgrade to version 9+
0x00025b22 in ?? ()
Program stopped, probably due to a reset and/or halt issued by debugger
add symbol table from file "c:/Projekt/nordic/aws_iot/build/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build/mcuboot/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build/mcuboot/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build/spm/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build/spm/zephyr/zephyr.elf...
2
Resetting target

Temporary breakpoint 2, main () at C:/Projekt/NordicSemi/v1.9.1/modules/hal/nordic/nrfx/hal/nrf_wdt.h:427
427	    p_reg->RR[rr_register] = NRF_WDT_RR_VALUE;

Fullscreen build_1_debug_options_enabled, debugging ok.txt Download

Reading symbols from c:/projekt\nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-objdump.exe --syms -C -h -w c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex
Reading symbols from c:/projekt/nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-nm.exe --defined-only -S -l -C -p c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex
Launching GDB: "c:\\Projekt\\NordicSemi\\v1.9.1\\toolchain\\opt\\bin\\arm-none-eabi-gdb.exe" -q --interpreter=mi2 "c:\\Projekt\\nordic\\aws_iot\\build_1_enable_debug_options\\zephyr\\merged.hex"
    Set "showDevDebugOutput": true in your "launch.json" to see verbose GDB transactions here. Helpful to debug issues or report problems
Launching gdb-server: "C:\\Program Files (x86)\\SEGGER\\JLink\\JLinkGDBServerCL.exe" -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device nRF9160_xxAA -select usb=960008799 -rtos "C:\\Program Files (x86)\\SEGGER\\JLink\\GDBServer\\RTOSPlugin_Zephyr.dll"
    Please check TERMINAL tab (gdb-server) for output from C:\Program Files (x86)\SEGGER\JLink\JLinkGDBServerCL.exe
Finished reading symbols from objdump: Time: 89 ms
Finished reading symbols from nm: Time: 92 ms
Reading symbols from c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex...
(No debugging symbols found in c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex)
WARNING: Cortex-Debug will deprecate use of GDB version 8 after July 2022. Please upgrade to version 9+
0x00000438 in ?? ()
Program stopped, probably due to a reset and/or halt issued by debugger
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/mcuboot/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/mcuboot/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/spm/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/spm/zephyr/zephyr.elf...
2
Resetting target
[New Remote target]
[Switching to Remote target]

Thread 2 hit Temporary breakpoint 2, main () at C:/Projekt/NordicSemi/v1.9.1/modules/hal/nordic/nrfx/hal/nrf_wdt.h:427
427	    p_reg->RR[rr_register] = NRF_WDT_RR_VALUE;

Fullscreen build_1_debug_options_enabled, debugging fail.txt Download

Reading symbols from c:/projekt\nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-objdump.exe --syms -C -h -w c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex
Reading symbols from c:/projekt/nordicsemi\v1.9.1\toolchain\opt\bin\arm-none-eabi-nm.exe --defined-only -S -l -C -p c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex
Launching GDB: "c:\\Projekt\\NordicSemi\\v1.9.1\\toolchain\\opt\\bin\\arm-none-eabi-gdb.exe" -q --interpreter=mi2 "c:\\Projekt\\nordic\\aws_iot\\build_1_enable_debug_options\\zephyr\\merged.hex"
    Set "showDevDebugOutput": true in your "launch.json" to see verbose GDB transactions here. Helpful to debug issues or report problems
Launching gdb-server: "C:\\Program Files (x86)\\SEGGER\\JLink\\JLinkGDBServerCL.exe" -singlerun -nogui -if swd -port 50000 -swoport 50001 -telnetport 50002 -device nRF9160_xxAA -select usb=960008799 -rtos "C:\\Program Files (x86)\\SEGGER\\JLink\\GDBServer\\RTOSPlugin_Zephyr.dll"
    Please check TERMINAL tab (gdb-server) for output from C:\Program Files (x86)\SEGGER\JLink\JLinkGDBServerCL.exe
Finished reading symbols from objdump: Time: 81 ms
Finished reading symbols from nm: Time: 85 ms
Reading symbols from c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex...
(No debugging symbols found in c:\Projekt\nordic\aws_iot\build_1_enable_debug_options\zephyr\merged.hex)
WARNING: Cortex-Debug will deprecate use of GDB version 8 after July 2022. Please upgrade to version 9+
0x00025b22 in ?? ()
Program stopped, probably due to a reset and/or halt issued by debugger
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/mcuboot/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/mcuboot/zephyr/zephyr.elf...
add symbol table from file "c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/spm/zephyr/zephyr.elf"
(y or n) [answered Y; input not from terminal]
Reading symbols from c:/Projekt/nordic/aws_iot/build_1_enable_debug_options/spm/zephyr/zephyr.elf...
2
Resetting target
[New Thread 536930240]
[New Thread 536923376]
[New Thread 536925696]
[Switching to Thread 536930240]
Run to 'main' timed out. Trying to pause program

[Lars M]

Hi Dejan,

Great, thanks! This solves the problem to a great extent! Now it is possible to debug, but it only works every other time. It seems that I'm back to the behaviour I had in the beginning of this ticket. Which feels quite natural.

Every other time that I debug, the board is disconnected, and then reconnected. Then debugging doesn't work.

The other times debugging works. But this is only true if I have not checked the "Enable debug options"-checkbox when making a build configuration.

Is it you at Nordic that are making this extension? Is there a forecast when this can be fixed?

Best regards,
Lars

[dejans]

Hi Lars,

Lars M said:

Is it you at Nordic that are making this extension? Is there a forecast when this can be fixed?

We are not making the Cortex-Debug extension.

Have you tried changing the board, rebooting the computer and/or replacing the cables?

How does Ozone debugger behave now with aws_iot sample? 

Best regards,
Dejan

[Lars M]

Hi Dejan,

I have another DK and another cable compared to what I had in the beginning of this ticket. And my computer has been restarted several times. The version I now use of Cortex-Debug is 1.4.4. I also tested 1.4.0, but didn't see any difference.

Debugging with Ozone works fine for hello_world if I don't check "Enable debug options". If I check it, debugging with Ozone doesn't work. For aws_iot, debugging with Ozone doesn't work at all. It feels like I'm also here is back where I was before.

Are you able to repeat what I see? Both when it comes to this disconnection-thing with VSCode, the different behaviour when checking/not checking the "Enable debug options"-box, and what I see when trying to debug with Ozone?

Best regards,
Lars

[dejans]

Hi Lars,

Do you have any other computer to perform debug testing on?

Have you tried any other application excluding hello_world and aws_iot?

Best regards,
Dejan

[Lars M]

Hi Dejan,

No, I haven't had time do to that yet.

Lars M said:

Are you able to repeat what I see? Both when it comes to this disconnection-thing with VSCode, the different behaviour when checking/not checking the "Enable debug options"-box, and what I see when trying to debug with Ozone?

Best regards,

Lars

[dejans]

Hi Lars,

I have tried to reproduce the behavior that you reported with partial success. I used aws_iot sample from NCS v2.0.0, Toolchain v2.0.0 and Cortex-Debug v1.4.4.
I did not observe connection/disconnection in VS Code as you described it. Ozone and Cortex-Debug did not work for me. nRF-Debug worked intermittently.
I would suggest you to try switching to nRF-Debug by going to nRF Connect for VS Code extension -> Extension Settings -> nRF Connect Debugging:Backend -> nRF Debug. What is the result of the Debug change?

Best regards,
Dejan

[Lars M]

Hi Dejan,

Now I have tested with nRF Debug, but it feels very unstable. I have just installed SDK 2.1.0, and initial tests show that debugging now works every time for aws_iot. And I run Cortex debug 1.6.3. If I go back to SDK 1.9.1 it seems I have to change to Cortex debug 1.4.4, and then debugging works every other time.

Best regards,
Lars

[dejans]

Hi Lars,

nRF Debug is our own debugger and it is recommended to be used going forward. 

Best regards,
Dejan

[Lars M]

Hi Dejan,

As I said I experience nRF Debug to be quite unstable. Debugging with "Enable debug options" checked never works. Sometimes the orange bar at the bottom appears, and the small control window with debug buttons appears, but I can't do anything. Pause doesn't work, and the program doesn't stop at breakpoints. Though, at start up this is printed to the console:

*** Booting Zephyr OS build v3.1.99-ncs1  ***
I: Starting bootloader
I: Primary image: magic=bad, swap_type=0x1, copy_done=0x2, image_ok=0x2
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
I: Bootloader chainload address offset: 0x10000
*** Booting Zephyr OS build v3.1.99-ncs1  ***

When I click the restart-button, this is printed:

*** Booting Zephyr OS build v3.1.99-ncs1  ***

Sometimes I get these errors no matter if "Enable debug options" is checked or unchecked:

ERROR: The JLinkARM DLL timed out while communicating to the J-Link probe.
ERROR: If the error condition persists, run the same command again with
ERROR: argument -- log, contact Nordic Semiconductor and provide the generated
ERROR: log.log file to them.
NOTE: For additional output, try running again with logging enabled (--log).
NOTE: Any generated log error messages will be displayed.
FATAL ERROR: command exited with status 35: nrfjprog --program 'C:\Projekt\nordic\aws_iot_210\build_debug_options_not_enabled\zephyr\merged.hex' --sectorerase --verify -f NRF91 --snr 960008799

Debugging with "Enable debug options" unchecked works sometimes. Then I get these printouts:

*** Booting Zephyr OS build v3.1.99-ncs1  ***
I: Starting bootloader
I: Primary image: magic=bad, swap_type=0x1, copy_done=0x2, image_ok=0x2
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
I: Bootloader chainload address offset: 0x10000
*** Booting Zephyr OS build v3.1.99-ncs1  ***
The AWS IoT sample started, version: v1.0.0

So here the last row is added to the printouts, which is the first row in main(). When I click the restart-button, all these rows are printed again, not just one row as when "Enable debug options" is checked and the restart-button is checked.

When I try to set breakpoints, they almost never appear where I want. They are often propagated to underlying functions, so it's really hard to get a good grasp of where they are.

Always starting a debug session takes quite a long time. So after these tests it feels as nRF Debug doesn't do what it should.

Best regards,
Lars

[dejans]

Hi Lars,

Could you please summarize current behavior of nRF Debug with and without "Enable Debug Options" using sample from NCS v2.1.0? 

Best regards,
Dejan