Not possible to restart debugging without reloading in VSCode (nrf9160dk)

[jpnorair]

I'm trying desperately to find a way to restart a debugging session (reset device and go back to main) on the nrf9160dk without reloading the FW onto the device.

I'm using VSCode and the nordic 2.3.0 sdk & toolchain.  

I'm desperate to find way(s) to improve the debugging features, which aren't at the same level as those found in more customary embedded IDEs, and it's affecting our ability to use the tools at all.  If there is anything I can do, I will consider it.  If I can fix problems by using an external JLink (for example) or using non-nordic extension, I will try anything.

[Håkon Alseth]

Hi,

 

I share your concerns.

This is an improvement task that I've raised internally.

 

As a workaround, you can issue gdb commands in the debug console by first appending "-exec", for instance monitor reset:

Could you try this and see if that works for you?

 

Kind regards,

Håkon

[jpnorair]

This workaround helps, although I do seriously hope your improvements also get implemented.  I will describe my experience with the workaround.

This workflow does NOT work.  It results in an error (copied below).

1. Set a breakpoint, and wait for it to hit.
2. apply -exec mon reset
3. Push resume button

Thread 15 hit Breakpoint 2 ...
-exec mon reset
Resetting target
[New Remote target]
/Volumes/Workspace/build/.build/HOST-aarch64-apple-darwin/arm-zephyr-eabi/src/gdb/gdb/infrun.c:5825: internal-error: finish_step_over: Assertion `ecs->event_thread->control.trap_expected' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) [answered Y; input not from terminal]
/Volumes/Workspace/build/.build/HOST-aarch64-apple-darwin/arm-zephyr-eabi/src/gdb/gdb/infrun.c:5825: internal-error: finish_step_over: Assertion `ecs->event_thread->control.trap_expected' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) [answered Y; input not from terminal]
ERROR: GDB exited unexpectedly with exit code 134 (0x86). Debugging will now abort.
The program '/Users/jpnorair/.../build/zephyr/zephyr.elf' has exited with code -1 (0xffffffff).

This other workflow does provide a workaround, however:

1. Set a user breakpoint at main()
2. Set another breakpoint where you want it, wait for it to hit.
3. Step the debugger
4. apply -exec mon reset
5. Push resume button
6. Debugger WILL arrive at the breakpoint on main()

With this workaround, I am at least able to debug the software we need to work-on.  Thank you: your suggestion was very helpful.

I will add an additional point -- I wrote another ticket recently about issues programming the target (https://devzone.nordicsemi.com/f/nordic-q-a/100497/vs-code-debug-flash-works-once-then-fails-on-second-instantiation).  By applying [-exec mon reset] when I am finished with the debugging session, I am able to flash the board again without problems (at least, so far).

[jpnorair]

This reset button is the first thing I tried.

It indeed performs a reloading of the fw onto the target, which is not only slow but also full of problems (you can see the other ticket for this).  The only way I've found to successfully restart the device (or reset it at all) is via the [-exec mon reset] command you presented.

Having a fast reset is very important if trying to breakpoint-debug in the presence of a second core which doesn't stop when the first core is breakpointed.  Most nordic products seem to have a second core for the comms stack.

[Håkon Alseth]

Thank you for elaborating on your process.

The intended process is that the device shall reset (ie. exit and re-enter debug mode) without having to flash again.

I have seen this behavior before, but I am not able to successfully recreate this at my end now, unfortunately. I've tested with a macbook m1 (big sur) and two different linux PCs (Intel and AMD based).

 

Q1: Are there specific projects that invoke this behavior at your end, or is it pretty much any sample in the NCS tree?

Q2: Could you share the report generated by executing this command in vscode?

  

Kind regards,

Håkon

[liteyear]

It's a pity the OP didn't continue with the thread, although their workaround is excellent.

This issue has plagued me for a long time. FWIW, here's my report. I've just replaced project/client names with placeholders.

Fullscreen nrf connect support info.txt Download

IMPORTANT: The data below could contain sensitive or confidential information about your environment.
If you do not wish this to be seen by others, please make sure to remove it before sharing.

For help and support, visit the Nordic DevZone at https://devzone.nordicsemi.com/.

{
  "platform": {
    "os": "darwin",
    "osVersion": "Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000",
    "osKernel": "23.3.0",
    "vscode": "1.88.1",
    "electron": "28.2.8",
    "node": "v18.18.2"
  },
  "system": {
    "date": "2024-04-15T06:08:02.126Z",
    "vscodeRoot": "/Applications/Visual Studio Code.app/Contents/Resources/app",
    "nrfConnectForDesktopInstalled": false,
    "vscodeUptime": "00:19:02",
    "osUptime": "172:14:34",
    "cpu": "Apple M1 Max"
  },
  "workspace": {
    "name": "project_name",
    "workspaceFile": null,
    "folders": [
      "/Users/liteyear/path/to/project_name"
    ]
  },
  "sdks": [
    {
      "version": "v2.5.2",
      "path": "/opt/nordic/ncs/v2.5.2"
    },
    {
      "version": "v2.5.0",
      "path": "/opt/nordic/ncs/v2.5.0"
    },
    {
      "version": "v2.3.0",
      "path": "/opt/nordic/ncs/v2.3.0"
    }
  ],
  "activeBuild": "/Users/liteyear/path/to/project_name/build_name",
  "apps": [
    {
      "workspace": "/Users/liteyear/path/to/project_name",
      "uri": "/Users/liteyear/path/to/project_name",
      "buildConfigurations": [
        {
          "id": "/Users/liteyear/path/to/project_name/build",
          "name": "build",
          "boardId": "board_name_cpuapp",
          "type": "Zephyr",
          "isChild": false,
          "isStale": false,
          "taskBindings": {
            "build": [],
            "pristineBuild": [],
            "flash": [],
            "eraseAndFlash": []
          }
        },
        {
          "id": "/Users/liteyear/path/to/project_name/build_name",
          "name": "build_name",
          "boardId": "board_name_cpuapp",
          "type": "Zephyr",
          "isChild": false,
          "isStale": false,
          "taskBindings": {
            "build": [],
            "pristineBuild": [],
            "flash": [],
            "eraseAndFlash": []
          }
        }
      ]
    }
  ],
  "topdir": "/opt/nordic/ncs/v2.5.2",
  "workspaceState": "freestanding-ready",
  "toolchains": [
    {
      "version": "2.5.2",
      "path": "/opt/nordic/ncs/toolchains/20d68df7e5"
    },
    {
      "version": "2.5.0",
      "path": "/opt/nordic/ncs/toolchains/20d68df7e5"
    },
    {
      "version": "2.3.0",
      "path": "/opt/nordic/ncs/toolchains/v2.3.0"
    }
  ],
  "connectedDevices": [
    {
      "serialNumber": "000176000446"
    }
  ],
  "deviceProvider": "nrfutil",
  "tools": {
    "/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc": "0.16.1",
    "/Applications/SEGGER/JLink/JLinkExe": "7.94e",
    "/usr/local/bin/nrfjprog": "10.24.0",
    "/usr/local/bin/nrfutil": "7.7.0",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/cmake": "3.20.5",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/west": "1.1.0",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/python3": "3.9.6",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/ninja": "1.10.2",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/gperf": "3.1",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/dtc": "1.6.1",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/gn": "2122",
    "/opt/nordic/ncs/toolchains/20d68df7e5/bin/git": "2.37.3",
    "gdbPath": "/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb"
  },
  "nrfutil": {
    "nrfutil-device": {
      "version": "2.0.0",
      "binPath": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-connect-2024.3.25-darwin-arm64/platform/nrfutil/bin/nrfutil-device"
    },
    "nrfutil-toolchain-manager": {
      "version": "0.14.1",
      "binPath": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-connect-2024.3.25-darwin-arm64/platform/nrfutil/bin/nrfutil-toolchain-manager"
    }
  },
  "environment": {
    "westEnv": {
      "ELECTRON_RUN_AS_NODE": "1",
      "GIT_EXEC_PATH": "/opt/nordic/ncs/toolchains/20d68df7e5/Cellar/git/2.37.3/libexec/git-core",
      "HOME": "/Users/liteyear",
      "PATH": "/opt/nordic/ncs/toolchains/20d68df7e5/bin:/opt/nordic/ncs/toolchains/20d68df7e5/usr/bin:/opt/nordic/ncs/toolchains/20d68df7e5/usr/local/bin:/opt/nordic/ncs/toolchains/20d68df7e5/opt/bin:/opt/nordic/ncs/toolchains/20d68df7e5/opt/nanopb/generator-bin:/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/aarch64-zephyr-elf/bin:/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/x86_64-zephyr-elf/bin:/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin:/Users/liteyear/.opam/default/bin:/Users/liteyear/.nvm/versions/node/v20.5.1/bin:/Users/liteyear/.cabal/bin:/Users/liteyear/.ghcup/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/liteyear/.cargo/bin:/Users/liteyear/Library/Python/3.11/bin:/opt/homebrew/opt/llvm@12/bin:/Users/liteyear/go/bin:/Users/liteyear/.opam/default/bin:/Users/liteyear/.nvm/versions/node/v20.5.1/bin:/Users/liteyear/.cabal/bin:/Users/liteyear/.ghcup/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/liteyear/.cargo/bin:/Users/liteyear/Library/Python/3.11/bin:/opt/homebrew/opt/llvm@12/bin:/Users/liteyear/go/bin",
      "ZEPHYR_BASE": "/opt/nordic/ncs/v2.5.2/zephyr",
      "ZEPHYR_TOOLCHAIN_VARIANT": "zephyr",
      "ZEPHYR_SDK_INSTALL_DIR": "/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk",
      "USER": "liteyear"
    },
    "inherited": {
      "ELECTRON_RUN_AS_NODE": "1",
      "HOME": "/Users/liteyear",
      "PATH": "/Users/liteyear/.opam/default/bin:/Users/liteyear/.nvm/versions/node/v20.5.1/bin:/Users/liteyear/.cabal/bin:/Users/liteyear/.ghcup/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/liteyear/.cargo/bin:/Users/liteyear/Library/Python/3.11/bin:/opt/homebrew/opt/llvm@12/bin:/Users/liteyear/go/bin",
      "USER": "liteyear"
    },
    "toolchainPath": "/opt/nordic/ncs/toolchains/20d68df7e5",
    "toolchainBinPath": "/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin",
    "toolchainVersion": "2.5.2"
  },
  "terminal": {
    "defaultProfile": null
  },
  "config": {
    "nordic-semiconductor.nrf-connect": {
      "topdir": "${nrf-connect.sdk:2.5.2}",
      "toolchain": {
        "path": "${nrf-connect.toolchain:2.5.2}"
      },
      "ozonePath": "",
      "applications": [],
      "applicationOptions": {},
      "kconfig": {
        "interface": "kconfig"
      },
      "west": {
        "env": {
          "$base": "terminal"
        }
      },
      "boardRoots": [],
      "taskBindings": {},
      "buildTerminal": {
        "condensedProgress": true
      },
      "debugging": {
        "flash": true,
        "bindings": {
          "${workspaceFolder}/build_name": "Launch build_name"
        },
        "justMyCode": true
      },
      "activeAppFollowActiveEditor": true,
      "deviceProvider": "",
      "flash": {
        "softreset": false,
        "erase": false,
        "recover": false
      },
      "enableTelemetry": true,
      "thirdpartyIntegration": {},
      "toolchainManager": {
        "indexURL": null,
        "installDirectory": null
      },
      "nrfutil": {
        "home": null
      },
      "defaultOpenAction": "ask"
    },
    "nordic-semiconductor.nrf-terminal": {
      "terminalMode": "character"
    }
  },
  "extensions": {
    "internal": {
      "nordic-semiconductor.nrf-connect": {
        "version": "2024.3.25",
        "path": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-connect-2024.3.25-darwin-arm64",
        "isActive": true
      },
      "nordic-semiconductor.nrf-terminal": {
        "version": "2024.3.15",
        "path": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-terminal-2024.3.15",
        "isActive": true
      },
      "nordic-semiconductor.nrf-devicetree": {
        "version": "2024.3.78",
        "path": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-devicetree-2024.3.78",
        "isActive": true
      },
      "nordic-semiconductor.nrf-kconfig": {
        "version": "2024.3.21",
        "path": "/Users/liteyear/.vscode/extensions/nordic-semiconductor.nrf-kconfig-2024.3.21",
        "isActive": true
      }
    },
    "external": {
      "marus25.cortex-debug": null,
      "ms-vscode.cpptools": "1.19.9",
      "ms-vscode.js-debug": "1.88.0",
      "ms-vscode.js-debug-companion": "1.1.2",
      "ms-vscode.vscode-js-profile-table": "1.0.8",
      "elmTooling.elm-ls-vscode": "2.8.0",
      "haskell.haskell": "2.4.3",
      "hbenl.vscode-test-explorer": "2.21.1",
      "JakeBecker.elixir-ls": "0.20.0",
      "janisdd.vscode-edit-csv": "0.9.1",
      "justusadam.language-haskell": "3.6.0",
      "ms-azuretools.vscode-docker": "1.29.0",
      "ms-python.debugpy": "2024.4.0",
      "ms-python.isort": "2023.10.1",
      "ms-python.python": "2024.4.1",
      "ms-python.vscode-pylance": "2024.4.1",
      "ms-toolsai.jupyter": "2024.3.1",
      "ms-toolsai.jupyter-keymap": "1.1.2",
      "ms-toolsai.jupyter-renderers": "1.0.17",
      "ms-toolsai.vscode-jupyter-cell-tags": "0.1.9",
      "ms-toolsai.vscode-jupyter-slideshow": "0.1.6",
      "ms-vscode-remote.remote-containers": "0.354.0",
      "ms-vscode.cpptools-extension-pack": "1.3.0",
      "ms-vscode.cpptools-themes": "2.0.0",
      "ms-vscode.test-adapter-converter": "0.1.9",
      "phoityne.phoityne-vscode": "0.0.29",
      "platformio.platformio-ide": "3.3.3",
      "rebornix.prolog": "0.0.4",
      "trond-snekvik.gnu-mapfiles": "1.1.0",
      "unison-lang.unison": "1.2.0",
      "vlanguage.vscode-vlang": "0.1.14"
    }
  }
}

[Håkon Alseth]

Hi,

 

Are you seeing the same gdb-assertion message?

...gdb/gdb/infrun.c:5825: internal-error: finish_step_over: Assertion `ecs->event_thread->control.trap_expected' failed.

 

If yes, this is an issue that we have reported back to the toolchain:

https://github.com/zephyrproject-rtos/sdk-ng/issues/727

 

The issue seems to occur if your first breakpoint is inside interrupt context, and you have thread aware configurations (DEBUG_THREAD_INFO / DEBUG_INFO) enabled. If your first breakpoint is in thread context (main or something); the issue seems to be less frequent.

 

Kind regards,

Håkon

[liteyear]

Håkon Alseth said:

Are you seeing the same gdb-assertion message?

I haven't even looked. My issue is as per the OP's. The workaround is just a workaround.

I want to be able to restart a debugging session without the firmware re-flashing. That fundamental function does not appear possible without this debug console workaround.

Are you saying that the original issue is due to the GDB issue?