Log and shell through RTT backend don't play nicely

RE: Log and shell through RTT backend don't play nicely

Simonr — Thu, 21 Jul 2022 06:45:47 GMT

This case is getting pretty old, and opening a new discussion in the middle of this one will make it messier I'm afraid. Can you create a new ticket instead and link to this one please? In the meantime I can recommend reading through the reported Zephyr issue and make sure you have tried what is suggested there. It's also recommended to move to the latest NCS/Zephyr version if you're starting on a new project.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

ahoward — Wed, 20 Jul 2022 16:41:26 GMT

Hi,

I am experiencing the same problem directing shell to both RTT and UART backend. As other have found above, uart outputs as expected until the RTT Viewer is disconnected. Then, some additional prints are outputted before uart stops.

I have looked through the other messages in this thread and cherry-picked the commits mentioned later in this thread from backport 2.7 of the Zephyr Project, but had no change in outcome. Here are some snips of the issue I’m seeing, which aligns with what was found in previous responses.

I am using the sample in zephyr/samples/subsys/shell/shell_module with the nRF52850dk.

Below is the zipped shell module sample including my board overlay for the nRF52840, build folder for reference, and prj.conf with added configs CONFIG_SHELL_BACKEND_RTT=y and CONFIG_SHELL_BACKEND_UART=y. I have also added a while(1) loop with prints in main to show this issue.

devzone.nordicsemi.com/.../shell_5F00_module.zip

Is there something I’m missing in regards to shell having multiple backends?

Thank you in advance for your help.

Alyssa

RE: Log and shell through RTT backend don't play nicely

Simonr — Tue, 19 Apr 2022 08:37:49 GMT

I finally heard back from the developers, as they've been out of office during Easter. This sounds similar to this issue in Zephyr from some time ago. If was fixed in Zephyr and backported to Zephyr v2.7 here. Since NCS v1.8.0 it should be possible to cherry-pick commits from this backport PR since it's based on Zephyr v2.7.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Simonr — Mon, 04 Apr 2022 13:04:47 GMT

We have started investigation, but I have not received any progress worth mentioning as of yet I'm afraid. I'm very sorry about the delay on this issue, but it wasn't as straight forward as initially expected. Thank you for your patience!

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Wed, 30 Mar 2022 17:42:41 GMT

any info yet? it's been quite some time..

RE: Log and shell through RTT backend don't play nicely

Simonr — Wed, 16 Mar 2022 11:48:19 GMT

No, I'm afraid we haven't made any progress yet, but we're looking into it, and I pinged them requesting an update. Hopefully I have some more info for you soon. I'm sorry this is taking so long.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Tue, 15 Mar 2022 22:08:22 GMT

Simonr any updates on this?

RE: Log and shell through RTT backend don't play nicely

Simonr — Mon, 21 Feb 2022 06:50:45 GMT

Thank you Martin, I have forwarded this information to our developers.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

mtsunstrum — Sat, 19 Feb 2022 04:52:16 GMT

I have experienced something similar, even simply with RTT logging, and no shell backend.

So it is possible it is a more fundamental issue that is easier to isolate

To recreate, take example application central_uart

In prj.conf file, add the following to get more log messages:

CONFIG_BT_DEBUG_CONN=y
CONFIG_BT_LOG_LEVEL_DBG=y

The second CONFIG_ entry will generate a flood of RTT messages. The RTT viewer will indicate that 500+ messages were missed, and will display 10 or more log lines. But after that, there no no more messages.

Application though is still running fine.

RE: Log and shell through RTT backend don't play nicely

Simonr — Fri, 18 Feb 2022 12:47:03 GMT

I agree that this sounds more like a bug than anything else, and I have reported it internally for the developers to get a better look into this. It might take a few days before we hear back from them, but I will keep you updated if there is any progress.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Thu, 17 Feb 2022 22:28:25 GMT

Hi,

I'm glad you were able to reproduce.

I do not see how those two settings could fix the problem, and I believe it is a bug that's causing this issue. Let me explain:

Those settings are controlling the RTT retry while in an active connection with RTT, if you read the description.

The problem in this case is RTT buffer is full, and we know that from "log mem" that I showed in the screenshot above.

However upon opening an RTT connection, while the buffer is full, the RTT doesn't pull the data from the buffer!!!

I suspect the RTT log backend is overwriting the data in the buffer, but somehow "the trigger to download (or upload to PC)" the buffer is not working.

Also from a different view, if you're suggesting increasing those params help, (which I don't think they do), they will only help up to when their limits are reached, so for a system that has been working for days, months, years, and has been overwriting the logs in the RTT buffer, eventually the retry ms and retry count will be reached and the same problem will occur. i.e. when you go connect via RTT viewer to such system, if we're beyond those timeout * counts, then it has stopped trying (based on what you suggested) and it will stop logging. This isn't the expected behavior.

The expected behavior is RTT buffer gets downloaded into the RTT viewer on connect (this always worked with previous versions, nRF SDK and nRF NCS) and this is failing in this example. Simply that is the root cause or the issue itself. If you don't believe this, when you have the issue reproduced, keep sending "log mem" and you will get Blocks Free: 0. no matter how long you wait.

In fact it looks like it downloads the buffer, but then it something breaks the RTT process? perhaps need more stack? higher system work Q buffer? something on those lines?

RE: Log and shell through RTT backend don't play nicely

Simonr — Thu, 17 Feb 2022 12:11:42 GMT

Thank you for the clarification, I had missed adding one of the overlays on my end, and was able to reproduce this. Have you tried playing with the defines that (by default) are set in the log_backend_rtt.c file? Here you have CONFIG_LOG_BACKEND_RTT_RETRY_DELAY_MS and CONFIG_LOG_BACKEND_RTT_RETRY_CNT that together sets up how often and how many times the backend will retry before dropping data and assuming an RTT session is inactive. Increasing both these values so they occur rarer and a higher total amount I think should help here.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Wed, 16 Feb 2022 21:54:38 GMT

This is how I build with approach 1:

west build -b nrf52840dk_nrf52840 -d build_appr1 -- -DOVERLAY_CONFIG='overlay-serial.conf;overlay-fs.conf;overlay-shell-mgmt.conf;overlay-approch1-shell-thru-rtt.conf'

I suspect you're connecting too quickly after disconnect. Please leave >10seconds after disconnect, and then connect.

This allows the buffer to fill up and I think that's what causes the issue.

OR From boot: Connect to the board via RTT viewer after it has been running for at least 10 seconds (from boot).

I've also attached my build folder for comparison, in case you're still not experiencing the same issue as I, try programming build_appr1\zephyr\merged.hex directly and see if you do.
if you do, then our build environment could be different, if you don't then it may have to do with Segger Software differences?! I use 7.58 and I also tried on an olde PC with 6.xx

.devzone.nordicsemi.com/.../build_5F00_appr1.zip

RE: Log and shell through RTT backend don't play nicely

Simonr — Fri, 11 Feb 2022 14:20:15 GMT

I tested the application and overlay files you sent over, but I'm not able to recreate what you describe in approach 1. After letting the log run for some time, I tried repeatedly to disconnect/close the RTT viewer, but the RTT viewer "smp_samples" kept trickling in without stopping, so I'm not sure if I need to make any changes to the project you uploaded or what?

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Fri, 11 Feb 2022 01:59:58 GMT

OK CONFIG_LOG_BACKEND_SHOW_COLOR=n does remove the extra "undefined characters"

However, the main issue is still there.

Why is Segger RTT unable to read the logs and interact with shell as well? It only provides shell in the "approach 2" explained above.

I have to open RTT viewer to interact with Shell and open RTT logger to see the logs. This isn't ideal, and opening RTT logger does break the RTT viewer connection.

I tried this both with Segger RTT Software 7.58b and 6.88a. No difference.

Regarding the RTT modes, I understand I think, but whether MODE_OVERWRITE or MODE_BLOCK, shouldn't it resume printing NEW logs (regardless of whether it overwrote or missed a bunch of logs) after RTT viewer is connected?

Again it would be really helpful if you can please load the sample mods I shared in the zip file. And see if you can get both Shell and Logs working in either approach 1 or 2. disconnect from RTT and then connect 5 seconds later to see if it resumes.

RE: Log and shell through RTT backend don't play nicely

Simonr — Thu, 10 Feb 2022 13:28:18 GMT

The "undefined" characters you're seeing seem to be color coding of the messages, which makes sense since CONFIG_LOG_BACKEND_SHOW_COLOR is set to Y by default.

Regarding the RTT "recovering" from a full buffer, that depends on how you handle a full buffer. You can for example check out the CONFIG_LOG_BACKEND_RTT_MODE_OVERWRITE which lets you overwrite messages if the buffer fills up. Another alternative is to wait until there's enough space in the buffer for a message with CONFIG_LOG_BACKEND_RTT_MODE_BLOCK.

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Thu, 10 Feb 2022 01:18:13 GMT

Hi Simon,

Those settings are already enabled. So adding them didn't make any difference.

Would you mind trying to build the project/ zip file I attached?

I faced this issue in our dev FW, then went back to SMP server which is the closest sample, and with small mods replicated the issue - hence this post.

The other post may have not been pushing much to the LOG buffer - perhaps that's why theirs works. and it's also based off of a different sample.

We expect the RTT to recover from a buffer full situation (i.e. if no RTT was connected to it for a while), then when RTT connection is established, it should download all the logs it has in its full buffer and resume downloading new logs, but it doesn't in this example.

RE: Log and shell through RTT backend don't play nicely

Simonr — Wed, 09 Feb 2022 10:04:12 GMT

The printk was mostly just because the user adds that themselves, but I imagined the configs would work similar for you. There was also another user just yesterday with similar issues, and there shouldn't be more to it than the following configs:

CONFIG_SHELL_BACKENDS=y
CONFIG_SHELL_BACKEND_RTT=y
CONFIG_LOG_BACKEND_RTT_BUFFER=1

You can see their full solution here: https://devzone.nordicsemi.com/f/nordic-q-a/84554/using-the-zigbee-shell-through-segger-rtt-backend

Best regards,

Simon

RE: Log and shell through RTT backend don't play nicely

Farhang — Tue, 08 Feb 2022 17:53:56 GMT

Hi,

I tried that but unfortunately it didn't fix the issue.

As soon as enough logs are backlogged in the buffer, connecting via RTT just drops the last set of logs and stops - i.e. doesn't show new logs - shell works.

So overall no difference.

My method for testing this was just making another overlay.conf file and copy pasted all of the kconfig setting in the post you mentioned above.

Also since I'm using LOG_XXX messages I don't see how printk() would have fixed it. But good to try.

RE: Log and shell through RTT backend don't play nicely

Simonr — Tue, 08 Feb 2022 14:04:49 GMT

We had a similar request a couple of months back where the user found the solution to enable both shell and log output via RTT and enable printk() RTT output. Please give it a try and see if that works better than the two approaches you mention above.

Best regards,

Simon