Best way for UART with dictionary based logging, DFU, Shell

[colar]

Hi,

in our current products we use UART for many purposes in development and service. We have

• a shell,
• advanced logging (colors, human readable text for error codes, dictionary based to save flash etc.),
• DFU,
• API for interacting with the device (like reading some ID or factory testing)

all over one UART. To achieve that, we have our own UART protocol to allow multiplexing and a software running on a host computer to talk to the µC with this protocol and map text to IDs (like the JSON from the dictionary based logging), forward firmware images and so on.

Now we transition to nRF5340 with Zephyr and I am evaluating the possibilities with Zephyr RTOS. As far as I can see, combining the shell with dicitionary based logging and DFU over one interface is not something that's intended to do with the default modules, as some multiplexing protocol and counterpart on the host computer is needed. Is this correct?

if so, I would stick with our own debugging protocol and integrate a logging backend into that to send the logging messages to the host encapsulated in our protocol. A shell would have to be done by my own on the host computer and for DFU, I would transfer data into an external flash to flash it with MCUBoot later on. So I would only use the Logging module from Zephyr. Would you recommend this way or are there better ways with Zephyr? I feel like this could be done easier with Zephyr modules.

Best regards,
Lars

[Einar Thorsrud]

Hi Lars,

Zephyr supports dictionary based logging, and that is demonstrated by the Dictionary-based Logging Sample. But you are of course free to use another dictoinary based loggign method of your own. Regarding multiplexing logging and serial DFU, that is also supported (see UART/serial and console).

It is difficult to recommend any specific approach, particularily if you have existing implementations, but Zephyr provides the features you describe.

Einar

[colar]

Hi Einar,

thanks for your answer.

We do have an implementation for our non-Zephyr devices, but let's forget about them for now, since I only wanted to demonstrate the needed features and I want to see what's possible with Zephyr.

The features on its own are all supported in some way - Dictionary based logging for logging, Shell for user input and DFU with mcumgr/SMP.

But can these features be combined onto a single UART interface? Would SMP be the underlying protocol and logging be packed into that with an application specific management group? In the end I would like to have an interactive shell where the logging output can be seen.

Best regards
Lars

[Einar Thorsrud]

Hi Lars,

In principle dictionary based logging is just logging some shorter identifiers (with added complexity in the buidl process an on the terminal side), so I do not see any fundamental reason it could not be combined. However, the way SMP multiplexing is implemented in Zephyr, the UART console handles mcumgr passthroiugh (with  CONFIG_UART_CONSOLE_MCUMGR enabled in the configuration), and that is the only combination is supported out of the box. Other configurations and combinations you would need to add support for yourself, but I do not see any fundamental reasons it should not be possible. You could use an application specific management group if you like also, which would mean that everything is encapsulated by SMP, but that is a different approach than what is used with the UART console (where the SMP frames are extracted based on a header).

Einar

[colar]

Hi Einar,

with the options

CONFIG_LOG=y
CONFIG_SHELL=y
CONFIG_SHELL_LOG_BACKEND=y
CONFIG_UART_CONSOLE_MCUMGR=y
CONFIG_MCUMGR=y
CONFIG_MCUMGR_SMP_SHELL=y
CONFIG_MCUMGR_CMD_OS_MGMT=y
CONFIG_MCUMGR_CMD_SHELL_MGMT=y
CONFIG_OS_MGMT_ECHO=y

I can use the shell, logging and SMP requests (tested it with mcumgr echo) and they seem to be extracted by their header as you said. CONFIG_UART_CONSOLE_MCUMGR does not seem to make a difference here. However, sometimes the commands timeout. Inspecting the traffic shows that in cases of a timeout, logging output is sent to mcumgr and some bytes in the beginning (including 0x06 0x09) are missing, so mcumgr cannot extract the answer and times out. So logging and SMP somehow disturb each other, is this due to an invalid config or a known bug? Or is the shell interfering?

Looking at the doc of CONFIG_LOG_DICTIONARY_SUPPORT, it states " Dictionary based logging is binary based where predefined strings are replaced with their equivalent addresses from the built binary image file in log output.". So all static strings stay in the binary, don't they? I initially wanted to have them removed for a smaller image size and to avoid string formatting on the end device, but this is not really supported, right?

Best regards,
Lars

[Einar Thorsrud]

Hi Lars,

You are right. I must admit I have never used the Dictionary feature from Zephyr or seen it in use, I just assumed that it supported not including the strings in the code. But you are right, it seems it is only intended for compressing the logs itself, so you would need to modify it to not include the full log strings in the binary. 

I am not aware of any known issues with mcumgr and console multiplexing, but that does not mean that it cannot exist. CONFIG_UART_CONSOLE_MCUMGR should matter, as that is used in the UART console implementation to include the code that handles extraction of mcumgr data (see the implementation in zephyr/drivers/console/uart_console.c). This is also used in SMP server sample if you build with overlay-serial-console.conf.

Einar

[colar]

Hi Einar,

okay, I will have to look further how to implement all features I want. Maybe the best way to have "compact" logging with minimal impact on the end device (string templates stored on host machine, formatting also done there) would be to create (or reuse) our own logging functions and integrate Zephyr logs into them using a custom backend.

But it seems that there does seem to be a problem with SMP and Logging, sometimes I get the expected response, sometimes the first few bytes of the response are missing and log output follows the invalid response to mcumgr. Sound like logging output at the wrong time is leading to that. Should I open a new ticket for that?

Best regards,
Lars

[Einar Thorsrud]

Hi Lars,

It makes sense to keep it in a separate case. Linking in here for reference.