Critical bug in nRF Connect SDK v2.1.0 no pin configuration in devicetree header

[Georgi Valkov]

I just upgraded my environment from nRF Connect SDK v1.9.1 to v2.1.0. This process raised one critical bug, as well as some very important questions:

My projects make very minor changes to a copy of existing samples and boards:

I maintain modified boards: nrf9160dk_nrf9160 and nrf9160dk_nrf52840 where a second UART on each CPU connects them together. Next I change the gnss sample: a copy of the GNSS data is sent over UART1 to nrf52840, which makes it available over BLE (modified connectivity_bridge), and also over the it's other UART which is exposed over USB by the board controller.

1. Let's start with my expectations: a well maintained SDK should allow user code to be ported to a new SDK version with minimal or no effort.

2. Unfortunately this has never been the case with nRF Connect SDK. Even though my changes are minor and I work on copies of each board or sample, they never build when a new SDK is released, because each time the DSK is updated, the changes to API and the build environment are so big that every single aspect of the code and Cmake configuration needs to be adapted. This may take up to 2-3 days.

3. Critical bugs in release SDK v2.1.0 versions:

Once all boards and projects are adapted to a new SDK, we often face critical bugs in nRF Connect SDK, which makes it completely impossible to run anything. After days of debugging I figured that the UART pins on nRF52840 are not configured. All pins get the default value of -1. The baud_rate configuration is valid. Below in a snippet of my driver.

.pin_select_dma =
{
.CTS = DT_PROP_OR(DT_NODELABEL(uart0), cts_pin, -1),
.RTS = DT_PROP_OR(DT_NODELABEL(uart0), rts_pin, -1),
.RXD = DT_PROP_OR(DT_NODELABEL(uart0), rx_pin, -1),
.TXD = DT_PROP_OR(DT_NODELABEL(uart0), tx_pin, -1),
},
.baud_rate = DT_PROP_OR(DT_NODELABEL(uart0), current_speed, 115200),
.enabled = DT_NODE_HAS_STATUS(DT_NODELABEL(uart0), okay),
.flow_control = DT_PROP_OR(DT_NODELABEL(uart0), hw_flow_control, 0),

As a workaround, I can define the following macros in one of my headers:

#define DT_N_S_soc_S_uart_40002000_P_tx_pin 5
#define DT_N_S_soc_S_uart_40002000_P_tx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_rx_pin 20
#define DT_N_S_soc_S_uart_40002000_P_rx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_rts_pin 15
#define DT_N_S_soc_S_uart_40002000_P_rts_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_cts_pin 7
#define DT_N_S_soc_S_uart_40002000_P_cts_pin_EXISTS 1

#define DT_N_S_soc_S_uart_40028000_P_tx_pin 17
#define DT_N_S_soc_S_uart_40028000_P_tx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_rx_pin 3
#define DT_N_S_soc_S_uart_40028000_P_rx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_rts_pin 40
#define DT_N_S_soc_S_uart_40028000_P_rts_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_cts_pin 22
#define DT_N_S_soc_S_uart_40028000_P_cts_pin_EXISTS 1

Then all pins are configured properly and the connectivity_bridge sample works. Further investigation shows that the build script responsible for generating devicetree_unfixed.h does not generate any pin configuration. All CPUs are affected: non-modified samples and boards produce devicetree withoult pin configuration. At this state any stock sample or user project compiled with nRF Connect SDK v2.1.0 will not function properly.

This opens the question: Why is a completely broken SDK released in first place? This is not the first time Nordic releases an SDK incapable of building working code.

4. To make debugging even harder, the following other issue comes into play: a crash in the board controller prevent users from flashing nRF9160dk if any of the UARTs are open.  nRF9160dk onboard controller crash 

[ovrebekk]

Hi 

Sorry for the slow response from the Nordic side. I took over this case from Håkon a while back, and have been observing the discussion from the side line without taking part. 

It has been a bit of a bumpy ride with the nRF Connect SDK introduction no doubt, but it is getting better day by day and we are putting a lot of effort into helping people get through the initial hurdle of getting started with it. 

My personal experience getting into the SDK is that the initial learning curve is very steep, but once you start to understand how the various build systems, API's etc are put together, and understand how to use the Zephyr RTOS to the best effect, then the productivity goes up considerably compared to the nRF5 SDK. 

As for complexity I believe this is a bit of a mixed bag. The build system is certainly more complex, and if for instance you want to make your own device driver this is quite challenging until you understand how the build system works, but I feel that most of the API's are clear and concise. 

As an example the BLE heart rate sample in Zephyr is 130 lines long, compared to close to a 1000 lines for the equivalent nRF5 SDK example, and a lot of this is down to the simpler Bluetooth API's. 

Project configuration with Kconfig is generally a lot easier than working with sdk_config.h too, and moving from one chip to another is easier because of the board architecture. 

Regarding UART drivers I agree there is some work to be done there. I am actually working on my own wrapper for the UART ASYNC driver, in order to simplify UART handling overall, handle application buffering of data and remove the need for dynamic memory allocation. Feel free to have a look at the driver here if you're interested. I'm planning to get it into the nRF Connect SDK once I have had more time to stress test it, and for what it's worth it only has two public functions and a single callback, so it should satisfy the 'simple API' criteria 

I don't know if this driver would help with the connectivity_bridge sample though, the main focus has been on the peripheral_uart sample. 

And in general never hesitate to get in touch with us regarding any issues you find in the SDK. It will never be perfect, and we can't satisfy every request, but it is always better to report something to us rather than just get frustrated on your own

Regarding LTS releases this is something that we are considering, but we don't have any concrete plans to share yet unfortunately. 

Best regards
Torbjørn

[Georgi Valkov]

Hello Overbekk!

ovrebekk said:

As an example the BLE heart rate sample in Zephyr is 130 lines long, compared to close to a 1000 lines for the equivalent nRF5 SDK example, and a lot of this is down to the simpler Bluetooth API's. 

I've been looking for information how the BLE driver and stack are organised in the build system. I would like to use them with a different OS. My goal would be to find all related files and copy them outside the SDK. Then I can provide all necessary API which Zephyr provides and build a library using arm-none-eabi-gcc and make. Any help on that would be very helpful.

ovrebekk said:

if for instance you want to make your own device driver this is quite challenging until you understand how the build system works

I agree! I actually implemented my own UART miniport driver, which can work on bare metal, or integrate with an OS such as Zephyr or nano RTOS so that tasks sleep while waiting for the hardware. The driver was designed with a simple API, high efficiency, low latency and portability in mind. So it can work with any RTOS that provides task wait and wake from interrupt API. Attached below is the source code integrated in connectivity_bridge. Can you please help me make it into a library for nRF Connect SDK? Ideally the user should be able to disable UARTE and enable UART miniport through KConfig, then use the driver API. It would be nice if we can make it part of the SDK to benefit everyone who wants to use it.

I did a performance comparison between original and modified versions of connectivity bridge. Consider an external CPU is sending a lot of data, keeping the UART RX busy 100% of the time at 115200 baud rate, e.g.

for (int i = 0; 1; i++) printf("UART test %8i\n", i);

With the original connectivity_bridge the BLE client receives large packets with a huge delay between them. With the modified code attached below the packet size is adaptive based on load, keeping latency as low as possible. CPU usage is also highly reduced. One change I had to make was to move the BLE send code to a dedicated higher priority task, instead of using the system work queue. The work queue would choke when a lot of data is received, especially if USB UART is also connected to a computer and in use. In my test I used connectivity_bridge_52840 which runs on nRF52840dk.

uart_miniport 2022-11-07.7z

ovrebekk said:

I am actually working on my own wrapper for the UART ASYNC driver, in order to simplify UART handling overall, handle application buffering of data and remove the need for dynamic memory allocation. Feel free to have a look at the driver here if you're interested. I'm planning to get it into the nRF Connect SDK once I have had more time to stress test it, and for what it's worth it only has two public functions and a single callback, so it should satisfy the 'simple API' criteria

That's great! Makes me curious if you can compare the performance to my driver? I'll be happy to see both available in the SDK. Since my code has no external dependencies, it should be easy to integrate. One important note about UART miniport: send and receive calls are somewhere in between synchronous and and asynchronous. As long as the driver buffer has enough space to store all TX data, the call returns immediately. If the buffer is full, the call returns as soon as all data is copied, transmission continues using DMA on the background. for RX, incoming data is stored in the RX buffer. A recv request returns as soon as any data is available to read, otherwise it blocks (it can also be configured to wait until all requested data is received). The benefit of this design is that the user can easily implement recv, process, send on one task without any risk of dropping data. And since there is no need for synchronisation between tasks or queues, this results in a very simple and reliable code.

ovrebekk said:

And in general never hesitate to get in touch with us regarding any issues you find in the SDK. It will never be perfect, and we can't satisfy every request, but it is always better to report something to us rather than just get frustrated on your own

Thank you very much! Indeed despite of the many issue, I can confirm that the Nordic team has always been very kind and helpful. 

[ovrebekk]

Hi Georgi

Georgi Valkov said:

I've been looking for information how the BLE driver and stack are organised in the build system. I would like to use them with a different OS. My goal would be to find all related files and copy them outside the SDK. Then I can provide all necessary API which Zephyr provides and build a library using arm-none-eabi-gcc and make. Any help on that would be very helpful.

Could you open a new ticket on this? 

This sounds like a pretty complex issue, and the case is getting pretty long already (the devzone is not really designed for handling multiple different issues in the same case). 

I will have to get back to you on the UART example, but I will take a look at it and get back to you in a day or two. 

Georgi Valkov said:

Thank you very much! Indeed despite of the many issue, I can confirm that the Nordic team has always been very kind and helpful. 

That is good to hear

Best regards
Torbjørn

[Georgi Valkov]

Thanks Torbjørn! The new ticket is here:
Use BLE driver and stack outside nRF Connect SDK

I made a tiny update to my driver.

uart_miniport 2022-11-20.7z

[ovrebekk]

Hi Georgi

Thanks for sharing your latest driver. I haven't been able to try it out yet, but I took a look at the code. 

A challenge with getting something like this into the SDK is that it is a complete rewrite of the current UART driver, basically replacing what is already there. 

For that to make sense it would basically have to be able to improve on the existing driver for all use cases across all nRF devices, and it would also need to follow the coding guidelines defined by Zephyr, and the standard device driver model that all Zephyr drivers incorporate. 

My UART driver by comparison is just a small wrapper on top of the UART ASYNC driver, and don't require very large changes to the existing framework. 

When implementing changes to Zephyr or any of the other repositories under the nRF Connect SDK umbrella it is also recommended to make a fork of the original repository to your own Github account, and implement the changes on this fork. 

Then you can easily generate a patch showing the difference between the original repository and your own changes, and in case you want to suggest an update to the upstream repositories you can issue a pull request. 

I hate to be negative, but in general getting alternative drivers and such into the official repositories is not very easy. If there is a problem with the official driver then this should be fixed in that driver, rather than having multiple competing drivers in the same repository. 
If you could implement your driver as a patch though then it would make it easy for people to try it out, and easy for you to patch it in after downloading the SDK, simplifying the process of adding the driver to an existing sample. 

By the way, I noticed you also had board files included in the zip. What did you change in the board files? 

Best regards
Torbjørn

[Georgi Valkov]

Hello Torbjørn!

I updated my forks. The UART miniport driver is inside nrf/applications/connectivity_bridge_*/src/uart/

https://github.com/httpstorm/sdk-nrf/tree/2022-11-20

https://github.com/httpstorm/sdk-zephyr/tree/2022-11-20

ovrebekk said:

I hate to be negative, but in general getting alternative drivers and such into the official repositories is not very easy.

Yes, I've heard this before. Like you said, the changes are too big. And this is intentional. If I build atop the original driver, simple API, low latency and high performance are out of the question. I also understand that such a change is hard to accept in the environment of nRF Connect SDK. That's fine. Replacing the original driver has never been my intent. I believe users will benefit if given the choice to select one or the other. Of course existing samples will not be compatible, though it is quite easy to support both using #ifdef like I did with connectivity_bridge, or create a dedicated sample. It would be safer to keep UARTE as default. Either way, my fork is now online so people can benefit from it.

A nice thing to do would be to make the UART miniport driver available as a module that can be selected using KConfig. Can you please help me with that?

Regarding the nRF9160dk board files: connectivity_bridge is designed to run on Thingy:91. In my environment I also use it on nRF9160dk. In order for this to work, some board changes are needed. A few pins are routed so that nRF9160 and nRF52840 have UART connectivity.

• nRF9160 UART0 RX, TX are routed to USB UART0 TX, RX
• nRF9160 UART1 TX is routed to nRF52840 UART0 RX
• nRF9160 UART1 RX is routed to nRF52840 UART1 TX
• nRF52840 UART0 TX is routed to USB UART1 RX
• nRF52840 UART1 RX is routed to USB UART1 TX

You likely noticed that UART0-1 on nRF52840 are cross-wired and this is intentional. Data from UART0 RX events is sent to UART0 TX, so anything received from nRF9160 is forwarded to USB UART1. Anything received from USB UART1 goes to nRF9160. This simplifies the connectivity_bridge design, where the common practice is to forward data between interfaces with the same index.

There is one more addition to nRF9160 board files: just like you figured the original API is overcomplicated, a couple of years ago, I needed a simple synchronous send for UART1. So this was my attempt which took over 300 lines of code, see uart_ble.c. Since then I lack any motivation to use the UARTE driver. There is a wisdom saying that API should be easy to use correctly and hard to use incorrectly.

Board changes diff https://github.com/httpstorm/sdk-zephyr/commit/79cffdd6fd352317b2ee55d2864259660c3494b0

BTW You might have noticed I also have connectivity bridge running on nRF52840. No board changes are needed. I just changed the advertised BLE name, because I usually have connectivity bridge running on Thingy:91, nRF9160dk and nRF52840dk at the same time. I need to make it so that instead of keeping 3 dedicated projects differing by the BLE name, I keep only one, and the actual name is selected by the target board. The code is identical.

[ovrebekk]

Hi Georgi

Thanks for sharing the updated forks

Georgi Valkov said:

If I build atop the original driver, simple API, low latency and high performance are out of the question.

I am confident I can fix the first of those three with my UART async wrapper, but if I can do anything about the other two only testing will tell. I am guessing it won't be as fast as your implementation

Georgi Valkov said:

Replacing the original driver has never been my intent. I believe users will benefit if given the choice to select one or the other. Of course existing samples will not be compatible, though it is quite easy to support both using #ifdef like I did with connectivity_bridge, or create a dedicated sample. It would be safer to keep UARTE as default. Either way, my fork is now online so people can benefit from it.

Yes, having options is great. The more options there are available for Zephyr developers the better, and the standard drivers will never be able to fit all use cases (not to say they can't be improved). Thanks again for making the code available

Georgi Valkov said:

A nice thing to do would be to make the UART miniport driver available as a module that can be selected using KConfig. Can you please help me with that?

A colleague of mine pointed me to a nice guide online walking through the specifics of making a Zephyr module. He also provides an example module to check out, and a sample showing how to use it. Please have a look here.

I used this as a reference when I 'modularized' my own UART wrapper, that I linked earlier. 

Please note that this module doesn't show how to implement device tree bindings, since it doesn't rely on peripheral drivers. I still need to add this to my UART library (hopefully I have some time next week). 

If you still need some assistance after looking through Iwasz' example and blog let me know, and I will do my best to help. If you have a lot of questions you might want to open a new ticket again (if you want it to be assigned to me just mention that I already helped you with a similar issue).

Best regards
Torbjørn