Critical bug in nRF Connect SDK v2.1.0 no pin configuration in devicetree header

[Georgi Valkov]

I just upgraded my environment from nRF Connect SDK v1.9.1 to v2.1.0. This process raised one critical bug, as well as some very important questions:

My projects make very minor changes to a copy of existing samples and boards:

I maintain modified boards: nrf9160dk_nrf9160 and nrf9160dk_nrf52840 where a second UART on each CPU connects them together. Next I change the gnss sample: a copy of the GNSS data is sent over UART1 to nrf52840, which makes it available over BLE (modified connectivity_bridge), and also over the it's other UART which is exposed over USB by the board controller.

1. Let's start with my expectations: a well maintained SDK should allow user code to be ported to a new SDK version with minimal or no effort.

2. Unfortunately this has never been the case with nRF Connect SDK. Even though my changes are minor and I work on copies of each board or sample, they never build when a new SDK is released, because each time the DSK is updated, the changes to API and the build environment are so big that every single aspect of the code and Cmake configuration needs to be adapted. This may take up to 2-3 days.

3. Critical bugs in release SDK v2.1.0 versions:

Once all boards and projects are adapted to a new SDK, we often face critical bugs in nRF Connect SDK, which makes it completely impossible to run anything. After days of debugging I figured that the UART pins on nRF52840 are not configured. All pins get the default value of -1. The baud_rate configuration is valid. Below in a snippet of my driver.

.pin_select_dma =
{
.CTS = DT_PROP_OR(DT_NODELABEL(uart0), cts_pin, -1),
.RTS = DT_PROP_OR(DT_NODELABEL(uart0), rts_pin, -1),
.RXD = DT_PROP_OR(DT_NODELABEL(uart0), rx_pin, -1),
.TXD = DT_PROP_OR(DT_NODELABEL(uart0), tx_pin, -1),
},
.baud_rate = DT_PROP_OR(DT_NODELABEL(uart0), current_speed, 115200),
.enabled = DT_NODE_HAS_STATUS(DT_NODELABEL(uart0), okay),
.flow_control = DT_PROP_OR(DT_NODELABEL(uart0), hw_flow_control, 0),

As a workaround, I can define the following macros in one of my headers:

#define DT_N_S_soc_S_uart_40002000_P_tx_pin 5
#define DT_N_S_soc_S_uart_40002000_P_tx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_rx_pin 20
#define DT_N_S_soc_S_uart_40002000_P_rx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_rts_pin 15
#define DT_N_S_soc_S_uart_40002000_P_rts_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40002000_P_cts_pin 7
#define DT_N_S_soc_S_uart_40002000_P_cts_pin_EXISTS 1

#define DT_N_S_soc_S_uart_40028000_P_tx_pin 17
#define DT_N_S_soc_S_uart_40028000_P_tx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_rx_pin 3
#define DT_N_S_soc_S_uart_40028000_P_rx_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_rts_pin 40
#define DT_N_S_soc_S_uart_40028000_P_rts_pin_EXISTS 1
#define DT_N_S_soc_S_uart_40028000_P_cts_pin 22
#define DT_N_S_soc_S_uart_40028000_P_cts_pin_EXISTS 1

Then all pins are configured properly and the connectivity_bridge sample works. Further investigation shows that the build script responsible for generating devicetree_unfixed.h does not generate any pin configuration. All CPUs are affected: non-modified samples and boards produce devicetree withoult pin configuration. At this state any stock sample or user project compiled with nRF Connect SDK v2.1.0 will not function properly.

This opens the question: Why is a completely broken SDK released in first place? This is not the first time Nordic releases an SDK incapable of building working code.

4. To make debugging even harder, the following other issue comes into play: a crash in the board controller prevent users from flashing nRF9160dk if any of the UARTs are open.  nRF9160dk onboard controller crash 

[jtrueb]

There is a need to break in order to fix and improve for the future of the RTOS and SDK; however, I agree the SDK software maturity levels don't match my expectations.

If the SDK has software maturity levels that are "end product" quality, then things like breaking pin configuration, removing nrfx layer functionality, or requiring significant migration burden need to be front and center in the documentatio prior to release. Yet, the emphasis cannot be only documentation; it must also focus on verification with NRF products, not just the general Zephyr tests for dev kits.

Take for example, pinctrl, which is the crux of your issue, several of my issues, and a number of others. It came with a migration guide in the SDK and an overview in the RTOS, I applaud the content created there as it is more usable than much of the older documentation with Zephyr. The problem lies in the gap between documentation, implementation, and test. Consider further the fallout of changing pin configuration like with serial interfaces like the TWIM drivers. Pinctrl broke the ability to use nrfx_twim through the nrfx interface. The cause is that the new configuration pattern can no longer generate the configuration and compile it. CONFIG_NRFX_TWIM0 can no longer be assigned without using a separate board dts (overlays are broken) and i2c_nrfx_twim doesn't compile with CONFIG_PINCTRL disabled.

Here is a snippet of code that shows that nrfx_twim was never compiled without pinctrl prior to releasing the SDK that broke usage of the ability to compile the nrfx_twim interface. This code is 3 months old. To my understanding, this means that many nrfx interfaces are not tested at all. This makes migrating legacy NRF SDK code to NRF Connect SDK either impossible or a complete re-development in my experience thus far.

I do not intend to discourage future changes. I want to emphasize and echo the struggles of a consumer of the SDK who isn't alone. I have experienced difficulty migrating release after release after release like no other beta software, but the documentation and tooling is getting better and the bugs less severe. The new developer academy and improved NRF Connect SDK website have been a welcome improvement as well.

Sorry this wasn't so helpful for your points 3 and 4.

[Georgi Valkov]

I created a new case for connectivity_bridge: malfunction due to improper use of system work queue
connectivity_bridge: malfunction due to improper use of system work queue 

jtrueb said:

If the SDK has software maturity levels that are "end product" quality, then things like breaking pin configuration, removing nrfx layer functionality, or requiring significant migration burden need to be front and center in the documentatio prior to release.

True. I started working on a project with nRF Connect SDK in 2018. Needless to say we lost the project a couple of years later. We faced an endless amount of serious issues. Put simply basic stuff does not work.

jtrueb said:

To my understanding, this means that many nrfx interfaces are not tested at all. This makes migrating legacy NRF SDK code to NRF Connect SDK either impossible or a complete re-development in my experience thus far.

My experience all shows that any change to the SDK requires a tremendous amount of work to port even simple applications. It took me two weeks to adapt to the new pinctrl and devicetree changes, and fix a serious flaw in connectivity bridge (link above).

Indeed testing is a very important part of development, and it is an even greater responsibility for SDKs to be robust. A wise man once said that a good API is easy to use correctly and hard to misuse. nRF Connect SDK is quite the opposite. Overcomplicated and resource demanding. My first attempt to use an UART in nRF Connect SDK resulted in 300 lines of complex code copied from an existing sample. Most users need a simple send and don't want to be bothered to service interrupts.

In connectivity_bridge, data travels through multiple hops from receive to send side. Each doing it's own buffer management. With my UART driver implementation we can receive and send in the same task without any risk of dropping data. The driver takes care of any background processing allowing the task to issue more send and receive requests.

jtrueb Thank you for your message and letting me know I am not alone. Would you be interested in testing nano RTOS? I recently wrote an light-weight operating system, which is very responsive. It is a new project of mine, and I had no time to create a web page yet.

[jtrueb]

I have also migrated a number of older projects from NRF SDK to NRF Connect SDK and NRF Connect SDK to NRF Connect SDK. I would estimate that on the average I also experienced similar schedule effort regarding the migration burden. Not mentioned yet is the known issues or regressions that I encountered during those upgrade attempts. For example, the HF clock power regression was immediately obvious on the power profiler. Another problematic one was the TX power level. I have become accustomed to scrolling through the Known Issues because so many of them are relevant to core functionality of an application. The PINCTRL issue earlier is a new known issue in 2.1.0.

jtrueb Thank you for your message and letting me know I am not alone. Would you be interested in testing nano RTOS? I recently wrote an light-weight operating system, which is very responsive. It is a new project of mine, and I had no time to create a web page yet.

I'm not ready to switch off of the NRF Connect SDK at the moment. I want an RTOS that can be certified similar to the NRF5 SDK. I want tools and build systems that work (not SEGGER). I want to debug my team's code, not someone else's. I want the other developers on my team to be able to use the same environment, toolchain and code to have reproducible builds and consistent bug reports. I want to have a path forward to using new products like the NRF53 series. I think Nordic + Zephyr RTOS can be all of that long term.

What I really want is for the NRF Connect SDK to stabilize. I am happy with the features that are already there (targeting the mature NRF52 series) and just want the sweeping changes and bugs to stop. I haven't been using any new SDK features for a while. I have to keep upgrading at the moment because the tooling keeps changing, stuff is not back-ported, and issues keep becoming blockers.

I wish that the SDK came with some commitment or compliance to quality that was enforced. Zephyr has an auditable LTS release train and is working towards MISRA compliance. Why is the NRF Connect SDK not aligned with the LTS release train for Zephyr? I am really hoping to stop all the upgrading and get an LTS of the SDK.

One thing about the issue you hit while upgrading was that you bumped a major version, so the LTS stuff is less relevant in this case. It sounds like you are more tolerant of release train and security issues since you are considering a new RTOS.

How are you handling CVEs? https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/security_chapter.html

There aren't any SDK references to vulnerabilities or hot fixes, etc. Many of the Zephyr CVEs are resolved like 'This has been fixed in main for v3.0.0' and backported (to a version that you can't install with the Toolchain Manager).

[jtrueb]

Sorry I don't know why the formatting is so bad there. It didn't look like that in preview.

[Georgi Valkov]

jtrueb 

Georgi Valkov said:

I'm not ready to switch off of the NRF Connect SDK at the moment. I want an RTOS that can be certified similar to the NRF5 SDK. I want tools and build systems that work (not SEGGER).

Alas I cannot offer certification, and the features are pretty basic at the moment. arm-none-eabi is used to build an .elf image.

jtrueb said:

It sounds like you are more tolerant of release train and security issues since you are considering a new RTOS.

Honestly, all I have for nRF52840 is a low latency UART driver with DMA. Whenever I need the BLE radio, I'm forced to use nRF Connect with my driver, because the original is too complicated, apart from reliability and latency issues.

Georgi Valkov said:

I have become accustomed to scrolling through the Known Issues because so many of them are relevant to core functionality of an application.

One issue I have with Zephyr is how overcomplicated everything is. This leads to mistakes. I started nano RTOS, because I needed simple API and concurrent tasks for a project based on Arduino. I do my best to consider security and reliability factors in my designs. And I follow the rule that API should be easy to use correctly and hard to use incorrectly. Many pitfalls arise from not knowing how to use a certain functionality.

jtrueb said:

How are you handling CVEs?

For Zephyr I have not used it professionally since we lost the project due to the many SDK issues. Nowadays I use it to run a few projects of mine as a hobby. At a later point I discovered a BLE exploit and got a bounty. I never imagined I would need to write an UART driver and make other changes to connectivity_bridge to get a stable link between UART and BLE.

helsing I'm having some issues writing this replay. Some quotes include my name instead of jtrueb. Also quotes appear three times. I get the impression this DevZone has been quite buggy in the past years. Do ask the web developers to have a look into that and try writing more reliable code in the future. Also note jtrueb complaining about formatting.