NCS v2.4.0 now breaks c++ compilation

[adamfowleruk]

Hi all,

After a terrible time setting up NCS v2.4.0 (the Nordic Connect tool on Mac does NOT set the environment up properly at all currently, by the way, so I've spent 4 hours trying to set that up properly) I've found that something within the Nordic toolset is downgrading the C++ standard used to -std=c++11. You can see this from the below:-

[build] FAILED: CMakeFiles/app.dir/REDACTED/herald-for-cpp/herald/src/datatype/base64_string.cpp.obj 
[build] ccache /opt/nordic/ncs/toolchains/4ef6631da0/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-g++ -DCONFIG_APP_LOG_LEVEL=4 -DHERALD_LOG_LEVEL=4 -DHERALD_MEMORYARENA_MAX=4096 -DKERNEL -DNRF52832_XXAA -DUSE_PARTITION_MANAGER=0 -D_ANSI_SOURCE -D_FORTIFY_SOURCE=1 -D__LINUX_ERRNO_EXTENSIONS__ -D__PROGRAM_START -D__ZEPHYR__=1 -I/REDACTED/herald-for-cpp/herald-venue-beacon/../herald/include -I/opt/nordic/ncs/v2.4.0/nrf/drivers/mpsl/clock_control -I/opt/nordic/ncs/v2.4.0/zephyr/include -I/REDACTED/herald-for-cpp/herald-venue-beacon/build/zephyr/include/generated -I/opt/nordic/ncs/v2.4.0/zephyr/soc/arm/nordic_nrf/nrf52 -I/opt/nordic/ncs/v2.4.0/zephyr/lib/libc/newlib/include -I/opt/nordic/ncs/v2.4.0/zephyr/soc/arm/nordic_nrf/common/. -I/opt/nordic/ncs/v2.4.0/zephyr/subsys/bluetooth -I/opt/nordic/ncs/v2.4.0/zephyr/subsys/settings/include -I/opt/nordic/ncs/v2.4.0/nrf/include -I/opt/nordic/ncs/v2.4.0/nrf/lib/multithreading_lock/. -I/opt/nordic/ncs/v2.4.0/nrf/subsys/bluetooth/controller/. -I/opt/nordic/ncs/v2.4.0/nrf/tests/include -I/opt/nordic/ncs/v2.4.0/modules/hal/cmsis/CMSIS/Core/Include -I/opt/nordic/ncs/v2.4.0/modules/hal/nordic/nrfx -I/opt/nordic/ncs/v2.4.0/modules/hal/nordic/nrfx/drivers/include -I/opt/nordic/ncs/v2.4.0/modules/hal/nordic/nrfx/mdk -I/opt/nordic/ncs/v2.4.0/zephyr/modules/hal_nordic/nrfx/. -I/opt/nordic/ncs/v2.4.0/modules/debug/segger/SEGGER -I/opt/nordic/ncs/v2.4.0/modules/debug/segger/Config -I/opt/nordic/ncs/v2.4.0/zephyr/modules/segger/. -I/opt/nordic/ncs/v2.4.0/modules/crypto/tinycrypt/lib/include -I/opt/nordic/ncs/v2.4.0/nrfxlib/mpsl/include -I/opt/nordic/ncs/v2.4.0/nrfxlib/mpsl/include/protocol -I/opt/nordic/ncs/v2.4.0/nrfxlib/softdevice_controller/include -g -std=gnu++17 -fno-strict-aliasing -Os -fcheck-new -std=c++11 -fno-exceptions -fno-rtti -imacros /REDACTED/herald-for-cpp/herald-venue-beacon/build/zephyr/include/generated/autoconf.h -fno-common -g -gdwarf-4 -fdiagnostics-color=always -mcpu=cortex-m4 -mthumb -mabi=aapcs --sysroot=/opt/nordic/ncs/toolchains/4ef6631da0/opt/zephyr-sdk/arm-zephyr-eabi/arm-zephyr-eabi -imacros /opt/nordic/ncs/v2.4.0/zephyr/include/zephyr/toolchain/zephyr_stdint.h -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=/REDACTED/herald-for-cpp/herald-venue-beacon=CMAKE_SOURCE_DIR -fmacro-prefix-map=/opt/nordic/ncs/v2.4.0/zephyr=ZEPHYR_BASE -fmacro-prefix-map=/opt/nordic/ncs/v2.4.0=WEST_TOPDIR -ffunction-sections -fdata-sections -specs=nano.specs -MD -MT CMakeFiles/app.dir/REDACTED/herald-for-cpp/herald/src/datatype/base64_string.cpp.obj -MF CMakeFiles/app.dir/REDACTED/herald-for-cpp/herald/src/datatype/base64_string.cpp.obj.d -o CMakeFiles/app.dir/REDACTED/herald-for-cpp/herald/src/datatype/base64_string.cpp.obj -c /REDACTED/herald-for-cpp/herald/src/datatype/base64_string.cpp

I suspect this is what has also caused this issue reported by someone else recently:  nRFConnect Examples fail to build with C++17

Although my CMake script is specifying C++17 (not GNU), as you can see from the above the Nordic tools are somehow rewriting this to -std=gnuc++17 and -std=c++11 at the same time. (Even though I'm originally directing CMake to use -stc=c++17 via the usual CMAKE vars). This causes my build to fail as I use c++17 code in Herald.I'm doing this on a mac at the moment but I doubt running this from elsewhere will change anything.

My CMake script hasn't changed in over 2 years so I'm confident it's an SDK issue.

Not a great user experience though. I have to waste a full day every time I upgrade NCS. Routinely. Like clockwork. Every. Single. Time. If this happens again I'm going to just drop the Nordic SDK and try just OSS Zephyr.

I need to get this working as our COVID-19 digital contact tracing library (Herald) needs to be programmed onto 150 nRF52832 based devices before August so we can conduct tests and trials. Not to mention other apps we have around health monitoring wearables and bluetooth mesh devices for potential use in hospitals.

I'm going to try downgrading to v2.2.0 then further back as I need to go. Please though fix this in the Nordic SDK and include a C++17 example in your test CI scripts before an SDK is released. This should have been caught in QA.

Thanks.

[Sigurd Hellesvik]

Hi,

Which version are you updating from?

Instead of setting the c++ version from CMake, I suggest that you use CONFIG_STD_CPP17.
According to our developers, using CMake directly for this option is not guaranteed to work in our build system.

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/develop/languages/cpp/index.html

I will forward your feedback about how time-consuming it is to update NCS version + update your application for the new version to our User Experience team.

Regards,
Sigurd Hellesvik

[adamfowleruk]

Thanks. I ended up just downloading vanilla Zephyr 3.4.0. I did indeed end up needing to use the STD_APP17 option, but this was easier to diagnose in Zephyr. It was also easier to set up a new install and repeatable build environment in Zephyr than in NCS. I was up and running in an hour.

I was using the latest NCS from the toolchain manager, 2.4.0, in my original attempt.

[Sigurd Hellesvik]

Good to hear that you made it work on Zephyr!

If you use Zephyr, we will be a bit limited in how we can help you, and you loose out on some of our NCS specific features.
However, if you feel like the reduced maintenance is a good tradeoff, that is of course up to you.
You can still make tickets here and ask for help, and we in technical support will try to help you as best we can.
Zephyr also have a discord server which you can use for Zephyr specific support, and they are skilled there as well.

I will fish for some more specific feedback.
If you do not feel like spending time answering, that is not problem
But I figure asking is free:

adamfowleruk said:

but this was easier to diagnose in Zephyr

How did Zephyr make this easier to diagnose?
Maybe knowing what they do better here can help us improve.

adamfowleruk said:

It was also easier to set up a new install and repeatable build environment in Zephyr than in NCS. I was up and running in an hour.

If there was anything specific here, i would be interested as well.
When you say "repeatable build environment", do you mean like Github CI?

Regards,
Sigurd Hellesvik

[adamfowleruk]

Great questions. I cannot recall the specific build error message that made the C++ standard fix more obvious, but I recall googling that build message pointed me to this flag. It failed on a different module (not an NCS module) so it may be that this particular module just had better reporting for some reason. Or I just got lucky with google.

As for the NCS install it's my machine, not CI. I had the toolchain manager just hang mid-download, necessitating a force quit, remove, and reinstall of the selected SDK version. I had set this off over lunch so wasn't impressed when I came back and it had hanged.

Then in the VSCode extension, specifying the nrf directives for NCS version 2.4.0 seemed to not set up the env properly as it used to. Specifically, none of the CMake scripts were in my user cmake folder so it looked like the zephyr-export wasn't executed properly.

The links to zephyr SDK were also not made. I noticed that v2.4.0 didn't create a /opt/local/nordic/toolchain/v2.4.0 folder but instead it has a very odd, seemingly random ID. Perhaps this was to blame?

I had to faff around a lot on the command line to preset a lot more info than in previous releases, mainly to work around the above, then launch VSCode from this terminal. I would expect the combination of the Toolchain Manager and NRF Connect VS Code extension to work, especially given I clicked on the 'Launch VS Code' button within the toolchain manager originally.

Also, in Zephyr's install guide you explicitly set up a new Python virtual env for the install of west and dependencies. I've never seen an option for that in NCS. Given the amount of other envs I have and data science work I do, this is a real concern.

It's a shame because NCS was pretty flawless about 18 months ago but it just feels fragile and unpredictable today.

I need it to be easy and predictable because if a mere muggle that picks up the Herald project's repo wanted to modify an app (E.g. for a health monitoring wearable to add, say, amovement sensor data which is a pretty trivial change) I don't want them to fall off an edge of complexity because a minor version upgrade of NCS doesn't work predictably out of the box.

As for reduced maintenance tradeoffs, I'm still using a lot of Nordic chips and Nordic DKs. (And saying nice things on Twitter tagging you in!) but I'm only really using vanilla Zephyr features. This is because it's possible adopters of Herald want to use us on non-Nordic devices that Zephyr supports. (The heathens!!!) The only Nordic specific thing I was doing on some debug builds was using the bluetooth serial feature for debug of certain devices. (We have a weird set of devices we've inherited, as well as our own ones we're designing and testing).

I'll also be wanting to acquire the new nRF54 chips / eval boards / DKs when they become available too. I suspect that chip will have PSA certification which we want for a potential future 'production/medical grade' opensource wearable base board design... Also because I'm basically a Nordic DK collector now. LOL.

[Sigurd Hellesvik]

This is good and constructive critique, thanks for taking the time!
I will make sure it get to by someone who work with this.
I can of course not promise any fixes, as it is subject to prioritization.
However, we strive to improve, so there is that.

adamfowleruk said:

As for reduced maintenance tradeoffs, I'm still using a lot of Nordic chips and Nordic DKs. (And saying nice things on Twitter tagging you in!) but I'm only really using vanilla Zephyr features. This is because it's possible adopters of Herald want to use us on non-Nordic devices that Zephyr supports. (The heathens!!!) The only Nordic specific thing I was doing on some debug builds was using the bluetooth serial feature for debug of certain devices. (We have a weird set of devices we've inherited, as well as our own ones we're designing and testing).

Makes sense to me
Since Zephyr is closely intertwined with the NCS, we have competence in that as well, so we likely can help you with issues here as well.

adamfowleruk said:

As for the NCS install it's my machine, not CI. I had the toolchain manager just hang mid-download, necessitating a force quit, remove, and reinstall of the selected SDK version. I had set this off over lunch so wasn't impressed when I came back and it had hanged.

Then in the VSCode extension, specifying the nrf directives for NCS version 2.4.0 seemed to not set up the env properly as it used to. Specifically, none of the CMake scripts were in my user cmake folder so it looked like the zephyr-export wasn't executed properly.

The links to zephyr SDK were also not made. I noticed that v2.4.0 didn't create a /opt/local/nordic/toolchain/v2.4.0 folder but instead it has a very odd, seemingly random ID. Perhaps this was to blame?

The Toolchain manager should work reliably, it is one of the major points of having a GUI after all. So the next point is not an excuse.
If you need an alternative if you decide to go back to NCS at a later point, you can always try the Manual Install.
The downside of this is that your toolchain (zephyr-sdk, python versions etc) may not match completely.
However, this is the method I use when working with the NCS, and I rarely have issues with these things.
On the other hand, I am biased as I am a user of linux and vim and so on.

adamfowleruk said:

I had to faff around a lot on the command line to preset a lot more info than in previous releases, mainly to work around the above, then launch VSCode from this terminal. I would expect the combination of the Toolchain Manager and NRF Connect VS Code extension to work, especially given I clicked on the 'Launch VS Code' button within the toolchain manager originally.

The VS Code extension should also work reliably, so no excuse. Just telling you in case you want alternatives.
I assume that you do not get such a IDE when developing with vanilla zephyr?
Instead you use west in the command line to build etc.
You do not have to use the the VS Code extension for the NCS if you do not want to.
It gives you some nice bonus features to use, but you can use VS Code as just a text editor and then build+flash from the terminal.
Again, I am biased since I do not use VS Code, but vim so take this with a pinch of salt.

adamfowleruk said:

Also, in Zephyr's install guide you explicitly set up a new Python virtual env for the install of west and dependencies. I've never seen an option for that in NCS. Given the amount of other envs I have and data science work I do, this is a real concern.

If not explicitly recommended for the NCS, I at least think virtual env should be suggested in our manual installation guide.
I will suggest this to our documentation team at once.

adamfowleruk said:

This is because it's possible adopters of Herald want to use us on non-Nordic devices that Zephyr supports. (The heathens!!!)

How dare they! But since they dare, I can understand the limitation as our license says that you must use nrf chips: https://github.com/nrfconnect/sdk-nrf/blob/main/LICENSE. It would only apply to the Nordic parts of the NCS I think, but with this you must thread a bit more carefully yes. Since we are talking about licenses, I just want to point out that I am not a lawyer and you should not use me for legal advice.

adamfowleruk said:

I'll also be wanting to acquire the new nRF54 chips / eval boards / DKs when they become available too. I suspect that chip will have PSA certification which we want for a potential future 'production/medical grade' opensource wearable base board design...

I can not tell you much about the nRF54 yet unfortunately.

However, for PSA Certification, I think that it will be a lot easier with the nRF Connect SDK instead of Zephyr. As far as I know, a lot of our TF-M features are nRF Connect SDK specific. If you want me to figure that out with more certainty, let me know, and can try to track down limitations.

adamfowleruk said:

Also because I'm basically a Nordic DK collector now. LOL.

Gotta catch them all!