All compilers ARE NOT created equal. Given the same c code, different compilers will generate machine code with varying performance & size. Generally the more efficient machine code (uses less CPU cycles/machine instructions to perform the same high level functionality) will use less power as it completes processing faster and can enter low power states for longer. It will also enable more complex algorithms to run within your timing constraints. In resource constrained devices more compressed machine code can eliminate the need for external flash, or provide more 'room' to work with. It can also eliminate the need to spend time optimizing code (and sacrificing readability/maintainability).
This post will benchmark the different IDEs/compilers most used by Nordic customers. We will compare the Coremark scores of code compiled by IAR, Keil (ARMCC) and GCC running on an nRF52832 device. We will also compare the code size of a project compiled by each of these compilers.
Coremark is provided as a group of files testing common processing seen in the real world. These files are compiled (after core_portme.h,c have been modified for your device). The executable is run to obtain the Coremark score and the size of this executable is recorded. These are reported for each compiler in the results table below.
To understand Coremark quickly watch this 5 minute video: https://www.iar.com/support/resources/videos/how-to-run-your-own-coremarks/
Coremark scores do not depend on external libraries. However libraries used do affect code size. When looking at the code size results please keep this in mind. The default IAR project settings in Nordic's SDK use the full configuration of the C/C++ runtime library (a smaller library is available). In Keil MicroLib is used and in GCC NewLib in nano version is used. These are the runtime libraries used in the code size benchmark below. It is likely that IAR can generate smaller executables by selecting smaller libraries in the project settings, but this comparison uses the same libraries that Nordic's SDK uses (and has been tested with).
An interesting result of these Coremark tests was the difference in score when the instruction cache is enabled. See https://infocenter.nordicsemi.com/topic/nrf52.v1.7/Chunk1983607998.html. With the cache enabled a 2.79 Coremark/MHz was acheived in Keil. Without the cache enabled a 2.20 Coremark/MHz was achieved in Keil. This was similar for IAR and GCC. So in your projects make sure you use this new feature on the nRF52 when appropriate! Also consider running code from Code RAM instead of FLASH for even better performance.
There were similar differences in Coremark scores generated by each compiler for the nRF51 series devices. If requested, Coremark scores for nRF51 can be posted in a future post.
All you need to do is start the external high frequency oscillator, enable the cache (for nRF52), and start and stop a timer. This is all done in core_portme.c. You will also need to redirect printf and define how many iterations you want this test to run for (I used 10,000). The only files you are to modify are core_port.h,c. Then just set up the compiler optimizations you want to use.
Do you guys want to see anything else here? Do you want to see results for nRF51? What do you think of these results? Is Coremark a realistic/fair score for a compiler? Or would you like to see a BLE application profiled in each compiler for a more realistic comparison (maybe some compilers focus on excelling at this benchmark more than others?) Leave comments below.
Average embedded development cycles are about a year with around 60% of them being finished "late or cancelled." 29% of this time is spent in detailed design and 21% is spent in debugging! And we all know time is $$$ and time to market is key! So you may want to rethink spending time optimizing code instead of just using the ideal tools for your project. The same goes for debugging - it is the 21st century, blinking LEDs & using UART to debug is unacceptable! (If this is you follow blogs and tutorials closely for the next month or so because we will be posting lots of easy to follow blogs/tutorials on this)!
The biggest challenges faced in the embedded design process are debugging and meeting deadlines. Most developers feel the debugging process is the greatest concern regarding development, and 71% of developers say the most important factor in choosing a processor is the software development tools available! WhatsGoingOnInEmbedded_Quinnell.pdf Our goal is to minimize these challenges for Nordic customers.
Great post! Would it be possible to update the post with the numbers for the current versions of the compilers?
Nice comparing and tt should be great If you could post nRF51 result as well. Thank you.
Hey Jan, I have been getting different suggestions for compiler/linker flags to use etc... Also have had to update compilers and tweak methodology of this comparison so the results aren't misleading. I have removed LLVM (the version in segger embedded studio) for now because our SDK won't work with it when compiling ble examples.
This post may change slightly still but the numbers are converging. For example I may update IAR to its most recent version and post those numbers. Also looking into www.keil.com/.../armcc_chr1359124934478.htm and an equivalent in IAR. Not sure if this is already happening or not when I compile since I am setting project settings in the IDE and not invoking the compiler manually from the command line.
If enough people are curious about LLVM I can benchmark that as well - but most likely a bit later when I turn this into a tutorial.
Thanks for the interest!
Hi Michael, great post. I've noticed that numbers changed since the original post yesterday, or is it just my bad memory? I've also thought you included SES LLVM compiler but I don't see the results now. Thx Jan