mjdietz2 gravatar image

Posted 2016-01-28 08:52:00 +0100

blogs->nordicers

Comparing compilers/IDEs for development with nRF5x series devices

Introduction

All compilers ARE NOT created equal. Given the same c code, different compilers will generate machine code with varying performance & size. Generally the more efficient machine code (uses less CPU cycles/machine instructions to perform the same high level functionality) will use less power as it completes processing faster and can enter low power states for longer. It will also enable more complex algorithms to run within your timing constraints. In resource constrained devices more compressed machine code can eliminate the need for external flash, or provide more 'room' to work with. It can also eliminate the need to spend time optimizing code (and sacrificing readability/maintainability).

This post will benchmark the different IDEs/compilers most used by Nordic customers. We will compare the Coremark scores of code compiled by IAR, Keil (ARMCC) and GCC running on an nRF52832 device. We will also compare the code size of a project compiled by each of these compilers.

Coremark is provided as a group of files testing common processing seen in the real world. These files are compiled (after core_portme.h,c have been modified for your device). The executable is run to obtain the Coremark score and the size of this executable is recorded. These are reported for each compiler in the results table below.

To understand Coremark quickly watch this 5 minute video: https://www.iar.com/support/resources/videos/how-to-run-your-own-coremarks/

Methodology

  1. Compile the Coremark project in each compiler optimizing for maximal speed/performance. Run the Coremark tests and record the results as well as the optimizations used.
  2. Compile an unmodified BLE example from Nordic's SDK (ble_app_hrs_s132_pca10040) using the exact same optimizations as in step 1 to get an understanding of the code size that corresponds with these Coremark results.
  3. Repeat steps 1 & 2 but this time optimize for minimum code size.

Coremark scores do not depend on external libraries. However libraries used do affect code size. When looking at the code size results please keep this in mind. The default IAR project settings in Nordic's SDK use the full configuration of the C/C++ runtime library (a smaller library is available). In Keil MicroLib is used and in GCC NewLib in nano version is used. These are the runtime libraries used in the code size benchmark below. It is likely that IAR can generate smaller executables by selecting smaller libraries in the project settings, but this comparison uses the same libraries that Nordic's SDK uses (and has been tested with).

Results

image description

image description

image description

image description

Important

An interesting result of these Coremark tests was the difference in score when the instruction cache is enabled. See https://infocenter.nordicsemi.com/topic/nrf52.v1.7/Chunk1983607998.html. With the cache enabled a 2.79 Coremark/MHz was acheived in Keil. Without the cache enabled a 2.20 Coremark/MHz was achieved in Keil. This was similar for IAR and GCC. So in your projects make sure you use this new feature on the nRF52 when appropriate! Also consider running code from Code RAM instead of FLASH for even better performance.

There were similar differences in Coremark scores generated by each compiler for the nRF51 series devices. If requested, Coremark scores for nRF51 can be posted in a future post.

Notes

  • These are not official results! I have validated them against multiple other employees who have ran Coremark profiles here. The official Coremark results we publish were generated in an earlier version of IAR so there are some very small differences.
  • The most up to date versions of Keil and GCC were used, but not IAR because I don't have that license.
  • The same code was used for each compiler with the exception of the startup assembler code and compiler specific system calls for SEGGER RTT (neither has any effect on Coremark results).
  • The stack and heap is not included in the reported RAM code size.
  • Keep your toolchain updated! I saw very big improvements in Coremark and code size between GCC 4.9.3 and GCC 5.2.1. There were also some improvements between ARMCC 5.05 and ARMCC 5.17 (RAM code size decreased).

Resources

All you need to do is start the external high frequency oscillator, enable the cache (for nRF52), and start and stop a timer. This is all done in core_portme.c. You will also need to redirect printf and define how many iterations you want this test to run for (I used 10,000). The only files you are to modify are core_port.h,c. Then just set up the compiler optimizations you want to use.

Future

Do you guys want to see anything else here? Do you want to see results for nRF51? What do you think of these results? Is Coremark a realistic/fair score for a compiler? Or would you like to see a BLE application profiled in each compiler for a more realistic comparison (maybe some compilers focus on excelling at this benchmark more than others?) Leave comments below.

Motivation

Average embedded development cycles are about a year with around 60% of them being finished "late or cancelled." 29% of this time is spent in detailed design and 21% is spent in debugging! And we all know time is $$$ and time to market is key! So you may want to rethink spending time optimizing code instead of just using the ideal tools for your project. The same goes for debugging - it is the 21st century, blinking LEDs & using UART to debug is unacceptable! (If this is you follow blogs and tutorials closely for the next month or so because we will be posting lots of easy to follow blogs/tutorials on this)!

The biggest challenges faced in the embedded design process are debugging and meeting deadlines. Most developers feel the debugging process is the greatest concern regarding development, and 71% of developers say the most important factor in choosing a processor is the software development tools available! WhatsGoingOnInEmbedded_Quinnell.pdf Our goal is to minimize these challenges for Nordic customers.

5 comments

endnode gravatar image

Posted Jan. 29, 2016, 2:23 p.m.

Hi Michael, great post. I've noticed that numbers changed since the original post yesterday, or is it just my bad memory? I've also thought you included SES LLVM compiler but I don't see the results now. Thx Jan

mjdietz2 gravatar image

Posted Jan. 29, 2016, 2:49 p.m.

Hey Jan, I have been getting different suggestions for compiler/linker flags to use etc... Also have had to update compilers and tweak methodology of this comparison so the results aren't misleading. I have removed LLVM (the version in segger embedded studio) for now because our SDK won't work with it when compiling ble examples.

This post may change slightly still but the numbers are converging. For example I may update IAR to its most recent version and post those numbers. Also looking into http://www.keil.com/support/man/docs/armcc/armcc_chr1359124934478.htm and an equivalent in IAR. Not sure if this is already happening or not when I compile since I am setting project settings in the IDE and not invoking the compiler manually from the command line.

If enough people are curious about LLVM I can benchmark that as well - but most likely a bit later when I turn this into a tutorial.

Thanks for the interest! -Mike

Posted Feb. 2, 2016, 12:20 p.m.

Great info.I like all your post.I will keep visiting this blog very often.It is good to see you verbalize from the heart and your clarity on this important subject can be easily observed. Thanks again! If you love gaming for unity games kindly visit….
http://unanimousstudios.com/marketplace

gordonlukch gravatar image

Posted Feb. 12, 2016, 7:40 a.m.

Dear Michael, Nice comparing and tt should be great If you could post nRF51 result as well. Thank you.

JaccoS gravatar image

Posted June 9, 2017, 10:51 a.m.

Hi Michael, Great post! Would it be possible to update the post with the numbers for the current versions of the compilers?

Sign in to comment.

Related posts by tag