Comparing compilers/IDEs for development with nRF5x series devices

#Introduction All compilers ARE NOT created equal. Given the same c code, different compilers will generate machine code with varying performance & size. Generally the more efficient machine code (uses less CPU cycles/machine instructions to perform the same high level functionality) will use less power as it completes processing faster and can enter low power states for longer. It will also enable more complex algorithms to run within your timing constraints. In resource constrained devices more compressed machine code can eliminate the need for external flash, or provide more 'room' to work with. It can also eliminate the need to spend time optimizing code (and sacrificing readability/maintainability).

This post will benchmark the different IDEs/compilers most used by Nordic customers. We will compare the Coremark scores of code compiled by IAR, Keil (ARMCC) and GCC running on an nRF52832 device. We will also compare the code size of a project compiled by each of these compilers.

Coremark is provided as a group of files testing common processing seen in the real world. These files are compiled (after core_portme.h,c have been modified for your device). The executable is run to obtain the Coremark score and the size of this executable is recorded. These are reported for each compiler in the results table below.

To understand Coremark quickly watch this 5 minute video: https://www.iar.com/support/resources/videos/how-to-run-your-own-coremarks/

#Methodology

1. Compile the Coremark project in each compiler optimizing for maximal speed/performance. Run the Coremark tests and record the results as well as the optimizations used.
2. Compile an unmodified BLE example from Nordic's SDK (ble_app_hrs_s132_pca10040) using the exact same optimizations as in step 1 to get an understanding of the code size that corresponds with these Coremark results.
3. Repeat steps 1 & 2 but this time optimize for minimum code size.

Coremark scores do not depend on external libraries. However libraries used do affect code size. When looking at the code size results please keep this in mind. The default IAR project settings in Nordic's SDK use the full configuration of the C/C++ runtime library (a smaller library is available). In Keil MicroLib is used and in GCC NewLib in nano version is used. These are the runtime libraries used in the code size benchmark below. It is likely that IAR can generate smaller executables by selecting smaller libraries in the project settings, but this comparison uses the same libraries that Nordic's SDK uses (and has been tested with).

#Results

#Important An interesting result of these Coremark tests was the difference in score when the instruction cache is enabled. See https://infocenter.nordicsemi.com/topic/nrf52.v1.7/Chunk1983607998.html. With the cache enabled a 2.79 Coremark/MHz was acheived in Keil. Without the cache enabled a 2.20 Coremark/MHz was achieved in Keil. This was similar for IAR and GCC. So in your projects make sure you use this new feature on the nRF52 when appropriate! Also consider running code from Code RAM instead of FLASH for even better performance.

There were similar differences in Coremark scores generated by each compiler for the nRF51 series devices. If requested, Coremark scores for nRF51 can be posted in a future post.

#Notes

• These are not official results! I have validated them against multiple other employees who have ran Coremark profiles here. The official Coremark results we publish were generated in an earlier version of IAR so there are some very small differences.
• The most up to date versions of Keil and GCC were used, but not IAR because I don't have that license.
• The same code was used for each compiler with the exception of the startup assembler code and compiler specific system calls for SEGGER RTT (neither has any effect on Coremark results).
• The stack and heap is not included in the reported RAM code size.
• Keep your toolchain updated! I saw very big improvements in Coremark and code size between GCC 4.9.3 and GCC 5.2.1. There were also some improvements between ARMCC 5.05 and ARMCC 5.17 (RAM code size decreased).

#Resources

• http://infocenter.arm.com/help/topic/com.arm.doc.dai0350a/DAI0350A_coremark_benchmarking.pdf
• You can run Coremark as well. Simply register and download Coremark files here: http://www.eembc.org/coremark/download.php.

All you need to do is start the external high frequency oscillator, enable the cache (for nRF52), and start and stop a timer. This is all done in core_portme.c. You will also need to redirect printf and define how many iterations you want this test to run for (I used 10,000). The only files you are to modify are core_port.h,c. Then just set up the compiler optimizations you want to use.

#Future Do you guys want to see anything else here? Do you want to see results for nRF51? What do you think of these results? Is Coremark a realistic/fair score for a compiler? Or would you like to see a BLE application profiled in each compiler for a more realistic comparison (maybe some compilers focus on excelling at this benchmark more than others?) Leave comments below.

#Motivation Average embedded development cycles are about a year with around 60% of them being finished "late or cancelled." 29% of this time is spent in detailed design and 21% is spent in debugging! And we all know time is $$$ and time to market is key! So you may want to rethink spending time optimizing code instead of just using the ideal tools for your project. The same goes for debugging - it is the 21st century, blinking LEDs & using UART to debug is unacceptable! (If this is you follow blogs and tutorials closely for the next month or so because we will be posting lots of easy to follow blogs/tutorials on this)!

The biggest challenges faced in the embedded design process are debugging and meeting deadlines. Most developers feel the debugging process is the greatest concern regarding development, and 71% of developers say the most important factor in choosing a processor is the software development tools available! WhatsGoingOnInEmbedded_Quinnell.pdf Our goal is to minimize these challenges for Nordic customers.