Related
Recommended

TensorFlow and nRF

MariaG
TensorFlow and nRF

Introduction

Machine learning(ML) is a still growing field, and with the TensorFlow Lite port, there is now support for machine learning on microcontrollers. TensorFlow is an end-to-end ML-platform owned by Google. We want to use TensorFlow Lite to implement support for nRF-chips, and demonstrate that it works for a simple example by TensorFlow. The chips nRF9160, nRF5340, and nRF52840 have all been tested for support.

The goal of this support is to expand the possible uses for nRF-microcontrollers. In addition, we hope that this project can get you started on your own ML application for embedded systems.

TensorFlow Lite Micro

On 9th of December 2019, microcontroller support for TensorFlow was moved out of the experimental folder. Multiple examples for different chips can be found in its git. However, these examples are contained inside the TensorFlow Library. Instead of using this method, this TensorFlow example will be built using NCS/Zephyr, because of the utility provided by these. This project is based on the External Library sample in Zephyr. The glue that connects the NCS project with the external library is the CMakeLists.txt file. Roughly, the CMakeLists.txt of the External Library example does three different things:

  1. Add local sources to the project. This does not affect the external library, and is done in all Zephyr projects:

target_sources(app PRIVATE src/main.c)

  1. Make the Makefile of the library to compile a statically linked library that can be imported. This statically linked library will have to be generated a bit differently to fit the core that will be used; Cortex-M33 for nRF9160 and nRF5340, and Cortex-M4 for the nRF52840 series:

zephyr_get_include_directories_for_lang_as_string(       C includes)
zephyr_get_system_include_directories_for_lang_as_string(C system_includes)
zephyr_get_compile_definitions_for_lang_as_string(       C definitions)
zephyr_get_compile_options_for_lang_as_string(           C options)
 
set(external_project_cflags "${includes} ${definitions} ${options} ${system_includes}")
 
include(ExternalProject)
# Add an external project to be able download and build the third
# party library. In this case downloading is not necessary as it has
# been committed to the repository.
set(mylib_src_dir   ${CMAKE_CURRENT_SOURCE_DIR}/mylib)
set(mylib_build_dir ${CMAKE_CURRENT_BINARY_DIR}/mylib)
set(MYLIB_LIB_DIR     ${mylib_build_dir}/lib)
set(MYLIB_INCLUDE_DIR ${mylib_src_dir}/include)
 
if(CMAKE_GENERATOR STREQUAL "Unix Makefiles")
# https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html
set(submake "$(MAKE)")
else() # Obviously no MAKEFLAGS. Let's hope a "make" can be found somewhere.
set(submake "make")
endif()
 
ExternalProject_Add(
  mylib_project                 # Name for custom target
  PREFIX     ${mylib_build_dir} # Root dir for entire project
  SOURCE_DIR ${mylib_src_dir}
  BINARY_DIR ${mylib_src_dir} # This particular build system is invoked from the root
  CONFIGURE_COMMAND ""    # Skip configuring the project, e.g. with autoconf
  BUILD_COMMAND
  ${submake}
  PREFIX=${mylib_build_dir}
  CC=${CMAKE_C_COMPILER}
  AR=${CMAKE_AR}
  CFLAGS=${external_project_cflags}
  INSTALL_COMMAND ""      # This particular build system has no install command
  BUILD_BYPRODUCTS ${MYLIB_LIB_DIR}/libmylib.a
  )

  1. Link the library with the project:

# Create a wrapper CMake library that our app can link with
add_library(mylib_lib STATIC IMPORTED GLOBAL)
add_dependencies(
  mylib_lib
  mylib_project
  )
set_target_properties(mylib_lib PROPERTIES IMPORTED_LOCATION             ${MYLIB_LIB_DIR}/libmylib.a)
set_target_properties(mylib_lib PROPERTIES INTERFACE_INCLUDE_DIRECTORIES ${MYLIB_INCLUDE_DIR})
 
target_link_libraries(app PUBLIC mylib_lib)


The final version of the CMakeLists.txt has a lot added, especially to step 2. With this, we managed to create a TensorFlow statically linked library file compatible with the Cortex-M33. Disclaimer: this CMakeLists.txt might have excess configurations. Step 2 above can be skipped if the library file already has been provided. A statically linked library file we generated is added to the git, and can be used for this purpose.
In addition to the CMakeLists.txt, some additions had to be made to the prj.conf file. This is mostly due to the fact that TensorFlow use C++, while NCS by default use C. In addition, floating point ABI had to be activated.

Hello world

The above method was used to successfully run TensorFlow Lite Micro’s Hello World example. This example will use a deep neural network to generate a sine wave. It is explained here. To show the result of this sine wave, a LED is dimmed using PWM.

Supported microcontrollers

The TensorFlow Hello World example was tested on the nRF9160DK and the nRF5340PDK. There is also a git branch with a sample that works with nRF52840DK. As mentioned above, the CMakeLists.txt was configured to work with an Arm Core. Because of this, it could be possible to use the same setup to run Zephyr with TensorFlow Lite Micro on other microcontrollers that use the same Arm Cores: Arm Cortex-M33 (nRF91 and nRF53) and Arm Cortex-M4 (nRF52). However, we tested only these three nRF microcontrollers.

Result

The project can be cloned from: https://github.com/oivoii/nrf-tensorflow

Disclaimer: This is a student project, and thus we can not guarantee that everything will work perfectly.