Introduction
Machine learning(ML) is a still growing field, and with the TensorFlow Lite port, there is now support for machine learning on microcontrollers. TensorFlow is an end-to-end ML-platform owned by Google. We want to use TensorFlow Lite to implement support for nRF-chips, and demonstrate that it works for a simple example by TensorFlow. The chips nRF9160, nRF5340, and nRF52840 have all been tested for support.
The goal of this support is to expand the possible uses for nRF-microcontrollers. In addition, we hope that this project can get you started on your own ML application for embedded systems.
TensorFlow Lite Micro
On 9th of December 2019, microcontroller support for TensorFlow was moved out of the experimental folder. Multiple examples for different chips can be found in its git. However, these examples are contained inside the TensorFlow Library. Instead of using this method, this TensorFlow example will be built using NCS/Zephyr, because of the utility provided by these. This project is based on the External Library sample in Zephyr. The glue that connects the NCS project with the external library is the CMakeLists.txt file. Roughly, the CMakeLists.txt of the External Library example does three different things:
target_sources(app PRIVATE src/main.c)
zephyr_get_include_directories_for_lang_as_string( C includes) zephyr_get_system_include_directories_for_lang_as_string(C system_includes) zephyr_get_compile_definitions_for_lang_as_string( C definitions) zephyr_get_compile_options_for_lang_as_string( C options) set(external_project_cflags "${includes} ${definitions} ${options} ${system_includes}") include(ExternalProject) # Add an external project to be able download and build the third # party library. In this case downloading is not necessary as it has # been committed to the repository. set(mylib_src_dir ${CMAKE_CURRENT_SOURCE_DIR}/mylib) set(mylib_build_dir ${CMAKE_CURRENT_BINARY_DIR}/mylib) set(MYLIB_LIB_DIR ${mylib_build_dir}/lib) set(MYLIB_INCLUDE_DIR ${mylib_src_dir}/include) if(CMAKE_GENERATOR STREQUAL "Unix Makefiles") # https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html set(submake "$(MAKE)") else() # Obviously no MAKEFLAGS. Let's hope a "make" can be found somewhere. set(submake "make") endif() ExternalProject_Add( mylib_project # Name for custom target PREFIX ${mylib_build_dir} # Root dir for entire project SOURCE_DIR ${mylib_src_dir} BINARY_DIR ${mylib_src_dir} # This particular build system is invoked from the root CONFIGURE_COMMAND "" # Skip configuring the project, e.g. with autoconf BUILD_COMMAND ${submake} PREFIX=${mylib_build_dir} CC=${CMAKE_C_COMPILER} AR=${CMAKE_AR} CFLAGS=${external_project_cflags} INSTALL_COMMAND "" # This particular build system has no install command BUILD_BYPRODUCTS ${MYLIB_LIB_DIR}/libmylib.a )
# Create a wrapper CMake library that our app can link with add_library(mylib_lib STATIC IMPORTED GLOBAL) add_dependencies( mylib_lib mylib_project ) set_target_properties(mylib_lib PROPERTIES IMPORTED_LOCATION ${MYLIB_LIB_DIR}/libmylib.a) set_target_properties(mylib_lib PROPERTIES INTERFACE_INCLUDE_DIRECTORIES ${MYLIB_INCLUDE_DIR}) target_link_libraries(app PUBLIC mylib_lib)
The final version of the CMakeLists.txt has a lot added, especially to step 2. With this, we managed to create a TensorFlow statically linked library file compatible with the Cortex-M33. Disclaimer: this CMakeLists.txt might have excess configurations. Step 2 above can be skipped if the library file already has been provided. A statically linked library file we generated is added to the git, and can be used for this purpose. In addition to the CMakeLists.txt, some additions had to be made to the prj.conf file. This is mostly due to the fact that TensorFlow use C++, while NCS by default use C. In addition, floating point ABI had to be activated.
Hello world
The above method was used to successfully run TensorFlow Lite Micro’s Hello World example. This example will use a deep neural network to generate a sine wave. It is explained here. To show the result of this sine wave, a LED is dimmed using PWM.
Supported microcontrollers
The TensorFlow Hello World example was tested on the nRF9160DK and the nRF5340PDK. There is also a git branch with a sample that works with nRF52840DK. As mentioned above, the CMakeLists.txt was configured to work with an Arm Core. Because of this, it could be possible to use the same setup to run Zephyr with TensorFlow Lite Micro on other microcontrollers that use the same Arm Cores: Arm Cortex-M33 (nRF91 and nRF53) and Arm Cortex-M4 (nRF52). However, we tested only these three nRF microcontrollers.
Result
The project can be cloned from: https://github.com/oivoii/nrf-tensorflow
Disclaimer: This is a student project, and thus we can not guarantee that everything will work perfectly.
is possible cooperation with Google coral?
Hi! How would I go about implementing this example in SEGGER ES? I already have a project to which I would like to add TFLite for microcontrollers functionality!
Hi CheMax!As we are student employees, we do not know if this will be added to SDK. Please contact your sales manager if you have more questions. For NCS, we might push the project as an example to be reviewed. Note that this is not a guarantee for it to be added to NCS.
This looks interesting. Is Nordic planning to add these examples to SDK?