Compute Library
 20.02.1
Introduction

Table of Contents

The Computer Vision and Machine Learning library is a set of functions optimised for both ARM CPUs and GPUs using SIMD technologies.Several builds of the library are available using various configurations:

  • OS: Linux, Android or bare metal.
  • Architecture: armv7a (32bit) or arm64-v8a (64bit)
  • Technology: NEON / OpenCL / GLES_COMPUTE / NEON and OpenCL and GLES_COMPUTE
  • Debug / Asserts / Release: Use a build with asserts enabled to debug your application and enable extra validation. Once you are sure your application works as expected you can switch to a release build of the library for maximum performance.

Contact / Support

Please email devel.nosp@m.oper.nosp@m.@arm..nosp@m.com

In order to facilitate the work of the support team please provide the build information of the library you are using. To get the version of the library you are using simply run: 

$ strings android-armv7a-cl-asserts/libarm_compute.so | grep arm_compute_version
arm_compute_version=v16.12 Build options: {'embed_kernels': '1', 'opencl': '1', 'arch': 'armv7a', 'neon': '0', 'asserts': '1', 'debug': '0', 'os': 'android', 'Werror': '1'} Git hash=f51a545d4ea12a9059fe4e598a092f1fd06dc858

Pre-built binaries

For each release we provide some pre-built binaries of the library here

These binaries have been built using the following toolchains:

  • Linux armv7a: gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf
  • Linux arm64-v8a: gcc-linaro-4.9-2016.02-x86_64_aarch64-linux-gnu
  • Android armv7a: clang++ / libc++ NDK r17c
  • Android am64-v8a: clang++ / libc++ NDK r17c
Warning
Make sure to use a compatible toolchain to build your application or you will get some std::bad_alloc errors at runtime.

File organisation

This archive contains:

  • The arm_compute header and source files
  • The latest Khronos OpenCL 1.2 C headers from the Khronos OpenCL registry
  • The latest Khronos cl2.hpp from the Khronos OpenCL registry (API version 2.1 when this document was written)
  • The latest Khronos OpenGL ES 3.1 C headers from the Khronos OpenGL ES registry
  • The latest Khronos EGL 1.5 C headers from the Khronos EGL registry
  • The sources for a stub version of libOpenCL.so, libGLESv1_CM.so, libGLESv2.so and libEGL.so to help you build your application.
  • An examples folder containing a few examples to compile and link against the library.
  • A utils folder containing headers with some boiler plate code used by the examples.
  • This documentation.

You should have the following file organisation: 

.
├── arm_compute --> All the arm_compute headers
│   ├── graph.h --> Includes all the Graph headers at once.
│   ├── core
│   │   ├── CL
│   │   │   ├── CLCoreRuntimeContext.h --> Manages all core OpenCL objects needed for kernel execution (cl_context, cl_kernel, cl_command_queue, etc).
│   │   │   ├── CLKernelLibrary.h --> Manages all the OpenCL kernels compilation and caching, provides accessors for the OpenCL Context.
│   │   │   ├── CLKernels.h --> Includes all the OpenCL kernels at once
│   │   │   ├── CL specialisation of all the generic objects interfaces (ICLTensor, ICLArray, etc.)
│   │   │   ├── kernels --> Folder containing all the OpenCL kernels
│   │   │   │   └── CL*Kernel.h
│   │   │   └── OpenCL.h --> Wrapper to configure the Khronos OpenCL C++ header
│   │   ├── CPP
│   │   │   ├── CPPKernels.h --> Includes all the CPP kernels at once
│   │   │   └── kernels --> Folder containing all the CPP kernels
│   │   │       └── CPP*Kernel.h
│   │   ├── GLES_COMPUTE
│   │   │   ├── GCCoreRuntimeContext.h --> Manages all core GLES objects needed for kernel execution.
│   │   │   ├── GCKernelLibrary.h --> Manages all the GLES kernels compilation and caching, provides accessors for the GLES Context.
│   │   │   ├── GCKernels.h --> Includes all the GLES kernels at once
│   │   │   ├── GLES specialisation of all the generic objects interfaces (IGCTensor etc.)
│   │   │   ├── kernels --> Folder containing all the GLES kernels
│   │   │   │   └── GC*Kernel.h
│   │   │   └── OpenGLES.h --> Wrapper to configure the Khronos EGL and OpenGL ES C header
│   │   ├── NEON
│   │   │   ├── kernels --> Folder containing all the NEON kernels
│   │   │   │   ├── assembly --> headers for assembly optimised NEON kernels.
│   │   │   │   ├── convolution --> headers for convolution assembly optimised NEON kernels.
│   │   │   │   │   ├── common --> headers for code which is common to several convolution implementations.
│   │   │   │   │   ├── depthwise --> headers for Depthwise convolultion assembly implementation
│   │   │   │   │   └── winograd --> headers for Winograd convolution assembly implementation
│   │   │   │   ├── detail --> Common code for several intrinsics implementations.
│   │   │   │   └── NE*Kernel.h
│   │   │   ├── wrapper --> NEON wrapper used to simplify code
│   │   │   │   ├── intrinsics --> NEON instrincs' wrappers
│   │   │   │   ├── scalar --> Scalar operations
│   │   │   │   ├── traits.h --> Traits defined on NEON vectors
│   │   │   │   └── wrapper.h --> Includes all wrapper headers at once
│   │   │   └── NEKernels.h --> Includes all the NEON kernels at once
│   │   ├── All common basic types (Types.h, Window, Coordinates, Iterator, etc.)
│   │   ├── All generic objects interfaces (ITensor, IArray, etc.)
│   │   └── Objects metadata classes (TensorInfo, MultiImageInfo)
│   ├── graph
│   │   ├── algorithms
│   │   │   └── Generic algorithms used by the graph backend (e.g Order of traversal)
│   │   ├── backends --> The backend specific code
│   │   │   ├── CL --> OpenCL specific operations
│   │   │   ├── GLES  --> OpenGLES Compute Shaders specific operations
│   │   │   └── NEON --> NEON specific operations
│   │   ├── detail
│   │   │   └── Collection of internal utilities.
│   │   ├── frontend
│   │   │   └── Code related to the stream frontend interface.
│   │   ├── mutators
│   │   │   └── Used to modify / optimise the Graph intermediate representation(Operator fusion, in place operations, etc.)
│   │   ├── nodes
│   │   │   └── The various nodes supported by the graph API
│   │   ├── printers
│   │   │   └── Debug printers
│   │   └── Graph objects ( INode, ITensorAccessor, Graph, etc.)
│   └── runtime
│       ├── common
│       │   └── Common utility code used by all backends
│       ├── CL
│       │   ├── CL objects & allocators (CLArray, CLTensor, etc.)
│       │   ├── functions --> Folder containing all the OpenCL functions
│       │   │   └── CL*.h
│       │   ├── CLScheduler.h --> Interface to enqueue OpenCL kernels and get/set the OpenCL CommandQueue and ICLTuner.
│       │   ├── CLFunctions.h --> Includes all the OpenCL functions at once
│       │   ├── ICLTuner.h --> Interface used to tune the local work-group size of OpenCL kernels
│       │   └── tuners
│       │       └── Local workgroup size tuners for specific architectures / GPUs
│       ├── CPP
│       │   ├── CPPKernels.h --> Includes all the CPP functions at once.
│       │   ├── CPPScheduler.h --> Basic pool of threads to execute CPP/NEON code on several cores in parallel
│       │   └── functions --> Folder containing all the CPP functions
│       │       └── CPP*.h
│       ├── GLES_COMPUTE
│       │   ├── GLES objects & allocators (GCArray, GCTensor, etc.)
│       │   ├── functions --> Folder containing all the GLES functions
│       │   │   └── GC*.h
│       │   ├── GCScheduler.h --> Interface to enqueue GLES kernels and get/set the GLES CommandQueue.
│       │   └── GCFunctions.h --> Includes all the GLES functions at once
│       ├── NEON
│       │   ├── functions --> Folder containing all the NEON functions
│       │   │   └── NE*.h
│       │   └── NEFunctions.h --> Includes all the NEON functions at once
│       ├── OMP
│       │   └── OMPScheduler.h --> OpenMP scheduler (Alternative to the CPPScheduler)
│       ├── Memory & weights manager files (LifetimeManager, PoolManager, etc.)
│       └── Basic implementations of the generic object interfaces (Array, Tensor, etc.)
├── data --> Contains test images and reference data dumps used by validation tests
├── docs --> Contains Doxyfile and Doxygen sources used to generate the HTML pages in the documentation folder.
├── documentation
│   ├── index.xhtml
│   └── ...
├── documentation.xhtml --> documentation/index.xhtml
├── examples
│   ├── cl_*.cpp --> OpenCL examples
│   ├── gc_*.cpp --> GLES compute shaders examples
│   ├── graph_*.cpp --> Graph examples
│   ├── neoncl_*.cpp --> NEON / OpenCL interoperability examples
│   └── neon_*.cpp --> NEON examples
├── include
│   ├── CL
│   │   └── Khronos OpenCL C headers and C++ wrapper
│   ├── half --> FP16 library available from http://half.sourceforge.net
│   ├── libnpy --> Library to load / write npy buffers, available from https://github.com/llohse/libnpy
│   ├── linux --> Headers only needed for Linux builds
│   │   └── Khronos EGL and OpenGLES headers
│   └── stb
│        └── stb_image.h --> Single header library to load image files, available from https://github.com/nothings/stb
├── scripts
│   ├── caffe_data_extractor.py --> Basic script to export weights from Caffe to npy files
│   └── tensorflow_data_extractor.py --> Basic script to export weights from Tensor Flow to npy files
├── src
│   ├── core
│   │   └── ... (Same structure as headers)
│   │       ├── CL
│   │       │   └── cl_kernels --> All the OpenCL kernels
│   │       └── GLES_COMPUTE
│   │           └── cs_shaders --> All the OpenGL ES Compute Shaders
│   ├── graph
│   │   └── ... (Same structure as headers)
│   └── runtime
│       └── ... (Same structure as headers)
├── support
│   └── Various headers to work around toolchains / platform issues.
├── tests
│   ├── All test related files shared between validation and benchmark
│   ├── benchmark --> Sources for benchmarking
│   │   ├── Benchmark specific files
│   │   ├── fixtures
│   │   │   └── Backend agnostic fixtures to initialise and run the functions to test.
│   │   ├── CL --> OpenCL benchmarking tests
│   │   ├── GLES_COMPUTE --> GLES benchmarking tests
│   │   └── NEON --> NEON benchmarking tests
│   ├── benchmark_examples --> Sources needed to wrap examples to run through our benchmarking framework.
│   ├── CL --> OpenCL accessors
│   ├── GLES_COMPUTE --> GLES accessors
│   ├── NEON --> NEON accessors
│   ├── datasets
│   │   └── Datasets for all the validation / benchmark tests, layer configurations for various networks, etc.
│   ├── framework
│   │   └── Boiler plate code for both validation and benchmark test suites (Command line parsers, instruments, output loggers, etc.)
│   ├── instruments --> User defined instruments that can be registered to the framework.
│   ├── validate_examples --> Sources needed to wrap examples to run through our validation framework.
│   └── validation --> Sources for validation
│       ├── Validation specific files
│       ├── fixtures
│       │   └── Backend agnostic fixtures to initialise and run the functions to test.
│       ├── reference
│       │   └── Reference implementation used to validate the results of the various backends.
│       ├── CL --> OpenCL validation tests
│       ├── GLES_COMPUTE --> GLES validation tests
│       ├── CPP --> C++ reference implementations
│       └── NEON --> NEON validation tests
└── utils --> Boiler plate code used by examples
    └── Various utilities to print types, load / store assets, etc.

Release versions and changelog

Release versions

All releases are numbered vYY.MM Where YY are the last two digits of the year, and MM the month number. If there is more than one release in a month then an extra sequential number is appended at the end: 

v17.03 (First release of March 2017)
v17.03.1 (Second release of March 2017)
v17.04 (First release of April 2017)
Note
We're aiming at releasing one major public release with new features per quarter. All releases in between will only contain bug fixes.

Changelog

v20.02.1 Maintenance release

  • Added Android-NN build script.

v20.02 Public major release

v19.11.1 Public maintenance release

  • Fix offset calculation in NEReductionOperationKernel.
  • Fix data layout in NEScaleKernel for nhwc.
  • Retain configuration step data layout to avoid side-effects.
  • Perform sqrt in double domain for L2 pooling.
  • Fix output shape calculation for Reduce Mean
  • Restrict cases where optimized NEPadLayer runs.

v19.11 Public major release

v19.08.1 Public maintenance release

  • Fix offset calculation in NEReductionOperationKernel.
  • Fix data layout in NEScaleKernel for nhwc.
  • Retain configuration step data layout to avoid side-effects.
  • Perform sqrt in double domain for L2 pooling.
  • Fix output shape calculation for Reduce Mean
  • Fix broadcast CLPixelwiseMultiplication with 5D tensors

v19.08 Public major release

v19.05 Public major release

v19.02 Public major release

v18.11 Public major release

v18.08 Public major release

v18.05 Public major release

v18.03 Public maintenance release

  • Various bug fixes.
  • Fixed bug in NEActivationLayer
  • Fix in CLTuner when using batches.
  • Updated recommended NDK version to r16b (And fixed warnings).
  • Fixed bug in validation code.
  • Added Inception v4 graph example.
  • Renamed NEWinogradLayer.cpp to NEWinogradConvolutionLayer

v18.02 Public major release

v18.01 Public maintenance release

v17.12 Public major release

v17.10 Public maintenance release

  • Bug fixes:
    • Check the maximum local workgroup size supported by OpenCL devices
    • Minor documentation updates (Fixed instructions to build the examples)
    • Introduced a graph::GraphContext
    • Added a few new Graph nodes, support for branches and grouping.
    • Automatically enable cl_printf in debug builds
    • Fixed bare metal builds for armv7a
    • Added AlexNet and cartoon effect examples
    • Fixed library builds: libraries are no longer built as supersets of each other.(It means application using the Runtime part of the library now need to link against both libarm_compute_core and libarm_compute)

v17.09 Public major release

v17.06 Public major release

v17.05 Public bug fixes release

  • Various bug fixes
  • Remaining of the functions ported to use accurate padding.
  • Library does not link against OpenCL anymore (It uses dlopen / dlsym at runtime instead to determine whether or not OpenCL is available).
  • Added "free" method to allocator.
  • Minimum version of g++ required for armv7 Linux changed from 4.8 to 4.9

v17.04 Public bug fixes release

The following functions have been ported to use the new accurate padding:

v17.03.1 First Major public release of the sources

v17.03 Sources preview

v17.02.1 Sources preview

v17.02 Sources preview

v16.12 Binary preview release

  • Original release

How to build the library and the examples

Build options

scons 2.3 or above is required to build the library. To see the build options available simply run scons -h: 

debug: Debug (yes|no)
    default: False
    actual: False

asserts: Enable asserts (this flag is forced to 1 for debug=1) (yes|no)
    default: False
    actual: False

arch: Target Architecture (armv7a|arm64-v8a|arm64-v8.2-a|x86_32|x86_64)
    default: armv7a
    actual: armv7a

os: Target OS (linux|android|bare_metal)
    default: linux
    actual: linux

build: Build type (native|cross_compile|embed_only)
    default: cross_compile
    actual: cross_compile

examples: Build example programs (yes|no)
    default: True
    actual: True

Werror: Enable/disable the -Werror compilation flag (yes|no)
    default: True
    actual: True

opencl: Enable OpenCL support (yes|no)
    default: True
    actual: True

neon: Enable Neon support (yes|no)
    default: False
    actual: False

gles_compute: Enable OpenGL ES Compute Shader support (yes|no)
    default: False
    actual: False

embed_kernels: Embed OpenCL kernels and OpenGL ES compute shader in library binary (yes|no)
    default: True
    actual: True

set_soname: Set the library's soname and shlibversion (requires SCons 2.4 or above) (yes|no)
    default: False
    actual: False

openmp: Enable OpenMP backend (yes|no)
    default: False
    actual: False

cppthreads: Enable C++11 threads backend (yes|no)
    default: True
    actual: True

build_dir: Specify sub-folder for the build ( /path/to/build_dir )
    default: .
    actual: .

extra_cxx_flags: Extra CXX flags to be appended to the build command
    default:
    actual:

pmu: Enable PMU counters (yes|no)
    default: False
    actual: False

mali: Enable Mali hardware counters (yes|no)
    default: False
    actual: False

validation_tests: Build validation test programs (yes|no)
    default: False
    actual: False

benchmark_tests: Build benchmark test programs (yes|no)
    default: False
    actual: False

debug / asserts:

  • With debug=1 asserts are enabled, and the library is built with symbols and no optimisations enabled.
  • With debug=0 and asserts=1: Optimisations are enabled and symbols are removed, however all the asserts are still present (This is about 20% slower than the release build)
  • With debug=0 and asserts=0: All optimisations are enable and no validation is performed, if the application misuses the library it is likely to result in a crash. (Only use this mode once you are sure your application is working as expected).

arch: The x86_32 and x86_64 targets can only be used with neon=0 and opencl=1.

os: Choose the operating system you are targeting: Linux, Android or bare metal.

Note
bare metal can only be used for NEON (not OpenCL), only static libraries get built and NEON's multi-threading support is disabled.

build: you can either build directly on your device (native) or cross compile from your desktop machine (cross-compile). In both cases make sure the compiler is available in your path.

Note
If you want to natively compile for 32bit on a 64bit ARM device running a 64bit OS then you will have to use cross-compile too.

There is also an 'embed_only' option which will generate all the .embed files for the OpenCL kernels and / or OpenGLES compute shaders. This might be useful if using a different build system to compile the library.

Werror: If you are compiling using the same toolchains as the ones used in this guide then there shouldn't be any warning and therefore you should be able to keep Werror=1. If with a different compiler version the library fails to build because of warnings interpreted as errors then, if you are sure the warnings are not important, you might want to try to build with Werror=0 (But please do report the issue either on Github or by an email to devel.nosp@m.oper.nosp@m.@arm..nosp@m.com so that the issue can be addressed).

opencl / neon / gles_compute: Choose which SIMD technology you want to target. (NEON for ARM Cortex-A CPUs or OpenCL / GLES_COMPUTE for ARM Mali GPUs)

embed_kernels: For OpenCL / GLES_COMPUTE only: set embed_kernels=1 if you want the OpenCL / GLES_COMPUTE kernels to be built in the library's binaries instead of being read from separate ".cl" / ".cs" files. If embed_kernels is set to 0 then the application can set the path to the folder containing the OpenCL / GLES_COMPUTE kernel files by calling CLKernelLibrary::init() / GCKernelLibrary::init(). By default the path is set to "./cl_kernels" / "./cs_shaders".

set_soname: Do you want to build the versioned version of the library ?

If enabled the library will contain a SONAME and SHLIBVERSION and some symlinks will automatically be created between the objects. Example: libarm_compute_core.so -> libarm_compute_core.so.1.0.0 libarm_compute_core.so.1 -> libarm_compute_core.so.1.0.0 libarm_compute_core.so.1.0.0

Note
This options is disabled by default as it requires SCons version 2.4 or above.

extra_cxx_flags: Custom CXX flags which will be appended to the end of the build command.

build_dir: Build the library in a subfolder of the "build" folder. (Allows to build several configurations in parallel).

examples: Build or not the examples

validation_tests: Enable the build of the validation suite.

benchmark_tests: Enable the build of the benchmark tests

pmu: Enable the PMU cycle counter to measure execution time in benchmark tests. (Your device needs to support it)

mali: Enable the collection of Mali hardware counters to measure execution time in benchmark tests. (Your device needs to have a Mali driver that supports it)

openmp Build in the OpenMP scheduler for NEON.

Note
Only works when building with g++ not clang++

cppthreads Build in the C++11 scheduler for NEON.

See also
Scheduler::set

Building for Linux

How to build the library ?

For Linux, the library was successfully built and tested using the following Linaro GCC toolchain:

  • gcc-linaro-4.9-2016.02-x86_64_arm-linux-gnueabihf
  • gcc-linaro-4.9-2016.02-x86_64_aarch64-linux-gnu

To cross-compile the library in debug mode, with NEON only support, for Linux 32bit: 

scons Werror=1 -j8 debug=1 neon=1 opencl=0 os=linux arch=armv7a

To cross-compile the library in asserts mode, with OpenCL only support, for Linux 64bit: 

scons Werror=1 -j8 debug=0 asserts=1 neon=0 opencl=1 embed_kernels=1 os=linux arch=arm64-v8a

To cross-compile the library in asserts mode, with GLES_COMPUTE only support, for Linux 64bit: 

scons Werror=1 -j8 debug=0 asserts=1 neon=0 opencl=0 gles_compute=1 embed_kernels=1 os=linux arch=arm64-v8a

You can also compile the library natively on an ARM device by using build=native: 

scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=linux arch=arm64-v8a build=native
scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=linux arch=armv7a build=native
Note
g++ for ARM is mono-arch, therefore if you want to compile for Linux 32bit on a Linux 64bit platform you will have to use a cross compiler.

For example on a 64bit Debian based system you would have to install g++-arm-linux-gnueabihf 

apt-get install g++-arm-linux-gnueabihf

Then run 

scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=linux arch=armv7a build=cross_compile

or simply remove the build parameter as build=cross_compile is the default value: 

scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=linux arch=armv7a
Attention
To cross compile with opencl=1 you need to make sure to have a version of libOpenCL matching your target architecture.

How to manually build the examples ?

The examples get automatically built by scons as part of the build process of the library described above. This section just describes how you can build and link your own application against our library.

Note
The following command lines assume the arm_compute binaries are present in the current directory or in the system library path. If this is not the case you can specify the location of the pre-built library with the compiler option -L. When building the OpenCL example the commands below assume that the CL headers are located in the include folder where the command is executed.

To cross compile a NEON example for Linux 32bit: 

arm-linux-gnueabihf-g++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -mfpu=neon -L. -larm_compute -larm_compute_core -o neon_convolution

To cross compile a NEON example for Linux 64bit: 

aarch64-linux-gnu-g++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -L. -larm_compute -larm_compute_core -o neon_convolution

(notice the only difference with the 32 bit command is that we don't need the -mfpu option and the compiler's name is different)

To cross compile an OpenCL example for Linux 32bit: 

arm-linux-gnueabihf-g++ examples/cl_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -mfpu=neon -L. -larm_compute -larm_compute_core -o cl_convolution -DARM_COMPUTE_CL

To cross compile an OpenCL example for Linux 64bit: 

aarch64-linux-gnu-g++ examples/cl_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -L. -larm_compute -larm_compute_core -o cl_convolution -DARM_COMPUTE_CL

To cross compile a GLES example for Linux 32bit: 

arm-linux-gnueabihf-g++ examples/gc_absdiff.cpp utils/Utils.cpp -I. -Iinclude/ -L. -larm_compute -larm_compute_core -std=c++11 -mfpu=neon -DARM_COMPUTE_GC -Iinclude/linux/ -o gc_absdiff

To cross compile a GLES example for Linux 64bit: 

aarch64-linux-gnu-g++ examples/gc_absdiff.cpp utils/Utils.cpp -I. -Iinclude/ -L. -larm_compute -larm_compute_core -std=c++11 -DARM_COMPUTE_GC -Iinclude/linux/ -o gc_absdiff

(notice the only difference with the 32 bit command is that we don't need the -mfpu option and the compiler's name is different)

To cross compile the examples with the Graph API, such as graph_lenet.cpp, you need to link the examples against arm_compute_graph.so too.

Note
The compute library must currently be built with both neon and opencl enabled - neon=1 and opencl=1

i.e. to cross compile the "graph_lenet" example for Linux 32bit: 

arm-linux-gnueabihf-g++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 -mfpu=neon -L. -larm_compute_graph -larm_compute -larm_compute_core -Wl,--allow-shlib-undefined -o graph_lenet

i.e. to cross compile the "graph_lenet" example for Linux 64bit: 

aarch64-linux-gnu-g++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 -L. -larm_compute_graph -larm_compute -larm_compute_core -Wl,--allow-shlib-undefined -o graph_lenet

(notice the only difference with the 32 bit command is that we don't need the -mfpu option and the compiler's name is different)

Note
If compiling using static libraries, this order must be followed when linking: arm_compute_graph_static, arm_compute, arm_compute_core

To compile natively (i.e directly on an ARM device) for NEON for Linux 32bit: 

g++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -mfpu=neon -larm_compute -larm_compute_core -o neon_convolution

To compile natively (i.e directly on an ARM device) for NEON for Linux 64bit: 

g++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute -larm_compute_core -o neon_convolution

(notice the only difference with the 32 bit command is that we don't need the -mfpu option)

To compile natively (i.e directly on an ARM device) for OpenCL for Linux 32bit or Linux 64bit: 

g++ examples/cl_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute -larm_compute_core -o cl_convolution -DARM_COMPUTE_CL

To compile natively (i.e directly on an ARM device) for GLES for Linux 32bit or Linux 64bit: 

g++ examples/gc_absdiff.cpp utils/Utils.cpp -I. -Iinclude/ -L. -larm_compute -larm_compute_core -std=c++11 -DARM_COMPUTE_GC -Iinclude/linux/ -o gc_absdiff

To compile natively the examples with the Graph API, such as graph_lenet.cpp, you need to link the examples against arm_compute_graph.so too.

Note
The compute library must currently be built with both neon and opencl enabled - neon=1 and opencl=1

i.e. to natively compile the "graph_lenet" example for Linux 32bit: 

g++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 -mfpu=neon -L. -larm_compute_graph -larm_compute -larm_compute_core -Wl,--allow-shlib-undefined -o graph_lenet

i.e. to natively compile the "graph_lenet" example for Linux 64bit: 

g++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 L. -larm_compute_graph -larm_compute -larm_compute_core -Wl,--allow-shlib-undefined -o graph_lenet

(notice the only difference with the 32 bit command is that we don't need the -mfpu option)

Note
If compiling using static libraries, this order must be followed when linking: arm_compute_graph_static, arm_compute, arm_compute_core
These two commands assume libarm_compute.so is available in your library path, if not add the path to it using -L
You might need to export the path to OpenCL library as well in your LD_LIBRARY_PATH if Compute Library was build with OpenCL enabled.

To run the built executable simply run: 

LD_LIBRARY_PATH=build ./neon_convolution

or 

LD_LIBRARY_PATH=build ./cl_convolution
Note
Examples accept different types of arguments, to find out what they are run the example with –help as an argument. If no arguments are specified then random values will be used to execute the graph.

For example: 

LD_LIBRARY_PATH=. ./graph_lenet --help

Below is a list of the common parameters among the graph examples :

/* Common graph parameters
*
* --help : Print the example's help message.
* --threads : The number of threads to be used by the example during execution.
* --target : Execution target to be used by the examples. Supported target options: NEON, CL, GC.
* --type : Data type to be used by the examples. Supported data type options: QASYMM8, F16, F32.
* --layout : Data layout to be used by the examples. Supported data layout options : NCHW, NHWC.
* --enable-tuner : Toggle option to enable the OpenCL dynamic tuner.
* --enable-cl-cache : Toggle option to load the prebuilt opencl kernels from a cache file.
* --fast-math : Toggle option to enable the fast math option.
* --data : Path that contains the trainable parameter files of graph layers.
* --image : Image to load and operate on. Image types supported: PPM, JPEG, NPY.
* --labels : File that contains the labels that classify upon.
* --validation-file : File that contains a list of image names with their corresponding label id (e.g. image0.jpg 5).
* This is used to run the graph over a number of images and report top-1 and top-5 metrics.
* --validation-path : The path where the validation images specified in the validation file reside.
* --validation-range : The range of the images to validate from the validation file (e.g 0,9).
* If not specified all the images will be validated.
* --tuner-file : The file to store the OpenCL dynamic tuner tuned parameters.
*
* Note that data, image and labels options should be provided to perform an inference run on an image.
* Note that validation-file and validation-path should be provided to perform a graph accuracy estimation.
* Note GLES target is not supported for most of the networks.
*
* Example execution commands:
*
* Execute a single inference given an image and a file containing the correspondence between label ids and human readable labels:
* ./graph_vgg16 --data=data/ --target=cl --layout=nhwc --image=kart.jpeg --labels=imagenet1000_clsid_to_human.txt
*
* Perform a graph validation on a list of images:
* ./graph_vgg16 --data=data/ --target=neon --threads=4 --layout=nchw --validation-file=val.txt --validation-path=ilsvrc_test_images/
*
* File formats:
*
* Validation file should be a plain file containing the names of the images followed by the correct label id.
* For example:
*
* image0.jpeg 882
* image1.jpeg 34
* image2.jpeg 354
*
* Labels file should be a plain file where each line is the respective human readable label (counting starts from 0).
* For example:
*
* 0: label0_name label0_name
* 1: label1_name or label1_name
* 2: label2_name label2_name
*/

Building for Android

For Android, the library was successfully built and tested using Google's standalone toolchains:

  • clang++ from NDK r17c for armv7a
  • clang++ from NDK r17c for arm64-v8a
  • clang++ from NDK r18-beta1 for arm64-v8.2-a with FP16 support

Here is a guide to create your Android standalone toolchains from the NDK

$NDK/build/tools/make_standalone_toolchain.py --arch arm64 --install-dir $MY_TOOLCHAINS/aarch64-linux-android-ndk-r17c --stl libc++ --api 21
$NDK/build/tools/make_standalone_toolchain.py --arch arm --install-dir $MY_TOOLCHAINS/arm-linux-android-ndk-r17c --stl libc++ --api 21
Attention
We used to use gnustl but as of NDK r17 it is deprecated so we switched to libc++
Note
Make sure to add the toolchains to your PATH: 
export PATH=$PATH:$MY_TOOLCHAINS/aarch64-linux-android-ndk-r17c/bin:$MY_TOOLCHAINS/arm-linux-android-ndk-r17c/bin

How to build the library ?

To cross-compile the library in debug mode, with NEON only support, for Android 32bit: 

CXX=clang++ CC=clang scons Werror=1 -j8 debug=1 neon=1 opencl=0 os=android arch=armv7a

To cross-compile the library in asserts mode, with OpenCL only support, for Android 64bit: 

CXX=clang++ CC=clang scons Werror=1 -j8 debug=0 asserts=1 neon=0 opencl=1 embed_kernels=1 os=android arch=arm64-v8a

To cross-compile the library in asserts mode, with GLES_COMPUTE only support, for Android 64bit: 

CXX=clang++ CC=clang scons Werror=1 -j8 debug=0 asserts=1 neon=0 opencl=0 gles_compute=1 embed_kernels=1 os=android arch=arm64-v8a

How to manually build the examples ?

The examples get automatically built by scons as part of the build process of the library described above. This section just describes how you can build and link your own application against our library.

Note
The following command lines assume the arm_compute binaries are present in the current directory or in the system library path. If this is not the case you can specify the location of the pre-built library with the compiler option -L. When building the OpenCL example the commands below assume that the CL headers are located in the include folder where the command is executed.

Once you've got your Android standalone toolchain built and added to your path you can do the following:

To cross compile a NEON example: 

#32 bit:
arm-linux-androideabi-clang++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o neon_convolution_arm -static-libstdc++ -pie
#64 bit:
aarch64-linux-android-clang++ examples/neon_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o neon_convolution_aarch64 -static-libstdc++ -pie

To cross compile an OpenCL example: 

#32 bit:
arm-linux-androideabi-clang++ examples/cl_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o cl_convolution_arm -static-libstdc++ -pie -DARM_COMPUTE_CL
#64 bit:
aarch64-linux-android-clang++ examples/cl_convolution.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o cl_convolution_aarch64 -static-libstdc++ -pie -DARM_COMPUTE_CL

To cross compile a GLES example: 

#32 bit:
arm-linux-androideabi-clang++ examples/gc_absdiff.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o gc_absdiff_arm -static-libstdc++ -pie -DARM_COMPUTE_GC
#64 bit:
aarch64-linux-android-clang++ examples/gc_absdiff.cpp utils/Utils.cpp -I. -Iinclude -std=c++11 -larm_compute-static -larm_compute_core-static -L. -o gc_absdiff_aarch64 -static-libstdc++ -pie -DARM_COMPUTE_GC

To cross compile the examples with the Graph API, such as graph_lenet.cpp, you need to link the library arm_compute_graph also. (notice the compute library has to be built with both neon and opencl enabled - neon=1 and opencl=1) 

#32 bit:
arm-linux-androideabi-clang++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 -Wl,--whole-archive -larm_compute_graph-static -Wl,--no-whole-archive -larm_compute-static -larm_compute_core-static -L. -o graph_lenet_arm -static-libstdc++ -pie -DARM_COMPUTE_CL
#64 bit:
aarch64-linux-android-clang++ examples/graph_lenet.cpp utils/Utils.cpp utils/GraphUtils.cpp utils/CommonGraphOptions.cpp -I. -Iinclude -std=c++11 -Wl,--whole-archive -larm_compute_graph-static -Wl,--no-whole-archive -larm_compute-static -larm_compute_core-static -L. -o graph_lenet_aarch64 -static-libstdc++ -pie -DARM_COMPUTE_CL
Note
Due to some issues in older versions of the Mali OpenCL DDK (<= r13p0), we recommend to link arm_compute statically on Android.
When linked statically the arm_compute_graph library currently needs the –whole-archive linker flag in order to work properly

Then you need to do is upload the executable and the shared library to the device using ADB: 

adb push neon_convolution_arm /data/local/tmp/
adb push cl_convolution_arm /data/local/tmp/
adb push gc_absdiff_arm /data/local/tmp/
adb shell chmod 777 -R /data/local/tmp/

And finally to run the example: 

adb shell /data/local/tmp/neon_convolution_arm
adb shell /data/local/tmp/cl_convolution_arm
adb shell /data/local/tmp/gc_absdiff_arm

For 64bit: 

adb push neon_convolution_aarch64 /data/local/tmp/
adb push cl_convolution_aarch64 /data/local/tmp/
adb push gc_absdiff_aarch64 /data/local/tmp/
adb shell chmod 777 -R /data/local/tmp/

And finally to run the example: 

adb shell /data/local/tmp/neon_convolution_aarch64
adb shell /data/local/tmp/cl_convolution_aarch64
adb shell /data/local/tmp/gc_absdiff_aarch64
Note
Examples accept different types of arguments, to find out what they are run the example with –help as an argument. If no arguments are specified then random values will be used to execute the graph.

For example: adb shell /data/local/tmp/graph_lenet –help

In this case the first argument of LeNet (like all the graph examples) is the target (i.e 0 to run on NEON, 1 to run on OpenCL if available, 2 to run on OpenCL using the CLTuner), the second argument is the path to the folder containing the npy files for the weights and finally the third argument is the number of batches to run.

Building for bare metal

For bare metal, the library was successfully built using linaros's latest (gcc-linaro-6.3.1-2017.05) bare metal toolchains:

  • arm-eabi for armv7a
  • aarch64-elf for arm64-v8a

Download linaro for armv7a and arm64-v8a.

Note
Make sure to add the toolchains to your PATH: export PATH=$PATH:$MY_TOOLCHAINS/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-elf/bin:$MY_TOOLCHAINS/gcc-linaro-6.3.1-2017.05-x86_64_arm-eabi/bin

How to build the library ?

To cross-compile the library with NEON support for baremetal arm64-v8a: 

scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=bare_metal arch=arm64-v8a build=cross_compile cppthreads=0 openmp=0 standalone=1

How to manually build the examples ?

Examples are disabled when building for bare metal. If you want to build the examples you need to provide a custom bootcode depending on the target architecture and link against the compute library. More information about bare metal bootcode can be found here.

Building on a Windows host system

Using scons directly from the Windows command line is known to cause problems. The reason seems to be that if scons is setup for cross-compilation it gets confused about Windows style paths (using backslashes). Thus it is recommended to follow one of the options outlined below.

Bash on Ubuntu on Windows

The best and easiest option is to use Ubuntu on Windows. This feature is still marked as beta and thus might not be available. However, if it is building the library is as simple as opening a Bash on Ubuntu on Windows shell and following the general guidelines given above.

Cygwin

If the Windows subsystem for Linux is not available Cygwin can be used to install and run scons, the minimum Cygwin version must be 3.0.7 or later. In addition to the default packages installed by Cygwin scons has to be selected in the installer. (git might also be useful but is not strictly required if you already have got the source code of the library.) Linaro provides pre-built versions of GCC cross-compilers that can be used from the Cygwin terminal. When building for Android the compiler is included in the Android standalone toolchain. After everything has been set up in the Cygwin terminal the general guide on building the library can be followed.

The OpenCL stub library

In the opencl-1.2-stubs folder you will find the sources to build a stub OpenCL library which then can be used to link your application or arm_compute against.

If you preferred you could retrieve the OpenCL library from your device and link against this one but often this library will have dependencies on a range of system libraries forcing you to link your application against those too even though it is not using them.

Warning
This OpenCL library provided is a stub and not a real implementation. You can use it to resolve OpenCL's symbols in arm_compute while building the example but you must make sure the real libOpenCL.so is in your PATH when running the example or it will not work.

To cross-compile the stub OpenCL library simply run: 

<target-prefix>-gcc -o libOpenCL.so -Iinclude opencl-1.2-stubs/opencl_stubs.c -fPIC -shared

For example: 

#Linux 32bit
arm-linux-gnueabihf-gcc -o libOpenCL.so -Iinclude opencl-1.2-stubs/opencl_stubs.c -fPIC -shared
#Linux 64bit
aarch64-linux-gnu-gcc -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC
#Android 32bit
arm-linux-androideabi-clang -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC -shared
#Android 64bit
aarch64-linux-android-clang -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC -shared

The Linux OpenGLES and EGL stub libraries

In the opengles-3.1-stubs folder you will find the sources to build stub EGL and OpenGLES libraries which then can be used to link your Linux application of arm_compute against.

Note
The stub libraries are only needed on Linux. For Android, the NDK toolchains already provide the meta-EGL and meta-GLES libraries.

To cross-compile the stub OpenGLES and EGL libraries simply run: 

<target-prefix>-gcc -o libEGL.so -Iinclude/linux opengles-3.1-stubs/EGL.c -fPIC -shared
<target-prefix>-gcc -o libGLESv2.so -Iinclude/linux opengles-3.1-stubs/GLESv2.c -fPIC -shared

#Linux 32bit
arm-linux-gnueabihf-gcc -o libEGL.so -Iinclude/linux opengles-3.1-stubs/EGL.c -fPIC -shared
arm-linux-gnueabihf-gcc -o libGLESv2.so -Iinclude/linux opengles-3.1-stubs/GLESv2.c -fPIC -shared

#Linux 64bit
aarch64-linux-gnu-gcc -o libEGL.so -Iinclude/linux opengles-3.1-stubs/EGL.c -fPIC -shared
aarch64-linux-gnu-gcc -o libGLESv2.so -Iinclude/linux opengles-3.1-stubs/GLESv2.c -fPIC -shared

OpenCL DDK Requirements

Hard Requirements

Compute Library requires OpenCL 1.1 and above with support of non uniform workgroup sizes, which is officially supported in the Mali OpenCL DDK r8p0 and above as an extension (respective extension flag is -cl-arm-non-uniform-work-group-size).

Enabling 16-bit floating point calculations require cl_khr_fp16 extension to be supported. All Mali GPUs with compute capabilities have native support for half precision floating points.

Use of CLMeanStdDev function requires 64-bit atomics support, thus cl_khr_int64_base_atomics should be supported in order to use.

Performance improvements

Integer dot product built-in function extensions (and therefore optimized kernels) are available with Mali OpenCL DDK r22p0 and above for the following GPUs : G71, G76. The relevant extensions are cl_arm_integer_dot_product_int8, cl_arm_integer_dot_product_accumulate_int8 and cl_arm_integer_dot_product_accumulate_int16.

OpenCL kernel level debugging can be simplified with the use of printf, this requires the cl_arm_printf extension to be supported.

SVM allocations are supported for all the underlying allocations in Compute Library. To enable this OpenCL 2.0 and above is a requirement.

OpenCL Tuner

The OpenCL tuner, a.k.a. CLTuner, is a module of Arm Compute Library that can improve the performance of the OpenCL kernels tuning the Local-Workgroup-Size (LWS). The optimal LWS for each unique OpenCL kernel configuration is stored in a table. This table can be either imported or exported from/to a file. The OpenCL tuner runs the same OpenCL kernel for a range of local workgroup sizes and keeps the local workgroup size of the fastest run to use in subsequent calls to the kernel. It supports three modes of tuning with different trade-offs between the time taken to tune and the kernel execution time achieved using the best LWS found. In the Exhaustive mode, it searches all the supported values of LWS. This mode takes the longest time to tune and is the most likely to find the optimal LWS. Normal mode searches a subset of LWS values to yield a good approximation of the optimal LWS. It takes less time to tune than Exhaustive mode. Rapid mode takes the shortest time to tune and finds an LWS value that is at least as good or better than the default LWS value. The mode affects only the search for the optimal LWS and has no effect when the LWS value is imported from a file. In order for the performance numbers to be meaningful you must disable the GPU power management and set it to a fixed frequency for the entire duration of the tuning phase.

If you wish to know more about LWS and the important role on improving the GPU cache utilization, we suggest having a look at the presentation "Even Faster CNNs: Exploring the New Class of Winograd Algorithms available at the following link:

https://www.embedded-vision.com/platinum-members/arm/embedded-vision-training/videos/pages/may-2018-embedded-vision-summit-iodice

Tuning a network from scratch can be long and affect considerably the execution time for the first run of your network. It is recommended for this reason to store the CLTuner's result in a file to amortize this time when you either re-use the same network or the functions with the same configurations. The tuning is performed only once for each OpenCL kernel.

CLTuner looks for the optimal LWS for each unique OpenCL kernel configuration. Since a function (i.e. Convolution Layer, Pooling Layer, Fully Connected Layer ...) can be called multiple times but with different parameters, we associate an "id" (called "config_id") to each kernel to distinguish the unique configurations. 

#Example: 2 unique Matrix Multiply configurations
TensorShape a0 = TensorShape(32,32);
TensorShape b0 = TensorShape(32,32);
TensorShape c0 = TensorShape(32,32);
TensorShape a1 = TensorShape(64,64);
TensorShape b1 = TensorShape(64,64);
TensorShape c1 = TensorShape(64,64);
Tensor a0_tensor;
Tensor b0_tensor;
Tensor c0_tensor;
Tensor a1_tensor;
Tensor b1_tensor;
Tensor c1_tensor;
a0_tensor.allocator()->init(TensorInfo(a0, 1, DataType::F32));
b0_tensor.allocator()->init(TensorInfo(b0, 1, DataType::F32));
c0_tensor.allocator()->init(TensorInfo(c0, 1, DataType::F32));
a1_tensor.allocator()->init(TensorInfo(a1, 1, DataType::F32));
b1_tensor.allocator()->init(TensorInfo(b1, 1, DataType::F32));
c1_tensor.allocator()->init(TensorInfo(c1 1, DataType::F32));
CLGEMM gemm0;
CLGEMM gemm1;
// Configuration 0
gemm0.configure(&a0, &b0, nullptr, &c0, 1.0f, 0.0f);
// Configuration 1
gemm1.configure(&a1, &b1, nullptr, &c1, 1.0f, 0.0f);

How to use it

All the graph examples in the ACL's folder "examples" and the arm_compute_benchmark accept an argument to enable the OpenCL tuner and an argument to export/import the LWS values to/from a file 

#Enable CL tuner
./graph_mobilenet --enable-tuner –-target=CL
./arm_compute_benchmark --enable-tuner

#Export/Import to/from a file
./graph_mobilenet --enable-tuner --target=CL --tuner-file=acl_tuner.csv
./arm_compute_benchmark --enable-tuner --tuner-file=acl_tuner.csv

If you are importing the CLTuner'results from a file, the new tuned LWS values will be appended to it.

Either you are benchmarking the graph examples or the test cases in the arm_compute_benchmark remember to: 

-# Disable the power management
-# Keep the GPU frequency constant
-# Run multiple times the network (i.e. 10).

If you are not using the graph API or the benchmark infrastructure you will need to manually pass a CLTuner object to CLScheduler before configuring any function.

CLTuner tuner;
// Setup Scheduler
CLScheduler::get().default_init(&tuner);

After the first run, the CLTuner's results can be exported to a file using the method "save_to_file()".

  • tuner.save_to_file("results.csv");

This file can be also imported using the method "load_from_file("results.csv")".

  • tuner.load_from_file("results.csv");