ARM Compute Library  17.03.1
Documentation

# Introduction

The ARM Computer Vision and Machine Learning library is a set of functions optimised for both ARM CPUs and GPUs using SIMD technologies.

Several builds of the library are available using various configurations:

• OS: Linux, Android or bare metal.
• Architecture: armv7a (32bit) or arm64-v8a (64bit)
• Technology: NEON / OpenCL / NEON and OpenCL
• Debug / Asserts / Release: Use a build with asserts enabled to debug your application and enable extra validation. Once you are sure your application works as expected you can switch to a release build of the library for maximum performance.

## Contact / Support

In order to facilitate the work of the support team please provide the build information of the library you are using. To get the version of the library you are using simply run:

### How to build the library ?

Note
If you are building with opencl=1 then scons will expect to find libOpenCL.so either in the current directory or in "build" (See the section below if you need a stub OpenCL library to link against)

To cross-compile the library in debug mode, with NEON only support, for Android 32bit:

CXX=clang++ CC=clang scons Werror=1 -j8 debug=1 neon=1 opencl=0 os=android arch=armv7a

Attention
Due to some NDK issues make sure you use g++ & gnustl for aarch64 and clang++ & libc++ for armv7

To cross-compile the library in asserts mode, with OpenCL only support, for Android 64bit:

scons Werror=1 -j8 debug=0 asserts=1 neon=0 opencl=1 embed_kernels=1 os=android arch=arm64-v8a


### How to manually build the examples ?

The examples get automatically built by scons as part of the build process of the library described above. This section just describes how you can build and link your own application against our library.

Note
The following command lines assume the arm_compute binaries are present in the current directory or in the system library path.

Once you've got your Android standalone toolchain built and added to your path you can do the following:

To cross compile a NEON example:

#32 bit:
arm-linux-androideabi-clang++ examples/neon_convolution.cpp -I. -Iinclude -std=c++11 -larm_compute-static -L. -o neon_convolution_arm -static-libstdc++ -pie
#64 bit:
aarch64-linux-android-g++ examples/neon_convolution.cpp -I. -Iinclude -std=c++11 -larm_compute-static -L. -o neon_convolution_aarch64 -static-libstdc++ -pie


To cross compile an OpenCL example:

#32 bit:
arm-linux-androideabi-clang++ examples/cl_convolution.cpp -I. -Iinclude -std=c++11 -larm_compute-static -L. -o cl_convolution_arm -static-libstdc++ -pie -lOpenCL
#64 bit:
aarch64-linux-android-g++ examples/cl_convolution.cpp -I. -Iinclude -std=c++11 -larm_compute-static -L. -o cl_convolution_aarch64 -static-libstdc++ -pie -lOpenCL

Note
Due to some issues in older versions of the Mali OpenCL DDK (<= r13p0), we recommend to link arm_compute statically on Android.

Then you need to do is upload the executable and the shared library to the device using ADB:

adb push neon_convolution_arm /data/local/tmp/
adb shell chmod 777 -R /data/local/tmp/


And finally to run the example:

adb shell /data/local/tmp/neon_convolution_arm


For 64bit:

adb push neon_convolution_aarch64 /data/local/tmp/
adb shell chmod 777 -R /data/local/tmp/


And finally to run the example:

adb shell /data/local/tmp/neon_convolution_aarch64


## The OpenCL stub library

In the opencl-1.2-stubs folder you will find the sources to build a stub OpenCL library which then can be used to link your application or arm_compute against.

If you preferred you could retrieve the OpenCL library from your device and link against this one but often this library will have dependencies on a range of system libraries forcing you to link your application against those too even though it is not using them.

Warning
This OpenCL library provided is a stub and not a real implementation. You can use it to resolve OpenCL's symbols in arm_compute while building the example but you must make sure the real libOpenCL.so is in your PATH when running the example or it will not work.

To cross-compile the stub OpenCL library simply run:

<target-prefix>-gcc -o libOpenCL.so -Iinclude opencl-1.2-stubs/opencl_stubs.c -fPIC -shared


For example:

<target-prefix>-gcc -o libOpenCL.so -Iinclude opencl-1.2-stubs/opencl_stubs.c -fPIC -shared
#Linux 32bit
arm-linux-gnueabihf-gcc -o libOpenCL.so -Iinclude opencl-1.2-stubs/opencl_stubs.c -fPIC -shared
#Linux 64bit
aarch64-linux-gnu-gcc -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC
#Android 32bit
arm-linux-androideabi-clang -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC -shared
#Android 64bit
aarch64-linux-android-gcc -o libOpenCL.so -Iinclude -shared opencl-1.2-stubs/opencl_stubs.c -fPIC -shared


# Library Architecture

## Core vs Runtime libraries

The Core library is a low level collection of algorithms implementations, it is designed to be embedded in existing projects and applications:

• It doesn't allocate any memory (All the memory allocations/mappings have to be handled by the caller).
• It doesn't perform any kind of multi-threading (but provide information to the caller about how the workload can be split).

The Runtime library is a very basic wrapper around the Core library which can be used for quick prototyping, it is basic in the sense that:

• It allocates images and tensors are allocatd using standard malloc().
• It multi-threads NEON code in a very basic way using a very simple pool of threads.
• For OpenCL it will use the default CLScheduler command queue for all mapping operations and kernels.

For maximum performance, it is expected that the users would re-implement an equivalent to the runtime library which suits better their needs (With a more clever multi-threading strategy, load-balancing between NEON and OpenCL, etc.)

## Windows, kernels, multi-threading and functions

### Windows

A Window represents a workload to execute, it's made of up to Coordinates::num_max_dimensions dimensions. Each dimension is defined by a start, end and step.

It can split into subwindows as long as all the following rules remain true for all the dimensions:

• max[n].start() <= sub[n].start() < max[n].end()
• sub[n].start() < sub[n].end() <= max[n].end()
• max[n].step() == sub[n].step()
• (sub[n].start() - max[n].start()) % max[n].step() == 0
• (sub[n].end() - sub[n].start()) % max[n].step() == 0

### Kernels

Each implementation of the IKernel interface (base class of all the kernels in the core library) works in the same way:

OpenCL kernels:

// Initialise the CLScheduler with the default context and default command queue
// Also initialises the CLKernelLibrary to use ./cl_kernels as location for OpenCL kernels files and sets a default device for which OpenCL programs are built.
cl::CommandQueue q = CLScheduler::get().queue();
//Create a kernel object:
MyKernel kernel;
// Initialize the kernel with the input/output and options you want to use:
kernel.configure( input, output, option0, option1);
// Retrieve the execution window of the kernel:
const Window& max_window = kernel.window();
// Run the whole kernel in the current thread:
kernel.run( q, max_window ); // Enqueue the kernel to process the full window on the default queue
// Wait for the processing to complete:
q.finish();

NEON / CPP kernels:

//Create a kernel object:
MyKernel kernel;
// Initialize the kernel with the input/output and options you want to use:
kernel.configure( input, output, option0, option1);
// Retrieve the execution window of the kernel:
const Window& max_window = kernel.window();
// Run the whole kernel in the current thread:
kernel.run( max_window ); // Run the kernel on the full window

The previous section shows how to run a NEON / CPP kernel in the current thread, however if your system has several CPU cores, you will probably want the kernel to use several cores. Here is how this can be done:

const Window &max_window = kernel->window();
const int num_iterations = max_window.num_iterations(split_dimension);
{
kernel->run(max_window);
}
else
{
for(int t = 0; t < num_threads; ++t)
{
Window win = max_window.split_window(split_dimension, t, num_threads);
{
}
else
{
kernel->run(win);
}
}
try
{
for(int t = 1; t < num_threads; ++t)
{
}
}
catch(const std::system_error &e)
{
std::cout << "Caught system_error with code " << e.code() << " meaning " << e.what() << '\n';
}
}

This is the very basic implementation used in the NEON runtime library by all the NEON functions,

CPPScheduler.
Note
Some kernels like for example NEHistogramKernel need some local temporary buffer to perform their calculations. In order to avoid memory corruption between threads, the local buffer must be of size: memory_needed_per_thread * num_threads and each subwindow must be initialised by calling Window::set_thread_id() with a unique thread_id between 0 and num_threads.

### Functions

Functions will automatically allocate the temporary buffers mentioned above, and will automatically multi-thread kernels' executions using the very basic scheduler described in the previous section.

Simple functions are made of a single kernel (e.g NEConvolution3x3), while more complex ones will be made of a several kernels pipelined together (e.g NEGaussianPyramid, NEHarrisCorners), check their documentation to find out which kernels are used by each function.

//Create a function object:
MyFunction function;
// Initialize the function with the input/output and options you want to use:
function.configure( input, output, option0, option1);
// Execute the function:
function.run();
Warning
ARM Compute libraries require Mali OpenCL DDK r8p0 or above(OpenCL kernels are compiled using the -cl-arm-non-uniform-work-group-size flag)
Note
All OpenCL functions and objects in the runtime library use the command queue associated with CLScheduler for all operations, a real implementation would be expected to use different queues for mapping operations and kernels in order to reach a better GPU utilisation.

### OpenCL Scheduler and kernel library

The ARM Compute runtime uses a single command queue and context for all the operations.

The user can get / set this context and command queue through the CLScheduler's interface.

Attention
Make sure the application is using the same context as the library as in OpenCL it is forbidden to share objects across contexts. This is done by calling CLScheduler::init() or CLScheduler::default_init() at the beginning of your application.

All the OpenCL kernels used by the library are built and stored in the CLKernelLibrary. If the library is compiled with embed_kernels=0 the application can set the path to the OpenCL kernels by calling CLKernelLibrary::init(), by default the path is set to "./cl_kernels"

### OpenCL events and synchronisation

In order to block until all the jobs in the CLScheduler's command queue are done executing the user can call CLScheduler::sync() or create a sync event using CLScheduler::enqueue_sync_event()

For example:

CLImage src, tmp_scale_median, tmp_median_gauss, dst;
constexpr int scale_factor = 2;
if(argc < 2)
{
// Print help
std::cout << "Usage: ./build/cl_events [input_image.ppm]\n\n";
std::cout << "No input_image provided, creating a dummy 640x480 image\n";
// Create an empty grayscale 640x480 image
src.allocator()->init(TensorInfo(640, 480, Format::U8));
}
else
{
ppm.open(argv[1]);
ppm.init_image(src, Format::U8);
}
// Declare and configure the functions to create the following pipeline: scale -> median -> gauss
CLScale scale;
CLMedian3x3 median;
CLGaussian5x5 gauss;
TensorInfo dst_info(src.info()->dimension(0) / scale_factor, src.info()->dimension(1) / scale_factor, Format::U8);
// Configure the temporary and destination images
dst.allocator()->init(dst_info);
tmp_scale_median.allocator()->init(dst_info);
tmp_median_gauss.allocator()->init(dst_info);
//Configure the functions:
scale.configure(&src, &tmp_scale_median, InterpolationPolicy::NEAREST_NEIGHBOR, BorderMode::REPLICATE);
median.configure(&tmp_scale_median, &tmp_median_gauss, BorderMode::REPLICATE);
gauss.configure(&tmp_median_gauss, &dst, BorderMode::REPLICATE);
// Allocate all the images
src.allocator()->allocate();
dst.allocator()->allocate();
tmp_scale_median.allocator()->allocate();
tmp_median_gauss.allocator()->allocate();
// Fill the input image with the content of the PPM image if a filename was provided:
if(ppm.is_open())
{
ppm.fill_image(src);
}
// Enqueue and flush the scale OpenCL kernel:
scale.run();
// Create a synchronisation event between scale and median:
cl::Event scale_event = CLScheduler::get().enqueue_sync_event();
// Enqueue and flush the median OpenCL kernel:
median.run();
// Enqueue and flush the Gaussian OpenCL kernel:
gauss.run();
//Make sure all the OpenCL jobs are done executing:
scale_event.wait(); // Block until Scale is done executing (Median3x3 and Gaussian5x5 might still be running)
CLScheduler::get().sync(); // Block until Gaussian5x5 is done executing
// Save the result to file:
if(ppm.is_open())
{
const std::string output_filename = std::string(argv[1]) + "_out.ppm";
save_to_ppm(dst, output_filename); // save_to_ppm maps and unmaps the image to store as PPM
}

### OpenCL / NEON interoperability

You can mix OpenCL and NEON kernels and or functions, however it is the user's responsibility to handle the mapping unmapping of the OpenCL objects, for example:

CLImage src, scale_median, median_gauss, dst;
if(argc < 2)
{
// Print help
std::cout << "Usage: ./build/cl_convolution [input_image.ppm]\n\n";
std::cout << "No input_image provided, creating a dummy 640x480 image\n";
// Create an empty grayscale 640x480 image
src.allocator()->init(TensorInfo(640, 480, Format::U8));
}
else
{
ppm.open(argv[1]);
ppm.init_image(src, Format::U8);
}
TensorInfo scale_median_info(TensorInfo(src.info()->dimension(0) / 2, src.info()->dimension(1) / 2, Format::U8));
// Configure the temporary and destination images
scale_median.allocator()->init(scale_median_info);
median_gauss.allocator()->init(scale_median_info);
dst.allocator()->init(scale_median_info);
// Declare and configure the functions to create the following pipeline: scale -> median -> gauss
CLScale scale;
NEMedian3x3 median;
CLGaussian5x5 gauss;
scale.configure(&src, &scale_median, InterpolationPolicy::NEAREST_NEIGHBOR, BorderMode::REPLICATE);
median.configure(&scale_median, &median_gauss, BorderMode::REPLICATE);
gauss.configure(&median_gauss, &dst, BorderMode::REPLICATE);
// Allocate all the images
src.allocator()->allocate();
scale_median.allocator()->allocate();
median_gauss.allocator()->allocate();
dst.allocator()->allocate();
// Fill the input image with the content of the PPM image if a filename was provided:
if(ppm.is_open())
{
ppm.fill_image(src);
}
// Enqueue and flush the OpenCL kernel:
scale.run();
// Do a blocking map of the input and output buffers of the NEON function:
scale_median.map();
median_gauss.map();
// Run the NEON function:
median.run();
// Unmap the output buffer before it's used again by OpenCL:
scale_median.unmap();
median_gauss.unmap();
// Run the final OpenCL function:
gauss.run();
// Make sure all the OpenCL jobs are done executing:
// Save the result to file:
if(ppm.is_open())
{
const std::string output_filename = std::string(argv[1]) + "_out.ppm";
save_to_ppm(dst, output_filename); // save_to_ppm maps and unmaps the image to store as PPM
}
main_neoncl_scale_median_gaussian

## Algorithms

All algorithms in this library have been implemented following the OpenVX 1.1 specifications Please refer to the Khronos documentation for more information.

## Images, padding, border modes and tensors

Most kernels and functions in the library process images, however, in order to be future proof most of the kernels actually accept tensors, see below for more information about they are related.

Attention
Each memory object can be written by only one kernel, however it can be read by several kernels. Writing to the same object from several kernels will result in undefined behaviour. The kernel writing to an object must be configured before the kernel(s) reading from it.

Several algorithms rely on neighbour pixels to calculate the value of a given pixel: this means the algorithm will not be able to process the borders of the image unless you give it more information about what you want to happen for border pixels, this is the BorderMode.

You have 3 types of BorderMode :

• BorderMode::UNDEFINED : if you are missing pixel values then don't calculate the value. As a result all the pixels which are on the border will have a value which is undefined.
• BorderMode::REPLICATE : if you are missing pixel values then assume the missing pixels have the same value as the closest valid pixel.
• BorderMode::CONSTANT : if you are missing pixel values then assume the missing pixels all have the same constant value (The user can choose what this value should be).

Moreover both OpenCL and NEON use vector loads and stores instructions to access the data in buffers, so in order to avoid having special cases to handle for the borders all the images and tensors used in this library must be padded.

There are different ways padding can be calculated:

Image src, tmp, dst;
if(argc < 2)
{
// Print help
std::cout << "Usage: ./build/neon_convolution [input_image.ppm]\n\n";
std::cout << "No input_image provided, creating a dummy 640x480 image\n";
// Initialize just the dimensions and format of your buffers:
src.allocator()->init(TensorInfo(640, 480, Format::U8));
}
else
{
ppm.open(argv[1]);
// Initialize just the dimensions and format of your buffers:
ppm.init_image(src, Format::U8);
}
// Initialize just the dimensions and format of the temporary and destination images:
tmp.allocator()->init(*src.info());
dst.allocator()->init(*src.info());
NEConvolution3x3 conv3x3;
NEConvolution5x5 conv5x5;
// Apply a Gaussian 3x3 filter to the source image followed by a Gaussian 5x5:
// The function will automatically update the padding information inside input and output to match its requirements
conv3x3.configure(&src, &tmp, gaussian3x3, 0 /* Let arm_compute calculate the scale */, BorderMode::UNDEFINED);
conv5x5.configure(&tmp, &dst, gaussian5x5, 0 /* Let arm_compute calculate the scale */, BorderMode::UNDEFINED);
// Now that the padding requirements are known we can allocate the images:
src.allocator()->allocate();
tmp.allocator()->allocate();
dst.allocator()->allocate();
// Fill the input image with the content of the PPM image if a filename was provided:
if(ppm.is_open())
{
ppm.fill_image(src);
}
//Execute the functions:
conv3x3.run();
conv5x5.run();
// Save the result to file:
if(ppm.is_open())
{
const std::string output_filename = std::string(argv[1]) + "_out.ppm";
save_to_ppm(dst, output_filename);
}
Note
It's important to call allocate after the function is configured: if the image / tensor is already allocated then the function will shrink its execution window instead of increasing the padding. (See below for more details).
• Manual padding / no padding / auto padding: You can allocate your images / tensors up front (before configuring your functions), in that case the function will use whatever padding is available and will shrink its execution window if there isn't enough padding available (Which will translates into a smaller valid region for the output
valid_region). If you don't want to manually set the padding but still want to allocate your objects upfront then you can use auto_padding.
Image src, dst;
// Use auto padding for the input:
// Use manual padding for the destination image
dst.info()->init(src.info()->tensor_shape(), Format::U8, strides_in_bytes, offset_first_element_in_bytes, total_size_in_bytes);
// Allocate all the images
src.allocator()->allocate();
dst.allocator()->allocate();
// Fill the input image with the content of the PPM image if a filename was provided:
fill_image(src);
NEGaussian3x3 gauss;
// Apply a Gaussian 3x3 filter to the source image (Note: if the padding provided is not enough then the execution window and valid region of the output will be shrunk)
gauss.configure(&src, &dst, BorderMode::UNDEFINED);
//Execute the functions:
gauss.run();
Warning
Some kernels need up to 3 neighbour values to calculate the value of a given pixel, therefore to be safe we use a 4 pixels padding all around the image and some kernels read and write up to 32 pixels at the time, therefore we add an extra 32 pixels of padding at the end of each row to be safe. As a result auto padded buffers waste a lot of memory and are less cache friendly. It is therefore recommended to use accurate padding or manual padding wherever possible.

#### Valid regions

Some kernels (like edge detectors for example) need to read values of neighbouring pixels to calculate the value of a given pixel, it is therefore not possible to calculate the values of the pixels on the edges.

Another case is: if a kernel processes 8 pixels per iteration then if the image's dimensions is not a multiple of 8 and not enough padding is available then the kernel will not be able to process the pixels near the right edge as a result these pixels will be left undefined.

In order to know which pixels have been calculated, each kernel sets a valid region for each output image or tensor

TensorInfo::valid_region(), ValidRegion
Attention
Valid regions and accurate padding have only been introduced in the library recently therefore not all the kernels and functions have been ported to use them yet. All the non ported kernels will set the ValidRegion equal to the TensorShape.

List of kernels which haven't been ported yet:

### Tensors

Tensors are multi-dimensional arrays made of up to Coordinates::num_max_dimensions dimensions.

A simple vector of numbers can be represented as a 1D tensor, an image is actually just a 2D tensor, a 3D tensor can be seen as an array of images, a 4D tensor as a 2D array of images, etc.

Note
Most algorithms process images (i.e a 2D slice of the tensor), therefore only padding along the X and Y axes is required (2D slices can be stored contiguously in memory).

### Images and Tensors description conventions

Image objects are defined by a Format and dimensions expressed as [width, height, batch]

Tensors are defined by a DataType plus a number of channels (Always expected to be 1 for now) and their dimensions are expressed as [width, height, feature_maps, batch].

In other words, the lower three dimensions of a tensor specify a single input in [width, height, feature_maps], while any other specified dimension represents a batch in the appropriate dimension space. For example, a tensor with dimensions [128, 128, 64, 16] represents a 1D batch space with 16 batches of 128 elements in width and height and 64 feature maps each. Each kernel specifies the expected layout of each of its tensors in its documentation.

Note
Unless specified otherwise in the kernel's or function's documentation all tensors and images parameters passed must have identical dimensions.
Unless specified otherwise in the kernel's or function's documentation the number of channels for tensors is expected to be 1 (For images, the number of channels is inferred from the Format).

### Working with Images and Tensors

In the case that no padding exists in the Image/Tensor object you can linearize the object memory and directly copy to/from it.

// Create a tensor object
Tensor tensor;
// Operate on tensor
...
// Copy results
// Copy tensor as a linear bulk of memory if no padding exists
{
std::copy_n(tensor.buffer(), tensor.info()->total_size(), dst);
}

On the other hand, in case of padding, each row should be carefully copied separately.

// Create an image object
Image img;
// Initialize image