Assembly kernels
Arm Compute Library contains a collection of highly optimized assembly kernels for Arm® A profile architecture. At runtime the library selects the best kernel based on the CPU detected. For example if the CPU supports the dot product instruction the library will choose a GEMM kernel which uses the dot product instruction. There are various kernels using Neon™ and architecture extensions like FP16, Dot product, SVE, SVE2 and SME.
For example, some assembly kernels are located in the folders:
- src/core/NEON/kernels/arm_gemm/kernels
- src/core/NEON/kernels/arm_gemm/pooling
- src/core/NEON/kernels/arm_conv/depthwise
The assembly kernels are written using assembly mnemonics and the .inst directive which inserts the machine code to the output directly.
Below you can see a code block from one of the kernels in the library which uses the .inst directive to generate the sdot instruction. This code can be found in the kernel src/core/NEON/kernels/arm_gemm/kernels/a64_hybrid_s8qa_dot_4x16/a55.cpp
".inst 0x4f80eb10 // sdot v16.4s, v24.16b, v0.4b[2]\n"
".inst 0x4f81eb14 // sdot v20.4s, v24.16b, v1.4b[2]\n"
" ldr d24, [x12, #0xf0]\n"
" ldr x20, [x12, #0xf8]\n"
" .inst 0x4f80ebd1 // sdot v17.4s, v30.16b, v0.4b[2]\n"
" .inst 0x4f81ebd5 // sdot v21.4s, v30.16b, v1.4b[2]\n"
" mov v27.d[1], x23\n"
" .inst 0x4f80ebb2 // sdot v18.4s, v29.16b, v0.4b[2]\n"
" mov v26.d[1], x22\n"
" .inst 0x4f81ebb6 // sdot v22.4s, v29.16b, v1.4b[2]\n"
" mov v25.d[1], x21\n"
" .inst 0x4f80eb93 // sdot v19.4s, v28.16b, v0.4b[2]\n"
" mov v24.d[1], x20\n"
" .inst 0x4f81eb97 // sdot v23.4s, v28.16b, v1.4b[2]\n"
" add x9, x9, #0x10\n"
" add x28, x28, #0x10\n"
" add x12, x12, #0x100\n"
" .inst 0x4fa0eb70 // sdot v16.4s, v27.16b, v0.4b[3]\n"
" .inst 0x4fa1eb74 // sdot v20.4s, v27.16b, v1.4b[3]\n"
" .inst 0x4fa0eb51 // sdot v17.4s, v26.16b, v0.4b[3]\n"
" .inst 0x4fa1eb55 // sdot v21.4s, v26.16b, v1.4b[3]\n"
Note that every occurrence of .inst is accompanied by a comment with the original opcode for readability purposes.
The reason for using the opcodes instead of the mnemonic is that this approach will work on any toolchain, including the ones without support for the dot product mnemonic. The .inst directive is used to generate many other instructions and ensuring the code will compile on older toolchains that do not support them.
Windows
A Window represents a workload to execute, it can handle up to Coordinates::num_max_dimensions dimensions. Each dimension is defined by a start, end and step.
It can split into subwindows as long as all the following rules remain true for all the dimensions:
- max[n].start() <= sub[n].start() < max[n].end()
- sub[n].start() < sub[n].end() <= max[n].end()
- max[n].step() == sub[n].step()
- (sub[n].start() - max[n].start()) % max[n].step() == 0
- (sub[n].end() - sub[n].start()) % max[n].step() == 0
Kernels
Each implementation of the IKernel interface (base class of all the kernels in the core library) works in the same way:
OpenCL kernels:
MyKernel kernel;
kernel.configure(
input, output, option0, option1);
const Window& max_window = kernel.window();
kernel.run( q, max_window );
q.finish();
Neon / CPP kernels:
MyKernel kernel;
kernel.configure(
input, output, option0, option1);
const Window& max_window = kernel.window();
kernel.run( max_window );
Multi-threading
The previous section shows how to run a Arm® Neon™ / CPP kernel in the current thread, however if your system has several CPU cores, you will probably want the kernel to use several cores. Here is how this can be done:
info.cpu_info = &_cpu_info;
const Window &max_window = kernel->window();
const unsigned int num_iterations = max_window.num_iterations(split_dimension);
info.num_threads = std::min(num_iterations, _num_threads);
if(num_iterations == 0)
{
return;
}
if(!kernel->is_parallelisable() ||
info.num_threads == 1)
{
kernel->run(max_window,
info);
}
else
{
auto thread_it = _threads.begin();
for(;
t <
info.num_threads - 1; ++
t, ++thread_it)
{
Window win = max_window.split_window(split_dimension,
t,
info.num_threads);
thread_it->start(kernel, win,
info);
}
Window win = max_window.split_window(split_dimension,
t,
info.num_threads);
try
{
for(auto &thread : _threads)
{
thread.wait();
}
}
catch(const std::system_error &e)
{
std::cerr << "Caught system_error with code " << e.code() << " meaning " << e.what() << '\n';
}
}
This is a very basic implementation which was originally used in the Arm® Neon™ runtime library by all the Arm® Neon™ functions.
- See also
- CPPScheduler
- Note
- Some kernels need some local temporary buffer to perform their calculations. In order to avoid memory corruption between threads, the local buffer must be of size:
memory_needed_per_thread * num_threads
and a unique thread_id between 0 and num_threads must be assigned to the ThreadInfo object passed to the run
function.
OpenCL kernel library
All OpenCL kernels used by the library are built and stored in CLKernelLibrary. If the library is compiled with embed_kernels=0 the application can set the path to the OpenCL kernels by calling CLKernelLibrary::init(), by default the path is set to "./cl_kernels"