What is Behavioural Analysis?

Behavioural analysis of a given system requires building an understanding of the overall system state relative to the state of specific heuristics of interest. With such an understanding, a technique to automatically detect expected state changes in response to varying input parameters is very valuable. This is especially true when there are a large number of factors that can change the behaviour of the system and testing all possible permutations of these factors is hard. Presented here is a toolkit that provides an assertion based mechanism to help evaluate system performance and correct operation.

The Behavioural Analysis and Regression Toolkit is based on TRAPpy. The toolkit is targeted at the Linux kernel and it's primary goal is to help assert behaviours of interest using the FTrace output from the Linux kernel

Target Audience

The framework caters to a wide range of users spanning developers interested in deep observation of system behaviours as well as test engineers interested in automating the testing of "difficult to test" system behaviours.

Kernel Developers: Making sure that the code that you are writing is doing the right thing.

Performance Engineers: Plotting/Asserting performance behaviours between different revisions of the kernel.

Quality Assurance/Release Engineers: Verifying behaviours when different components/patches are integrated.

Installation

Clone the BART and TRAPpy repos

git clone https://github.com/ARM-software/bart.git
git clone https://github.com/ARM-software/trappy.git

Add the directories to your PYTHONPATH

export PYTHONPATH=$BASE_DIR/bart:$BASE_DIR/trappy:$PYTHONPATH

Install dependencies

apt install ipython-notebook python-pandas

IPython notebook is a web based interactive python programming interface. It is required if you plan to use interactive plotting in BART.

Trace Analysis Language

BART also provides a generic Trace Analysis Language, which allows the user to construct complex relation statements on Linux kernel trace data and assert their expected behaviours. As an example, the usage of this language's Analyzer module can be seen for evaluating thermal behaviours with the Linux kernel's thermal management frameworkhere

Specialized Modules

Thermal Assertions

This module provides assertions specific to the thermal subsystem and the goal is to assert the temperature control achieved and use these assertions for potential regressions in temperature control. One interesting assertion is, assertThermalResidency which accepts a temperature range and checks if the time spent in the given temperature is between expected limits.

assertThermalResidency(between_threshold,     # A boolean function that compares the
   					      # result with the expected value

		       expected_value,	      # Expected value of residency
					      # in temperature range

		       (low_temp, high_temp)  # Temperature range
)

Scheduler Assertions

The Linux scheduler is complicated and susceptible to regressions over kernel revisions. Fixing or adding one area of the scheduler could have unknown and subtle repercussions in a different heuristic. We have added a scheduler specific module which allows developers to assert points of interest and then calculate various parameters like the duty cycle, period, execution time, residency of tasks over a CPU or a cluster for a test workload. It also features an assertion API to check if a task context switched between two CPUs/Clusters in a given window of time. Here is a short example.

Consider a task, cpuhog, which has alternate phases of high and low duty cycles:

Scroll on the Plot area to Zoom. Click and Drag to Pan Zoom.

We have a dual cluster asymmetric hardware. The topology can be expressed as:

BIG = [1,2]
LITTLE = [0,3,4,5]
clusters = [BIG, LITTLE]

topology = Topology(clusters=clusters)

We can then check the behaviours of the task as follows:

s = SchedAssert(trace_file, topology, execname="cpuhog")

Assert the switch of task between clusters

s.assertSwitch("cluster",           # Topological Level
	       BIG,	            # From
	       LITTLE,              # To
	       window=(11.5, 12.5)  # Time Window
)

Assert the duty cycle of the task in a window of time

s.assertDutyCycle(10,		     # Expected Value of DutyCycle (%)
		 between_threshold,  # A boolean comparator function
		 window=(0, 1)      # Window of time
)

A notebook explaining the usage of the framework for asserting the deadline scheduler behaviours can be seen here