Utilities

The SDS-Framework includes the following utilities that are implemented in Python.

SDSIO-Server: enables recording and playback of SDS data files via socket (TCP/IP), USB (Bulk transfer) or serial (UART) connection.
SDS-View: graphical data viewer for SDS data files.
SDS-Convert: convert SDS data files into CSV, Qeexo V2 CSV, or WAV format.
SDS-Check: check SDS data files for correctness and consistency.

Requirements

Python 3.9 or later with packages:
- ifaddr
- matplotlib
- numpy
- pandas
- pyyaml
- pyserial
- libusb1

Setup

Note

These utilities are located in the SDS-Framework pack installation folder, subfolder /utilities. Replace 2.0.0 with the version number of your SDS pack installation.

Perform the following steps to setup the Python environment for using the SDS utilities.

Install Python or verify the version with:

>python --version

Navigate in the folder SDS/utilities and install the required Python packages with pip:

>cd %CMSIS_PACK_ROOT%/ARM/SDS/2.0.0/utilities
>pip install -r requirements.txt

Add to the system Path environment variable the path to the %CMSIS_PACK_ROOT%/ARM/SDS/2.0.0/utilities folder.

Notes

%CMSIS_PACK_ROOT% is just a placeholder for the Pack location on your local PC. The Path variable must be extended by the absolute path to the utilities folder.
Do not set the variable PYTHONPATH as this may conflict with the Python scripts that are used in AVH FVP models. AVH FVP use Python version 3.9 which may create a conflict.

Windows

On Windows, ensure that the environment variable PATHEXT contains the extension .PY.

Tip

When the Path environment variable is configured, you may simply start the utilities by using its name. For example entering >sdsio-server starts the utility.

SDSIO-Server

The Python utility SDSIO-Server enables recording and playback of SDS data files via socket (TCP/IP), USB (Bulk transfer), or serial (UART) connection. It communicates with the target using these SDSIO Client interfaces:

serial/usart for serial communication via CMSIS-Driver USART.
socket for TCP/IP communication via IoT Socket using MDK-Middleware, LwIP, or CMSIS-Driver WiFi.
usb/bulk for communication via USB Bulk transfer using MDK-Middleware.

The SDS data stream is recorded to files with the following naming convention:

<name>.<index>.sds

<name> is the name of the I/O stream specified with the function sdsRecOpen or sdsPlayOpen on the target.
<index> is the zero-based index which is incremented for each subsequent recording.
For more details see Filenames section

The data content of the <name>.<index>.sds is described with metadata file <name>.sds.yml in YAML format.

Usage

Setup the Python environment.
Depending on the SDS interface used on the target use either Serial Mode, Socket Mode or USB Mode as described below.
The SDSIO_Server terminates with Ctrl+C.

Serial Mode

usage: sdsio-server.py serial [-h]  
                              -p <Serial Port>  
                              [--baudrate <Baudrate>]  
                              [--parity <Parity>]  
                              [--stopbits <Stop bits>]  
                              [--connect-timeout <Timeout>]  
                              [--workdir <Work dir>]

optional arguments:
  -h, --help                   show this help message and exit

required:
  -p <Serial Port>             Serial port

optional:
  --baudrate <Baudrate>        Baudrate (default: 115200)
  --parity <Parity>            Parity: N=None, E=Even, O=Odd, M=Mark, S=Space (default: N)
  --stopbits <Stop bits>       Stop bits: 1, 1.5, 2 (default: 1)
  --connect-timeout <Timeout>  Serial port connection timeout in seconds (default: no timeout)
  --workdir <Work dir>         Directory for SDS files (default: current directory)

Example:

python sdsio-server.py serial -p COM0 --baudrate 115200 --workdir ./work_dir

Socket Mode

usage: sdsio-server.py socket [-h]  
                              [--ipaddr <IP> | --interface <Interface>]  
                              [--port <TCP Port>]  
                              [--workdir <Work dir>]

optional arguments:
  -h, --help               show this help message and exit

optional:
  --ipaddr <IP>            Server IP address (cannot be used with --interface)
  --interface <Interface>  Network interface (cannot be used with --ipaddr)
  --port <TCP Port>        TCP port (default: 5050)
  --workdir <Work dir>     Directory for SDS files (default: current directory)

Note

The --ipaddr and --interface options are mutually exclusive.
SDSIO Server only supports IPv4 addresses.

Example:

For Microsoft Windows (using default computer IP):

python sdsio-server.py socket --workdir ./work_dir

For Linux:

python sdsio-server.py socket --interface eth0 --workdir ./work_dir

USB Mode

usage: sdsio-server.py usb [-h]  
                           [--workdir <Work dir>]  
                           [--high-priority]

optional arguments:
  -h, --help            show this help message and exit

optional:
  --workdir <Work dir>  Directory for SDS files (default: current directory)
  --high-priority       Enable high-priority threading for USB server (default: off)

Note

For more reliable operation at higher data transfer rates, it is recommended to enable the --high-priority option. This increases the thread priority of the SDSIO-Server process.
When using --high-priority, elevated privileges are required depending on your operating system:
- Windows: Run the Python script as an administrator.
- macOS/Linux: Execute the script with sudo or ensure the user has sufficient permissions.

Example:

python sdsio-server.py usb --workdir ./work_dir

SDS-View

The Python utility SDS-View outputs a time-based plot of SDS data files (<name>.<index>.sds) based on the meta-data file (<name>.sds.yml).

The horizontal time scale is derived from the number of data points in a recording and frequency provided in the metadata description. All plots form a single recording will be displayed on the same figure (shared vertical scale).

If there are 3 values described in the metadata file, an optional 3D view may be displayed.

Limitations

Data in recording must all be of the same type (float, uint32_t, uint16_t, ...)

Usage

Setup the Python environment.
Invoke the tool as explained below.

usage: sds-view.py [-h]  
                   -y <yaml_file>  
                   -s <sds_file> [<sds_file> ...]  
                   [--3D]

View SDS data

options:
  -h, --help                      show this help message and exit

required:
  -y <yaml_file>                  YAML sensor description file
  -s <sds_file> [<sds_file> ...]  SDS data recording file

optional:
  --3D                            Plot 3D view in addition to normal 2D

Example:

python sds-view.py -y test/Gyroscope.sds.yml -s test/Gyroscope.0.sds

Example display:

Example output

SDS-Convert

The Python utility SDS-Convert converts SDS data files to selected format based on description in metadata (YAML) files.

Usage

Setup the Python environment.
Depending on the required format use the tool as shown below.

Audio WAV

The audio_wav mode converts raw microphone data from .sds files into a standard RIFF/WAV file using linear PCM encoding. The conversion process involves appending a WAV header, generated from parameters specified in the associated metadata .yml file, to the raw audio data extracted from the .sds stream. The metadata defines essential audio parameters such as channel configuration (mono or stereo), sample rate (frame rate), and sample width (bit depth).

usage: sds-convert.py audio_wav [-h]  
                                -i <input_file> [<input_file> ...]  
                                -o <output_file>  
                                [-y <yaml_file> [<yaml_file> ...]]

options:
  -h, --help                          show this help message and exit

required:
  -i <input_file> [<input_file> ...]  Input file
  -o <output_file>                    Output file

optional:
  -y <yaml_file> [<yaml_file> ...]    YAML sensor description file

Note

The metadata and SDS data file pairs must be passed as arguments in the same order to decode the data correctly.
The tool expects the SDS stream to be strictly audio - no header markers or custom formatting.

Example of metadata yml file for mono microphone:

sds:
  name: Microphone
  description: Mono microphone with 16kHz sample rate
  frequency: 16000
  content:
  - value: Mono
    type: int16_t

Example of metadata yml file for stereo microphone:

sds:
  name: Microphone
  description: Stereo microphone with 16kHz sample rate
  frequency: 16000
  content:
  - value: Left channel
    type: int16_t
  - value: Right channel
    type: int16_t

Example:

python sds-convert.py audio_wav -i Microphone.0.sds -o microphone.wav -y Microphone.sds.yml

Simple CSV

The simple_csv mode converts sensor data from .sds files into a human-readable CSV format. This mode is designed for exporting data from a single sensor. If the sensor has multiple channels, each channel will appear as a separate column in the output CSV.

Timestamps are represented in floating-point format, in seconds. Using the --normalize flag causes all timestamps in the input file to be offset so that the first timestamp is 0.

Users may specify a time range selection of the input data to be processed using the following flags:

--start-timestamp <timestamp>: Starting input data timestamp in floating-point format, in seconds.
--stop-timestamp <timestamp>: Stopping input data timestamp in floating-point format, in seconds.

usage: sds-convert.py simple_csv [-h]  
                                 -i <input_file> [<input_file> ...]  
                                 -o <output_file>  
                                 [-y <yaml_file> [<yaml_file> ...]]  
                                 [--normalize]  
                                 [--start-timestamp <timestamp>]  
                                 [--stop-timestamp <timestamp>]

options:
  -h, --help                          show this help message and exit

required:
  -i <input_file> [<input_file> ...]  Input file
  -o <output_file>                    Output file

optional:
  -y <yaml_file> [<yaml_file> ...]    YAML sensor description file
  --normalize                         Normalize timestamps so they start with 0
  --start-timestamp <timestamp>       Starting input data timestamp, in seconds (default: None)
  --stop-timestamp <timestamp>        Stopping input data timestamp, in seconds (default: None)

Note

The metadata and SDS data file pairs must be passed as arguments in the same order to decode the data correctly.
Current implementation assumes that the tick frequency is 1000 Hz and does not use the tick-frequency value from the metadata file.

Example of metadata yml file for gyroscope:

sds:
  name: Gyroscope
  description: Gyroscope with 1667Hz sample rate
  frequency: 1667
  content:
  - value: x
    type: int16_t
    scale: 0.07
    unit: dps
  - value: y
    type: int16_t
    scale: 0.07
    unit: dps
  - value: z
    type: int16_t
    scale: 0.07
    unit: dps

Example:

python sds-convert.py simple_csv -i Gyroscope.0.sds -o gyroscope_simple.csv -y Gyroscope.sds.yml --normalize --start-tick 0.2 --stop-tick 0.3

Qeexo V2 CSV

The qeexo_v2_csv mode converts sensor data from .sds files into a Qeexo V2 CSV format.

Link to Qeexo V2 CSV format specification.

Timestamps are represented in integer format, in milliseconds. Using the --normalize flag causes all timestamps in the input file to be offset so that the first timestamp is 0.

Users may specify a time range selection of the input data to be processed using the following flags:

--start-timestamp <timestamp>: Starting input data timestamp in integer format, in milliseconds.
--stop-timestamp <timestamp>: Stopping input data timestamp in integer format, in milliseconds.

By default, the output file will have raw timestamps in integer format, in milliseconds. The default output timestamp interval is set to 50 ms. To override this setting use the --interval <ms> flag, where <ms> is the desired interval in milliseconds.

An optional label can be added to the output by providing a string argument to the --label <text> flag. This <text> will populate the label column in the output file.

usage: sds-convert.py qeexo_v2_csv [-h] -i <input_file> [<input_file> ...]  
                                   -o <output_file>  
                                   [-y <yaml_file> [<yaml_file> ...]]  
                                   [--normalize]  
                                   [--start-timestamp <timestamp>]  
                                   [--stop-timestamp <timestamp>]  
                                   [--label 'label']  
                                   [--interval <interval>]  
                                   [--sds_index <sds_index>]

options:
  -h, --help                          show this help message and exit

required:
  -i <input_file> [<input_file> ...]  Input file
  -o <output_file>                    Output file

optional:
  -y <yaml_file> [<yaml_file> ...]    YAML sensor description file
  --normalize                         Normalize timestamps so they start with 0
  --start-timestamp <timestamp>       Starting input data timestamp, in ms (default: None)
  --stop-timestamp <timestamp>        Stopping input data timestamp, in ms (default: None)
  --label 'label'                     Qeexo class label for sensor data (default: None)
  --interval <interval>               Qeexo timestamp interval, in ms (default: 50)
  --sds_index <sds_index>             SDS file index to write (default: <sensor>.0.sds)

Note

The metadata and SDS data file pairs must be passed as arguments in the same order to decode the data correctly.
Current implementation assumes that the tick frequency is 1000 Hz and does not use the tick-frequency value from the metadata file.

Example of metadata yml file for accelerometer:

sds:
  name: Accelerometer
  description: Accelerometer with 1667Hz sample rate
  frequency: 1667
  content:
  - value: x
    type: int16_t
    scale: 0.000061
    unit: G
  - value: y
    type: int16_t
    scale: 0.000061
    unit: G
  - value: z
    type: int16_t
    scale: 0.000061
    unit: G

Examples:

Convert SDS data files to Qeexo V2 CSV files:

python sds-convert.py qeexo_v2_csv -i Gyroscope.0.sds Accelerometer.0.sds -o sensor_fusion.csv -y Gyroscope.sds.yaml Accelerometer.sds.yaml --normalize --start-tick 200 --stop-tick 300

Convert Qeexo V2 CSV files to SDS data files:

python sds-convert.py qeexo_v2_csv -i accelerometer_data.csv -o accelerometer.sds

SDS-Check

The Python utility SDS-Check checks SDS data files for correctness and consistency.

The following checks are performed:

Size consistency check: data size of all records should match the size of the SDS file.
Timestamp consistency check: verify that timestamps of the records are in ascending order.
Jitter check: print the record with the largest deviation from the average timestamp interval.
Delta time check: find the record with the largest timestamps difference from the following record.
Duplicate timestamp check: find records that have identical timestamps.

Usage

Setup the Python environment.
Invoke the tool as explained below.

usage: sds-check.py [-h] -s <sds_file>

SDS data validation

options:
  -h, --help     show this help message and exit

required:
  -s <sds_file>  SDS data recording file

Example:

python sds-check.py -s Accelerometer.0.sds
File     : Accelerometer.0.sds
DataSize : 156.020 bytes
Records  : 289
BlockSize: 532 bytes
Largest  : 552 bytes
Smallest : 462 bytes
Interval : 50 ms
DataRate : 10.640 byte/s
Jitter   : 0 ms
Validation passed

Note

The time values assume an tick frequency of 1000 Hz.

Summary Report

After processing the SDS data file, the SDS-Check utility prints a summary report with statistics:

DataSize: total size of the data in bytes
Records: total number of records
BlockSize: average block size of a data record
Largest: largest block size, if different from the average block size
Smallest: smallest block size, if different from the average block size
Interval: time interval of the recording in milliseconds
DataRate: recorded data rate in bytes per second
Jitter: deviation from the expected timestamps
DeltaTime: largest difference of the neighboring timestamps, if deviating from the recording interval (optional)
DupStamps: number of duplicated timestamps, if found

Size consistency check

This check processes the SDS data records and calculates the total size of the SDS data. It is the sum of all data records (header + data). This data size should match the size of the SDS file.

If the sizes do not match this error is printed:

Error: File size mismatch. Expected 360 bytes, but file contains 363 bytes.

Timestamp consistency check

This check processes the SDS records and ensures that the timestamps recorded in the records are arranged in ascending order. If the utility detects that the timestamp of the subsequent data record is lower than the current one, this error is printed:

Error: Timestamp not in ascending order in record 23.

Jitter check

This check processes the SDS data records and searches for a maximum deviation of the recorded timestamps from the expected ones. If the deviation is found, the maximum deviation is evaluated as jitter and together with record number printed out in the summary report.

File     : Gyroscope.0.sds
DataSize : 153.334 bytes
Records  : 284
BlockSize: 532 bytes
Largest  : 606 bytes
Smallest : 444 bytes
Interval : 50 ms
DataRate : 10.640 byte/s
Jitter   : 0 ms
Validation passed

Delta time check

This check processes the SDS records and tries to find the largest difference in timestamps between two neighboring records, called DeltaTime.

For normally recorded files, the delta time and the recording interval are identical, so no information about the delta time status is printed. If the delta time and the recording interval are not identical, i.e. a difference is detected, the DeltaTime together with record number is printed out in the summary report.

File     : Temperature.0.sds
DataSize : 360 bytes
Records  : 30
BlockSize: 4 bytes
Interval : 1.024 ms
DataRate : 4 byte/s
Jitter   : 59 ms, record 19
DeltaTime: 1.050 ms, record 2
Validation passed

This is not an error, but a report of an anomaly. If the time delta is significantly larger than the sampling interval, e.g. many times longer, this may indicate that one or more data records are missing from the recorded file.

Duplicate timestamp check

This check processes the SDS records in search of duplicated timestamps. This means that the same timestamp is used in several consecutive data records.

This may indicate that the recording loop in an embedded application is not set up correctly. It is also possible that duplicate timestamps are caused by unexpected thread delays in the embedded application.

Duplicate timestamps are unusual in typical recording files. If multiple timestamps with the same value are found in the SDS file, DupStamps will be added in the summary report.

File     : DataInput.0.sds
DataSize : 17.509.440 bytes
Records  : 47.580
BlockSize: 360 bytes
Interval : 1 ms
DataRate : 360.000 byte/s
Jitter   : 4 ms, record 4
DeltaTime: 5 ms, record 5
DupStamps: 4, record 1
Validation passed

This is not an error, but a report of an anomaly. The report contains the number of records with the same timestamp and the position in the SDS file where the anomaly was detected (record number).

Note

Only the first occurrence of a duplicate timestamp is reported.