Compute Library
 22.08
Framework Class Referencefinal

Main framework class. More...

#include <Framework.h>

Public Member Functions

std::set< InstrumentsDescriptionavailable_instruments () const
 Supported instrument types for benchmarking. More...
 
void init (const FrameworkConfig &config)
 Init the framework. More...
 
void push_suite (std::string name)
 Add a new test suite. More...
 
void pop_suite ()
 Remove innermost test suite. More...
 
template<typename T >
void add_test_case (std::string test_name, DatasetMode mode, TestCaseFactory::Status status)
 Add a test case to the framework. More...
 
template<typename T , typename D >
void add_data_test_case (std::string test_name, DatasetMode mode, TestCaseFactory::Status status, std::string description, D &&data)
 Add a data test case to the framework. More...
 
void add_test_info (std::string info)
 Add info string for the next expectation/assertion. More...
 
void clear_test_info ()
 Clear the collected test info. More...
 
bool has_test_info () const
 Check if any info has been registered. More...
 
void print_test_info (std::ostream &os) const
 Print test info. More...
 
void log_test_start (const TestInfo &info)
 Tell the framework that execution of a test starts. More...
 
void log_test_skipped (const TestInfo &info)
 Tell the framework that a test case is skipped. More...
 
void log_test_end (const TestInfo &info)
 Tell the framework that a test case finished. More...
 
void log_failed_expectation (const TestError &error)
 Tell the framework that the currently running test case failed a non-fatal expectation. More...
 
void log_info (const std::string &info)
 Print the debug information that has already been logged. More...
 
int num_iterations () const
 Number of iterations per test case. More...
 
void set_num_iterations (int num_iterations)
 Set number of iterations per test case. More...
 
bool throw_errors () const
 Should errors be caught or thrown by the framework. More...
 
void set_throw_errors (bool throw_errors)
 Set whether errors are caught or thrown by the framework. More...
 
bool stop_on_error () const
 Indicates if test execution is stopped after the first failed test. More...
 
void set_stop_on_error (bool stop_on_error)
 Set whether to stop execution after the first failed test. More...
 
bool error_on_missing_assets () const
 Indicates if a test should be marked as failed when its assets are missing. More...
 
void set_error_on_missing_assets (bool error_on_missing_assets)
 Set whether a test should be considered as failed if its assets cannot be found. More...
 
bool run ()
 Run all enabled test cases. More...
 
void set_test_result (TestInfo info, TestResult result)
 Set the result for an executed test case. More...
 
void print_test_results (Printer &printer) const
 Use the specified printer to output test results from the last run. More...
 
Profiler get_profiler () const
 Factory method to obtain a configured profiler. More...
 
void add_printer (Printer *printer)
 Set the printer used for the output of test results. More...
 
std::vector< TestInfotest_infos () const
 List of TestInfo's. More...
 
LogLevel log_level () const
 Get the current logging level. More...
 
void set_instruments_info (InstrumentsInfo instr_info)
 Sets instruments info. More...
 
bool configure_only () const
 Get the configure only flag. More...
 
bool new_fixture_call () const
 Return whether the new fixture has been called. More...
 
void set_new_fixture_call (bool val)
 Set the new fixture call flag. More...
 

Static Public Member Functions

static Frameworkget ()
 Access to the singleton. More...
 

Detailed Description

Main framework class.

Keeps track of the global state, owns all test cases and collects results.

Definition at line 95 of file Framework.h.

Member Function Documentation

◆ add_data_test_case()

void add_data_test_case ( std::string  test_name,
DatasetMode  mode,
TestCaseFactory::Status  status,
std::string  description,
D &&  data 
)
inline

Add a data test case to the framework.

Parameters
[in]test_nameName of the new test case.
[in]modeMode in which to include the test.
[in]statusStatus of the test case.
[in]descriptionDescription of data.
[in]dataData that will be used as input to the test.

Definition at line 380 of file Framework.h.

References FrameworkConfig::mode.

Referenced by TestCaseRegistrar< T >::TestCaseRegistrar().

381 {
382  // WORKAROUND for GCC 4.9
383  // The function should get *it which is tuple but that seems to trigger a
384  // bug in the compiler.
385  auto tmp = std::unique_ptr<DataTestCaseFactory<T, decltype(*std::declval<D>())>>(new DataTestCaseFactory<T, decltype(*std::declval<D>())>(current_suite_name(), std::move(test_name), mode, status,
386  std::move(description), *data));
387  _test_factories.emplace_back(std::move(tmp));
388 }

◆ add_printer()

void add_printer ( Printer printer)

Set the printer used for the output of test results.

Parameters
[in]printerPointer to a printer.

Definition at line 672 of file Framework.cpp.

Referenced by main(), and arm_compute::utils::run_example().

673 {
674  _printers.push_back(printer);
675 }

◆ add_test_case()

void add_test_case ( std::string  test_name,
DatasetMode  mode,
TestCaseFactory::Status  status 
)
inline

Add a test case to the framework.

Parameters
[in]test_nameName of the new test case.
[in]modeMode in which to include the test.
[in]statusStatus of the test case.

Definition at line 374 of file Framework.h.

Referenced by arm_compute::utils::run_example(), and TestCaseRegistrar< T >::TestCaseRegistrar().

375 {
376  _test_factories.emplace_back(std::make_unique<SimpleTestCaseFactory<T>>(current_suite_name(), std::move(test_name), mode, status));
377 }

◆ add_test_info()

void add_test_info ( std::string  info)

Add info string for the next expectation/assertion.

Parameters
[in]infoInfo string.

Definition at line 156 of file Framework.cpp.

157 {
158  _test_info.emplace_back(std::move(info));
159 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ available_instruments()

std::set< InstrumentsDescription > available_instruments ( ) const

Supported instrument types for benchmarking.

Returns
Set of all available instrument types.

Definition at line 100 of file Framework.cpp.

Referenced by CommonOptions::CommonOptions().

101 {
102  std::set<InstrumentsDescription> types;
103 
104  for(const auto &instrument : _available_instruments)
105  {
106  types.emplace(instrument.first);
107  }
108 
109  return types;
110 }

◆ clear_test_info()

void clear_test_info ( )

Clear the collected test info.

Definition at line 161 of file Framework.cpp.

Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().

162 {
163  _test_info.clear();
164 }

◆ configure_only()

bool configure_only ( ) const

Get the configure only flag.

Returns
The current configure only flag.

Definition at line 709 of file Framework.cpp.

710 {
711  return _configure_only;
712 }

◆ error_on_missing_assets()

◆ get()

◆ get_profiler()

Profiler get_profiler ( ) const

Factory method to obtain a configured profiler.

The profiler enables all instruments that have been passed to the init method.

Returns
Configured profiler to collect benchmark results.

Definition at line 643 of file Framework.cpp.

References Profiler::add(), arm_compute::test::framework::ALL, is_selected, and type.

Referenced by Framework::error_on_missing_assets(), and main().

644 {
645  Profiler profiler;
646 
647  const bool all_instruments = std::any_of(
648  _instruments.begin(),
649  _instruments.end(),
650  [](InstrumentsDescription type) -> bool { return type.first == InstrumentType::ALL; });
651 
652  auto is_selected = [&](InstrumentsDescription instrument) -> bool
653  {
654  return std::find_if(_instruments.begin(), _instruments.end(), [&](InstrumentsDescription type) -> bool {
655  const auto group = static_cast<InstrumentType>(static_cast<uint64_t>(type.first) & 0xFF00);
656  return (group == instrument.first) && (instrument.second == type.second);
657  })
658  != _instruments.end();
659  };
660 
661  for(const auto &instrument : _available_instruments)
662  {
663  if(all_instruments || is_selected(instrument.first))
664  {
665  profiler.add(instrument.second());
666  }
667  }
668 
669  return profiler;
670 }
std::pair< InstrumentType, ScaleFactor > InstrumentsDescription
Definition: Instruments.h:69
decltype(strategy::transforms) typedef type
const BatchNormalizationSelectorPtr is_selected

◆ has_test_info()

bool has_test_info ( ) const

Check if any info has been registered.

Returns
True if there is test info.

Definition at line 166 of file Framework.cpp.

167 {
168  return !_test_info.empty();
169 }

◆ init()

void init ( const FrameworkConfig config)

Init the framework.

See also
TestFilter::TestFilter for the format of the string to filter ids.
Parameters
[in]configFramework configuration meta-data.

Definition at line 130 of file Framework.cpp.

References FrameworkConfig::configure_only, FrameworkConfig::cooldown_sec, arm_compute::mlgo::parser::end(), FrameworkConfig::id_filter, FrameworkConfig::instruments, arm_compute::test::join(), FrameworkConfig::log_level, FrameworkConfig::mode, FrameworkConfig::name_filter, and FrameworkConfig::num_iterations.

Referenced by main(), and arm_compute::utils::run_example().

131 {
132  _test_filter.reset(new TestFilter(config.mode, config.name_filter, config.id_filter));
133  _num_iterations = config.num_iterations;
134  _log_level = config.log_level;
135  _cooldown_sec = config.cooldown_sec;
136  _configure_only = config.configure_only;
137 
138  _instruments = std::set<framework::InstrumentsDescription>(std::begin(config.instruments), std::end(config.instruments));
139 }
void end(TokenStream &in, bool &valid)
Definition: MLGOParser.cpp:290

◆ log_failed_expectation()

void log_failed_expectation ( const TestError error)

Tell the framework that the currently running test case failed a non-fatal expectation.

Parameters
[in]errorDescription of the error.

Definition at line 226 of file Framework.cpp.

References ARM_COMPUTE_ERROR_ON, TestCaseFactory::EXPECTED_FAILURE, TestResult::FAILED, TestError::level(), Printer::print_error(), TestResult::status, and TestInfo::status.

227 {
228  ARM_COMPUTE_ERROR_ON(_current_test_info == nullptr);
229  ARM_COMPUTE_ERROR_ON(_current_test_result == nullptr);
230 
231  const bool is_expected_failure = _current_test_info->status == TestCaseFactory::Status::EXPECTED_FAILURE;
232 
233  if(_log_level >= error.level())
234  {
235  func_on_all_printers([&](Printer * p)
236  {
237  p->print_error(error, is_expected_failure);
238  });
239  }
240 
241  _current_test_result->status = TestResult::Status::FAILED;
242 }
#define ARM_COMPUTE_ERROR_ON(cond)
If the condition is true then an error message is printed and an exception thrown.
Definition: Error.h:466
Status status
Execution status.
Definition: TestResult.h:74
TestCaseFactory::Status status
Test status.
Definition: Framework.h:83

◆ log_info()

void log_info ( const std::string &  info)

Print the debug information that has already been logged.

Parameters
[in]infoDescription of the log info.

Definition at line 244 of file Framework.cpp.

References arm_compute::test::framework::DEBUG, and Printer::print_info().

Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().

245 {
246  if(_log_level >= LogLevel::DEBUG)
247  {
248  func_on_all_printers([&](Printer * p)
249  {
250  p->print_info(info);
251  });
252  }
253 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ log_level()

LogLevel log_level ( ) const

Get the current logging level.

Returns
The current logging level.

Definition at line 698 of file Framework.cpp.

699 {
700  return _log_level;
701 }

◆ log_test_end()

void log_test_end ( const TestInfo info)

Tell the framework that a test case finished.

Parameters
[in]infoTest info.

Definition at line 206 of file Framework.cpp.

References arm_compute::test::framework::MEASUREMENTS, Printer::print_measurements(), Printer::print_profiler_header(), Printer::print_test_footer(), and arm_compute::test::framework::TESTS.

Referenced by Framework::error_on_missing_assets().

207 {
208  if(_log_level >= LogLevel::MEASUREMENTS)
209  {
210  func_on_all_printers([&](Printer * p)
211  {
212  p->print_profiler_header(_test_results.at(info).header_data);
213  p->print_measurements(_test_results.at(info).measurements);
214  });
215  }
216 
217  if(_log_level >= LogLevel::TESTS)
218  {
219  func_on_all_printers([](Printer * p)
220  {
221  p->print_test_footer();
222  });
223  }
224 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ log_test_skipped()

void log_test_skipped ( const TestInfo info)

Tell the framework that a test case is skipped.

Parameters
[in]infoTest info.

Definition at line 201 of file Framework.cpp.

References arm_compute::test::validation::info.

Referenced by Framework::error_on_missing_assets().

202 {
203  static_cast<void>(info);
204 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ log_test_start()

void log_test_start ( const TestInfo info)

Tell the framework that execution of a test starts.

Parameters
[in]infoTest info.

Definition at line 190 of file Framework.cpp.

References Printer::print_test_header(), and arm_compute::test::framework::TESTS.

Referenced by Framework::error_on_missing_assets().

191 {
192  if(_log_level >= LogLevel::TESTS)
193  {
194  func_on_all_printers([&](Printer * p)
195  {
196  p->print_test_header(info);
197  });
198  }
199 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ new_fixture_call()

bool new_fixture_call ( ) const

Return whether the new fixture has been called.

Returns
The current new fixture call flag.

Definition at line 714 of file Framework.cpp.

715 {
716  return _new_fixture_call;
717 }

◆ num_iterations()

int num_iterations ( ) const

Number of iterations per test case.

Returns
Number of iterations per test case.

Definition at line 255 of file Framework.cpp.

Referenced by Framework::set_num_iterations().

256 {
257  return _num_iterations;
258 }

◆ pop_suite()

void pop_suite ( )

Remove innermost test suite.

Warning
Cannot be used at execution time. It can only be used for registering test cases.

Definition at line 151 of file Framework.cpp.

Referenced by TestSuiteRegistrar::TestSuiteRegistrar().

152 {
153  _test_suite_name.pop_back();
154 }

◆ print_test_info()

void print_test_info ( std::ostream &  os) const

Print test info.

Parameters
[out]osOutput stream.

Definition at line 171 of file Framework.cpp.

References arm_compute::mlgo::parser::end(), arm_compute::utility::for_each(), and caffe_mnist_image_extractor::str.

Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().

172 {
173  if(!_test_info.empty())
174  {
175  os << "CONTEXT:\n";
176 
177  for(const auto &str : _test_info)
178  {
179  os << " " << str << "\n";
180  }
181  }
182 }

◆ print_test_results()

void print_test_results ( Printer printer) const

Use the specified printer to output test results from the last run.

This method can be used if the test results need to be obtained using a different printer than the one managed by the framework.

Parameters
[in]printerPrinter used to output results.

Definition at line 628 of file Framework.cpp.

References Printer::print_measurements(), Printer::print_profiler_header(), Printer::print_run_footer(), Printer::print_run_header(), Printer::print_test_footer(), and Printer::print_test_header().

629 {
630  printer.print_run_header();
631 
632  for(const auto &test : _test_results)
633  {
634  printer.print_test_header(test.first);
635  printer.print_profiler_header(test.second.header_data);
636  printer.print_measurements(test.second.measurements);
637  printer.print_test_footer();
638  }
639 
640  printer.print_run_footer();
641 }

◆ push_suite()

void push_suite ( std::string  name)

Add a new test suite.

Warning
Cannot be used at execution time. It can only be used for registering test cases.
Parameters
[in]nameName of the added test suite.
Returns
Name of the current test suite.

Definition at line 146 of file Framework.cpp.

Referenced by TestSuiteRegistrar::TestSuiteRegistrar().

147 {
148  _test_suite_name.emplace_back(std::move(name));
149 }
const char * name

◆ run()

bool run ( )

Run all enabled test cases.

Returns
True if all test cases executed successful.

Definition at line 543 of file Framework.cpp.

References CLKernelLibrary::clear_programs_cache(), CLScheduler::context(), TestResult::CRASHED, TestResult::DISABLED, arm_compute::mlgo::parser::end(), TestResult::EXPECTED_FAILURE, TestResult::FAILED, CLKernelLibrary::get(), CLScheduler::get(), TestCaseFactory::mode(), TestCaseFactory::name(), arm_compute::test::framework::NONE, arm_compute::opencl_is_available(), Printer::print_run_footer(), Printer::print_run_header(), CLScheduler::queue(), CLScheduler::set_context(), CLScheduler::set_queue(), arm_compute::test::framework::sleep_in_seconds(), TestCaseFactory::status(), TestResult::SUCCESS, and arm_compute::test::framework::TESTS.

Referenced by main(), and arm_compute::utils::run_example().

544 {
545  // Clear old test results
546  _test_results.clear();
547 
548  if(_log_level >= LogLevel::TESTS)
549  {
550  func_on_all_printers([](Printer * p)
551  {
552  p->print_run_header();
553  });
554  }
555 
556  const std::chrono::time_point<std::chrono::high_resolution_clock> start = std::chrono::high_resolution_clock::now();
557 
558  int id = 0;
559  int id_run_test = 0;
560 
561  for(auto &test_factory : _test_factories)
562  {
563  const std::string test_case_name = test_factory->name();
564  const TestInfo test_info{ id, test_case_name, test_factory->mode(), test_factory->status() };
565 
566  if(_test_filter->is_selected(test_info))
567  {
568 #ifdef ARM_COMPUTE_CL
569  // Every 100 tests, reset the OpenCL context to release the allocated memory
570  if(opencl_is_available() && (id_run_test % 100) == 0)
571  {
572  auto ctx_properties = CLScheduler::get().context().getInfo<CL_CONTEXT_PROPERTIES>(nullptr);
573  auto queue_properties = CLScheduler::get().queue().getInfo<CL_QUEUE_PROPERTIES>(nullptr);
574 
575  cl::Context new_ctx = cl::Context(CL_DEVICE_TYPE_DEFAULT, ctx_properties.data());
576  cl::CommandQueue new_queue = cl::CommandQueue(new_ctx, CLKernelLibrary::get().get_device(), queue_properties);
577 
579  CLScheduler::get().set_context(new_ctx);
580  CLScheduler::get().set_queue(new_queue);
581  }
582 #endif // ARM_COMPUTE_CL
583 
584  run_test(test_info, *test_factory);
585 
586  ++id_run_test;
587 
588  // Run test delay
589  sleep_in_seconds(_cooldown_sec);
590  }
591 
592  ++id;
593  }
594 
595  const std::chrono::time_point<std::chrono::high_resolution_clock> end = std::chrono::high_resolution_clock::now();
596 
597  if(_log_level >= LogLevel::TESTS)
598  {
599  func_on_all_printers([](Printer * p)
600  {
601  p->print_run_footer();
602  });
603  }
604 
605  auto runtime = std::chrono::duration_cast<std::chrono::seconds>(end - start);
606  std::map<TestResult::Status, int> results = count_test_results();
607 
608  if(_log_level > LogLevel::NONE)
609  {
610  std::cout << "Executed " << _test_results.size() << " test(s) ("
611  << results[TestResult::Status::SUCCESS] << " passed, "
612  << results[TestResult::Status::EXPECTED_FAILURE] << " expected failures, "
613  << results[TestResult::Status::FAILED] << " failed, "
614  << results[TestResult::Status::CRASHED] << " crashed, "
615  << results[TestResult::Status::DISABLED] << " disabled) in " << runtime.count() << " second(s)\n";
616  }
617 
618  int num_successful_tests = results[TestResult::Status::SUCCESS] + results[TestResult::Status::EXPECTED_FAILURE] + results[TestResult::Status::DISABLED];
619 
620  return (static_cast<unsigned int>(num_successful_tests) == _test_results.size());
621 }
static CLScheduler & get()
Access the scheduler singleton.
void set_context(cl::Context context)
Accessor to set the CL context to be used by the scheduler.
static CLKernelLibrary & get()
Access the KernelLibrary singleton.
void clear_programs_cache()
Clear the library&#39;s cache of binary programs.
cl::Context & context()
Accessor for the associated CL context.
Definition: CLScheduler.cpp:36
void end(TokenStream &in, bool &valid)
Definition: MLGOParser.cpp:290
cl::CommandQueue & queue()
Accessor for the associated CL command queue.
Definition: CLScheduler.cpp:43
void set_queue(cl::CommandQueue queue)
Accessor to set the CL command queue to be used by the scheduler.
Definition: CLScheduler.cpp:59
void sleep_in_seconds(float seconds)
Makes the calling thread to sleep for a specified number of seconds.
Definition: Utils.cpp:38
bool opencl_is_available()
Check if OpenCL is available.
Definition: OpenCL.cpp:188

◆ set_error_on_missing_assets()

void set_error_on_missing_assets ( bool  error_on_missing_assets)

Set whether a test should be considered as failed if its assets cannot be found.

Parameters
[in]error_on_missing_assetsTrue if a test should be marked as failed when its assets are missing.

Definition at line 285 of file Framework.cpp.

References Framework::error_on_missing_assets().

Referenced by main().

286 {
287  _error_on_missing_assets = error_on_missing_assets;
288 }
bool error_on_missing_assets() const
Indicates if a test should be marked as failed when its assets are missing.
Definition: Framework.cpp:290

◆ set_instruments_info()

void set_instruments_info ( InstrumentsInfo  instr_info)

Sets instruments info.

Note
TODO(COMPMID-2638) : Remove once instruments are transferred outside the framework.
Parameters
[in]instr_infoInstruments info to set

Definition at line 703 of file Framework.cpp.

References ARM_COMPUTE_ERROR_ON.

Referenced by main().

704 {
706  *instruments_info = instr_info;
707 }
std::unique_ptr< InstrumentsInfo > instruments_info
Definition: Framework.cpp:50
#define ARM_COMPUTE_ERROR_ON(cond)
If the condition is true then an error message is printed and an exception thrown.
Definition: Error.h:466

◆ set_new_fixture_call()

void set_new_fixture_call ( bool  val)

Set the new fixture call flag.

Parameters
[in]valValue to set for the flag

Definition at line 719 of file Framework.cpp.

720 {
721  _new_fixture_call = val;
722 }

◆ set_num_iterations()

void set_num_iterations ( int  num_iterations)

Set number of iterations per test case.

Parameters
[in]num_iterationsNumber of iterations per test case.

Definition at line 260 of file Framework.cpp.

References Framework::num_iterations().

261 {
262  _num_iterations = num_iterations;
263 }
int num_iterations() const
Number of iterations per test case.
Definition: Framework.cpp:255

◆ set_stop_on_error()

void set_stop_on_error ( bool  stop_on_error)

Set whether to stop execution after the first failed test.

Parameters
[in]stop_on_errorTrue if execution is going to be stopped after first failed test.

Definition at line 275 of file Framework.cpp.

References Framework::stop_on_error().

Referenced by main().

276 {
277  _stop_on_error = stop_on_error;
278 }
bool stop_on_error() const
Indicates if test execution is stopped after the first failed test.
Definition: Framework.cpp:280

◆ set_test_result()

void set_test_result ( TestInfo  info,
TestResult  result 
)

Set the result for an executed test case.

Parameters
[in]infoTest info.
[in]resultExecution result.

Definition at line 623 of file Framework.cpp.

Referenced by Framework::error_on_missing_assets().

624 {
625  _test_results.emplace(std::move(info), std::move(result));
626 }
ScaleKernelInfo info(interpolation_policy, default_border_mode, PixelValue(), sampling_policy, false)

◆ set_throw_errors()

void set_throw_errors ( bool  throw_errors)

Set whether errors are caught or thrown by the framework.

Parameters
[in]throw_errorsTrue if errors should be thrown.

Definition at line 265 of file Framework.cpp.

References Framework::throw_errors().

Referenced by main(), and arm_compute::utils::run_example().

266 {
267  _throw_errors = throw_errors;
268 }
bool throw_errors() const
Should errors be caught or thrown by the framework.
Definition: Framework.cpp:270

◆ stop_on_error()

bool stop_on_error ( ) const

Indicates if test execution is stopped after the first failed test.

Returns
True if the execution is going to be stopped after the first failed test.

Definition at line 280 of file Framework.cpp.

Referenced by Framework::set_stop_on_error().

281 {
282  return _stop_on_error;
283 }

◆ test_infos()

std::vector< TestInfo > test_infos ( ) const

List of TestInfo's.

Returns
Vector with all test ids.

Definition at line 677 of file Framework.cpp.

References TestInfo::name.

Referenced by main().

678 {
679  std::vector<TestInfo> ids;
680 
681  int id = 0;
682 
683  for(const auto &factory : _test_factories)
684  {
685  const TestInfo test_info{ id, factory->name(), factory->mode(), factory->status() };
686 
687  if(_test_filter->is_selected(test_info))
688  {
689  ids.emplace_back(std::move(test_info));
690  }
691 
692  ++id;
693  }
694 
695  return ids;
696 }

◆ throw_errors()

bool throw_errors ( ) const

Should errors be caught or thrown by the framework.

Returns
True if errors are thrown.

Definition at line 270 of file Framework.cpp.

Referenced by Framework::set_throw_errors().

271 {
272  return _throw_errors;
273 }

The documentation for this class was generated from the following files: