Compute Library
 21.11
TokenStream Class Reference

A stream of token. More...

#include <MLGOParser.h>

Public Member Functions

 TokenStream (std::istream &s, const std::string &delims=",\)
 Constructor. More...
 
 operator bool () const
 Check if there're more (non-End) Tokens. More...
 
Token take ()
 Get and pop off the current token. More...
 
Token peek (size_t i=0)
 Peek the next ith token. More...
 
CharPosition current_pos () const
 Get the position of the current token. More...
 

Static Public Attributes

static constexpr size_t max_look_ahead = 10
 

Detailed Description

A stream of token.

Definition at line 124 of file MLGOParser.h.

Constructor & Destructor Documentation

◆ TokenStream()

TokenStream ( std::istream &  s,
const std::string &  delims = ",\n" 
)

Constructor.

Parameters
[in]sInput stream
[in]delimsDelimiter characters packed in a string. Each char from the string can be used as a delim on its own

Definition at line 111 of file MLGOParser.cpp.

Referenced by arm_compute::mlgo::parser::parse_mlgo().

112  : _delims{ delims }, _istream{ s }, _tokens{}, _lookahead_pos{}
113 {
114  read();
115 }

Member Function Documentation

◆ current_pos()

CharPosition current_pos ( ) const
inline

Get the position of the current token.

Returns
CharPosition

Definition at line 162 of file MLGOParser.h.

References arm_compute::mlgo::parser::parse_mlgo().

163  {
164  return _tokens.front().pos;
165  }

◆ operator bool()

operator bool ( ) const
explicit

Check if there're more (non-End) Tokens.

Returns
true If there are more tokens
false If reached end of stream (only End token)

Definition at line 117 of file MLGOParser.cpp.

References ARM_COMPUTE_ERROR_ON_MSG.

118 {
119  ARM_COMPUTE_ERROR_ON_MSG(_tokens.empty(), "TokenStream can never be empty");
120  return !reached_end();
121 }
#define ARM_COMPUTE_ERROR_ON_MSG(cond, msg)
Definition: Error.h:456

◆ peek()

Token peek ( size_t  i = 0)

Peek the next ith token.

Parameters
[in]iThe next ith token. i < max_look_ahead.
Returns
Token

Definition at line 134 of file MLGOParser.cpp.

References ARM_COMPUTE_ERROR_ON_MSG, and TokenStream::max_look_ahead.

Referenced by arm_compute::mlgo::parser::accept_l_list(), arm_compute::mlgo::parser::accept_r_list(), and arm_compute::mlgo::parser::accept_text().

135 {
136  ARM_COMPUTE_ERROR_ON_MSG(_tokens.empty(), "TokenStream can never be empty");
137  ARM_COMPUTE_ERROR_ON_MSG(i >= max_look_ahead, "TokenStream: Exceeding max look ahead");
138  // NOTE: If i exceeds the stream (_istream.eof()), read() automatically appends a End token at the end
139  while(_istream && _tokens.size() <= i)
140  {
141  read();
142  }
143  size_t ind = std::min(i, _tokens.size() - 1);
144  return _tokens[ind];
145 }
#define ARM_COMPUTE_ERROR_ON_MSG(cond, msg)
Definition: Error.h:456
static constexpr size_t max_look_ahead
Definition: MLGOParser.h:128

◆ take()

Token take ( )

Get and pop off the current token.

Returns
Token

Definition at line 123 of file MLGOParser.cpp.

References ARM_COMPUTE_ERROR_ON_MSG, and tf_frozen_model_extractor::t.

Referenced by arm_compute::mlgo::parser::accept_l_list(), arm_compute::mlgo::parser::accept_r_list(), arm_compute::mlgo::parser::accept_text(), arm_compute::mlgo::parser::bool_val(), arm_compute::mlgo::parser::end(), arm_compute::mlgo::parser::float_val(), arm_compute::mlgo::parser::heuristic_type(), arm_compute::mlgo::parser::int_val(), and arm_compute::mlgo::parser::text_val().

124 {
125  ARM_COMPUTE_ERROR_ON_MSG(_tokens.empty(), "TokenStream can never be empty");
126  Token t = _tokens.front();
127  _tokens.pop_front();
128  if(_tokens.empty())
129  {
130  read();
131  }
132  return t;
133 }
#define ARM_COMPUTE_ERROR_ON_MSG(cond, msg)
Definition: Error.h:456

Field Documentation

◆ max_look_ahead

constexpr size_t max_look_ahead = 10
static

Definition at line 128 of file MLGOParser.h.

Referenced by TokenStream::peek().


The documentation for this class was generated from the following files: