File Information
Library: Foundation
Package: Streams
Header: Poco/StreamTokenizer.h
Description
A stream tokenizer splits an input stream into a sequence of tokens of different kinds. Various token kinds can be registered with the tokenizer.
Inheritance
Known Derived Classes: Poco::CppParser::Tokenizer
Member Summary
Member Functions: addToken, attachToStream, next
Constructors
StreamTokenizer
Creates a StreamTokenizer with no attached stream.
StreamTokenizer
StreamTokenizer(
std::istream & istr
);
Creates a StreamTokenizer with no attached stream.
Destructor
~StreamTokenizer 
virtual ~StreamTokenizer();
Destroys the StreamTokenizer and deletes all registered tokens.
Member Functions
addToken
void addToken(
Token * pToken
);
Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. Comment and whitespace tokens will be marked as ignorable, which means that next() will not return them.
addToken
void addToken(
Token * pToken,
bool ignore
);
Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. If ignore is true, the token will be marked as ignorable, which means that next() will not return it.
attachToStream
void attachToStream(
std::istream & istr
);
Attaches the tokenizer to an input stream.
next
const Token * next();
Extracts the next token from the input stream. Returns a pointer to an EOFToken if there are no more characters to read. Returns a pointer to an InvalidToken if an invalid character is encountered. If a token is marked as ignorable, it will not be returned, and the next token will be examined. Never returns a NULL pointer. You must not delete the token returned by next().