ForwardTokenizer Class Reference

#include <forwardTokenizer.h>

Inheritance diagram for ForwardTokenizer:

Collaboration diagram for ForwardTokenizer:

List of all members.

Public Member Functions
	ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual	~ForwardTokenizer ()
virtual int	countTokens ()
virtual bool	hasMoreTokens () const
virtual std::string	nextToken ()
virtual double	progress () const

---

Detailed Description

Tokenizes a stream from the beginning to the end. _ _ _ _ _ _ _ |_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file forwardTokenizer.h.

---

Constructor & Destructor Documentation

ForwardTokenizer::ForwardTokenizer	(	std::istream &	stream,
		const std::string	blankspaces,
		const std::string	separators
	)

Definition at line 27 of file forwardTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

ForwardTokenizer::~ForwardTokenizer

(

)

[virtual]

Definition at line 36 of file forwardTokenizer.cpp.

---

Member Function Documentation

int ForwardTokenizer::countTokens

(

)

[virtual]

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 39 of file forwardTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

bool ForwardTokenizer::hasMoreTokens

(

)

const [virtual]

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 61 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the caller graph for this function:

std::string ForwardTokenizer::nextToken

(

)

[virtual]

Returns the next token.

Implements Tokenizer.

Definition at line 72 of file forwardTokenizer.cpp.

References endl(), Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the call graph for this function:

Here is the caller graph for this function:

double ForwardTokenizer::progress

(

)

const [virtual]

Returns progress percentage.

Implements Tokenizer.

Definition at line 151 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by main().

Here is the caller graph for this function:

---

The documentation for this class was generated from the following files:

• /build/buildd/presage-0.8.7/src/lib/core/tokenizer/forwardTokenizer.h
• /build/buildd/presage-0.8.7/src/lib/core/tokenizer/forwardTokenizer.cpp