ReverseTokenizer Class Reference

#include <reverseTokenizer.h>

Inheritance diagram for ReverseTokenizer:

Collaboration diagram for ReverseTokenizer:

List of all members.

Public Member Functions
	ReverseTokenizer (std::istream &stream, const std::string blanks, const std::string separs)
virtual	~ReverseTokenizer ()
virtual int	countTokens ()
virtual bool	hasMoreTokens () const
virtual std::string	nextToken ()
virtual double	progress () const

---

Detailed Description

Tokenizes a stream from the end to the beginning. _ _ _ _ _ _ _ |_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file reverseTokenizer.h.

---

Constructor & Destructor Documentation

ReverseTokenizer::ReverseTokenizer	(	std::istream &	stream,
		const std::string	blanks,
		const std::string	separs
	)

Definition at line 27 of file reverseTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

ReverseTokenizer::~ReverseTokenizer

(

)

[virtual]

Definition at line 44 of file reverseTokenizer.cpp.

---

Member Function Documentation

int ReverseTokenizer::countTokens

(

)

[virtual]

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 47 of file reverseTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

bool ReverseTokenizer::hasMoreTokens

(

)

const [virtual]

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 69 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

Referenced by countTokens().

Here is the caller graph for this function:

std::string ReverseTokenizer::nextToken

(

)

[virtual]

Returns the next token.

Implements Tokenizer.

Definition at line 80 of file reverseTokenizer.cpp.

References Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offbeg, Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), and ContextTracker::update().

Here is the call graph for this function:

Here is the caller graph for this function:

double ReverseTokenizer::progress

(

)

const [virtual]

Returns progress percentage.

Implements Tokenizer.

Definition at line 174 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, Tokenizer::offend, and Tokenizer::offset.

---

The documentation for this class was generated from the following files:

• /build/buildd/presage-0.8.7/src/lib/core/tokenizer/reverseTokenizer.h
• /build/buildd/presage-0.8.7/src/lib/core/tokenizer/reverseTokenizer.cpp