Tokenizer

Essential Tools Module User's Guide
Rogue Wave web site: Home Page | Main Documentation Page

4.8 Tokenizer

You can use the class RWCTokenizer to break up a string into tokens separated by arbitrary white spaces. Here's an example:

#include <rw/ctoken.h>
#include <rw/cstring.h>
#include <rw/rstream.h>
int main(){
  RWCString a("a string with five tokens");
  RWCTokenizer next(a);
  int i = 0;
  // Advance until the null string is returned:
  while( !next().isNull() ) i++;
     std::cout << i << std::endl;
  return 0;
}

Program Output:

This program counts the number of tokens in the string. The function call operator for class RWCTokenizer has been overloaded to mean "advance to the next token and return it as an RWCSubString," much like other Essential Tools Module iterators. When there are no more tokens, it returns the null substring. Class RWCSubString has a member function isNull() which returns true if the substring is the null substring. Hence, the loop is broken. See the Reference Guide entry for RWCTokenizer for details.

The Rogue Wave name and logo, and SourcePro, are registered trademarks of Rogue Wave Software. All other trademarks are the property of their respective owners.
Provide feedback to Rogue Wave about its documentation.