4.8 Tokenizer

You can use the class RWCTokenizer to break up a string into tokens separated by arbitrary white spaces. Here's an example:

#include <rw/ctoken.h>
#include <rw/cstring.h>
#include <rw/rstream.h>
int main(){
  RWCString a("a string with five tokens");
  RWCTokenizer next(a);
  int i = 0;
  // Advance until the null string is returned:
  while( !next().isNull() ) i++;
     std::cout << i << std::endl;
  return 0;
}

Program Output:

5

This program counts the number of tokens in the string. The function call operator for class RWCTokenizer has been overloaded to mean "advance to the next token and return it as an RWCSubString," much like other Essential Tools Module iterators. When there are no more tokens, it returns the null substring. Class RWCSubString has a member function isNull() which returns true if the substring is the null substring. Hence, the loop is broken. See the Essential Tools Module Reference Guide under RWCTokenizer for details.

© Copyright Rogue Wave Software, Inc. All Rights Reserved.
Rogue Wave and SourcePro are registered trademarks of Rogue Wave Software, Inc. in the United States and other countries. All other trademarks are the property of their respective owners.
Contact Rogue Wave about documentation or support issues.