138 lines
5.2 KiB
Plaintext
138 lines
5.2 KiB
Plaintext
[/
|
|
/ Copyright (c) 2008 Eric Niebler
|
|
/
|
|
/ Distributed under the Boost Software License, Version 1.0. (See accompanying
|
|
/ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
|
|
/]
|
|
|
|
[section String Splitting and Tokenization]
|
|
|
|
_regex_token_iterator_ is the Ginsu knife of the text manipulation world. It slices! It dices! This section describes
|
|
how to use the highly-configurable _regex_token_iterator_ to chop up input sequences.
|
|
|
|
[h2 Overview]
|
|
|
|
You initialize a _regex_token_iterator_ with an input sequence, a regex, and some optional configuration parameters.
|
|
The _regex_token_iterator_ will use _regex_search_ to find the first place in the sequence that the regex matches. When
|
|
dereferenced, the _regex_token_iterator_ returns a ['token] in the form of a `std::basic_string<>`. Which string it returns
|
|
depends on the configuration parameters. By default it returns a string corresponding to the full match, but it could also
|
|
return a string corresponding to a particular marked sub-expression, or even the part of the sequence that ['didn't] match.
|
|
When you increment the _regex_token_iterator_, it will move to the next token. Which token is next depends on the configuration
|
|
parameters. It could simply be a different marked sub-expression in the current match, or it could be part or all of the
|
|
next match. Or it could be the part that ['didn't] match.
|
|
|
|
As you can see, _regex_token_iterator_ can do a lot. That makes it hard to describe, but some examples should make it clear.
|
|
|
|
[h2 Example 1: Simple Tokenization]
|
|
|
|
This example uses _regex_token_iterator_ to chop a sequence into a series of tokens consisting of words.
|
|
|
|
std::string input("This is his face");
|
|
sregex re = +_w; // find a word
|
|
|
|
// iterate over all the words in the input
|
|
sregex_token_iterator begin( input.begin(), input.end(), re ), end;
|
|
|
|
// write all the words to std::cout
|
|
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
|
|
std::copy( begin, end, out_iter );
|
|
|
|
This program displays the following:
|
|
|
|
[pre
|
|
This
|
|
is
|
|
his
|
|
face
|
|
]
|
|
|
|
[h2 Example 2: Simple Tokenization, Reloaded]
|
|
|
|
This example also uses _regex_token_iterator_ to chop a sequence into a series of tokens consisting of words,
|
|
but it uses the regex as a delimiter. When we pass a `-1` as the last parameter to the _regex_token_iterator_
|
|
constructor, it instructs the token iterator to consider as tokens those parts of the input that ['didn't]
|
|
match the regex.
|
|
|
|
std::string input("This is his face");
|
|
sregex re = +_s; // find white space
|
|
|
|
// iterate over all non-white space in the input. Note the -1 below:
|
|
sregex_token_iterator begin( input.begin(), input.end(), re, -1 ), end;
|
|
|
|
// write all the words to std::cout
|
|
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
|
|
std::copy( begin, end, out_iter );
|
|
|
|
This program displays the following:
|
|
|
|
[pre
|
|
This
|
|
is
|
|
his
|
|
face
|
|
]
|
|
|
|
[h2 Example 3: Simple Tokenization, Revolutions]
|
|
|
|
This example also uses _regex_token_iterator_ to chop a sequence containing a bunch of dates into a series of
|
|
tokens consisting of just the years. When we pass a positive integer [^['N]] as the last parameter to the
|
|
_regex_token_iterator_ constructor, it instructs the token iterator to consider as tokens only the [^['N]]-th
|
|
marked sub-expression of each match.
|
|
|
|
std::string input("01/02/2003 blahblah 04/23/1999 blahblah 11/13/1981");
|
|
sregex re = sregex::compile("(\\d{2})/(\\d{2})/(\\d{4})"); // find a date
|
|
|
|
// iterate over all the years in the input. Note the 3 below, corresponding to the 3rd sub-expression:
|
|
sregex_token_iterator begin( input.begin(), input.end(), re, 3 ), end;
|
|
|
|
// write all the words to std::cout
|
|
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
|
|
std::copy( begin, end, out_iter );
|
|
|
|
This program displays the following:
|
|
|
|
[pre
|
|
2003
|
|
1999
|
|
1981
|
|
]
|
|
|
|
[h2 Example 4: Not-So-Simple Tokenization]
|
|
|
|
This example is like the previous one, except that instead of tokenizing just the years, this program
|
|
turns the days, months and years into tokens. When we pass an array of integers [^['{I,J,...}]] as the last
|
|
parameter to the _regex_token_iterator_ constructor, it instructs the token iterator to consider as tokens the
|
|
[^['I]]-th, [^['J]]-th, etc. marked sub-expression of each match.
|
|
|
|
std::string input("01/02/2003 blahblah 04/23/1999 blahblah 11/13/1981");
|
|
sregex re = sregex::compile("(\\d{2})/(\\d{2})/(\\d{4})"); // find a date
|
|
|
|
// iterate over the days, months and years in the input
|
|
int const sub_matches[] = { 2, 1, 3 }; // day, month, year
|
|
sregex_token_iterator begin( input.begin(), input.end(), re, sub_matches ), end;
|
|
|
|
// write all the words to std::cout
|
|
std::ostream_iterator< std::string > out_iter( std::cout, "\n" );
|
|
std::copy( begin, end, out_iter );
|
|
|
|
This program displays the following:
|
|
|
|
[pre
|
|
02
|
|
01
|
|
2003
|
|
23
|
|
04
|
|
1999
|
|
13
|
|
11
|
|
1981
|
|
]
|
|
|
|
The `sub_matches` array instructs the _regex_token_iterator_ to first take the value of the 2nd sub-match, then
|
|
the 1st sub-match, and finally the 3rd. Incrementing the iterator again instructs it to use _regex_search_ again
|
|
to find the next match. At that point, the process repeats -- the token iterator takes the value of the 2nd
|
|
sub-match, then the 1st, et cetera.
|
|
|
|
[endsect]
|