183 lines
4.5 KiB
HTML
183 lines
4.5 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Language" content="en-us">
|
|
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
|
|
<meta name="GENERATOR" content="Microsoft FrontPage 6.0">
|
|
<meta name="ProgId" content="FrontPage.Editor.Document">
|
|
|
|
<title>TokenizerFunction Concept</title>
|
|
</head>
|
|
|
|
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE" vlink="#551A8B" alink=
|
|
"#FF0000">
|
|
<p><img src="../../../boost.png" alt="C++ Boost" width="277" height="86"></p>
|
|
|
|
<h1 align="center">TokenizerFunction Concept</h1>
|
|
|
|
<p>A TokenizerFunction is a functor whose purpose is to parse a given
|
|
sequence until exactly 1 token has been found or the end is reached. It
|
|
then updates the token, and informs the caller of the location in the
|
|
sequence of the next element immediately after the last element of the
|
|
sequence that was parsed for the current token.</p>
|
|
|
|
<h2>Refinement of</h2>
|
|
|
|
<p>Assignable, CopyConstructable</p>
|
|
|
|
<h2>Notation</h2>
|
|
|
|
<table border="1" summary="">
|
|
<tr>
|
|
<td valign="top"><tt>X</tt></td>
|
|
|
|
<td valign="top">A type that is a model of TokenizerFunction</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td valign="top"><tt>func</tt></td>
|
|
|
|
<td valign="top">Object of type <tt>X</tt></td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td valign="top"><tt>tok</tt></td>
|
|
|
|
<td valign="top">Object of Token</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>next</td>
|
|
|
|
<td>iterator that points to the first unparsed element of the sequence
|
|
being parsed</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>end</td>
|
|
|
|
<td>iterator that points to the past the end of the sequence being
|
|
parsed</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Definitions</h2>
|
|
|
|
<p>A token is the result of parsing a sequence.</p>
|
|
|
|
<h2>Valid expressions</h2>
|
|
|
|
<p>In addition to the expression in Assignable and CopyConstructable the
|
|
following expressions are valid</p>
|
|
|
|
<table border="1" summary="">
|
|
<tr>
|
|
<th>Name</th>
|
|
|
|
<th>Expression</th>
|
|
|
|
<th>Return type</th>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td valign="top">Functor</td>
|
|
|
|
<td valign="top"><tt>func(next, end, tok)</tt></td>
|
|
|
|
<td valign="top"><tt>bool</tt></td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td valign="top">reset</td>
|
|
|
|
<td valign="top"><tt>reset()</tt></td>
|
|
|
|
<td valign="top"><tt>void</tt></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Expression semantics</h2>
|
|
|
|
<p>In addition to the expression semantics in Assignable and
|
|
CopyConstructable, TokenizerFunction has the following expression
|
|
semantcs</p>
|
|
|
|
<table border="1" summary="">
|
|
<tr>
|
|
<th>Name</th>
|
|
|
|
<th>Expression</th>
|
|
|
|
<th>Precondition</th>
|
|
|
|
<th>Semantics</th>
|
|
|
|
<th>Postcondition</th>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>operator()</td>
|
|
|
|
<td><tt>func(next, end, tok)</tt></td>
|
|
|
|
<td><tt>next</tt> and <tt>end</tt> are valid iterators to the same
|
|
sequence. next is a reference the function is free to modify. tok is
|
|
constructed.</td>
|
|
|
|
<td>The return value indicates whether a new token was found in the
|
|
sequence [next,end)</td>
|
|
|
|
<td>If the return value is true, the new token is assigned to tok. next
|
|
is always updated to the position where parsing should start on the
|
|
subsequent call.</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>reset</td>
|
|
|
|
<td><tt>reset()</tt></td>
|
|
|
|
<td><tt>None</tt></td>
|
|
|
|
<td>Clears out all state variables that are used by the object in
|
|
parsing the current sequence.</td>
|
|
|
|
<td>A new sequence to parse can be given.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Complexity guarantees</h2>
|
|
|
|
<p>No guarantees. Models of TokenizerFunction are free to define their own
|
|
complexity</p>
|
|
|
|
<h2>Models</h2>
|
|
|
|
<p><a href="escaped_list_separator.htm">escaped_list_separator</a></p>
|
|
|
|
<p><a href="offset_separator.htm">offset_separator</a></p>
|
|
|
|
<p><a href=
|
|
"char_delimiters_separator.htm">char_delimiters_separator</a></p>
|
|
|
|
<p> </p>
|
|
<hr>
|
|
|
|
<p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
|
|
"../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional"
|
|
height="31" width="88"></a></p>
|
|
|
|
<p>Revised
|
|
<!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25
|
|
December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>
|
|
|
|
<p><i>Copyright © 2001 John R. Bandela</i></p>
|
|
|
|
<p><i>Distributed under the Boost Software License, Version 1.0. (See
|
|
accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
|
|
copy at <a href=
|
|
"http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
|
|
</body>
|
|
</html>
|