test/doc/testing_tools/boost_test_collection_comparison.qbk

[/
 / Copyright (c) 2015 Boost.Test contributors
 /
 / Distributed under the Boost Software License, Version 1.0. (See accompanying
 / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 /]

[section:collections Collections comparison]

Instead of comparing a single value against another, there is often a need for comparing /collections/ of values.
A collection and indirectly the values it contains may be considered in several ways:

* collection as a /sequence of values/: this is the case for instance when `N` values are stored in a
  container. Containers in this case are used for storing several values, and iterating over the containers yields
  sequences that can be compared *element-wise*. The iteration should be in an order that is /a priori/ known
  [footnote this might not be the case
  for e.g. `std::unordered_map`, for which the buckets might be filled differently depending on the insertion order.],
  for being able to compare the sequences. The values in the collection are independent each other, and subsets can be compared as well.
* collection as an /ensemble/: this is the case where the elements of the collection define an /entity/,
  and no element can be dissociated from the others.
  An example would be a collection of letters for a specific word in the natural language; in this settings
  any of the character in the word/collection depends /semantically/ on the other and it is not possible to take
  a subset of it without breaking the meaning of the word. Another example would be a vector of size `N` representing a
  point in a `N` dimensional space, compared to another point with the relation "`<`": the comparison is application
  specific and a possible comparison would be the lexicographical ordering
  [footnote in this case `v_a < v_b` means that the point `v_a` is inside the rectangle (origin, `v_b`)].

The following observations can be done:

* the methods employed for comparing collections should be chosen adequately with the meaning of the collection,
* comparing sequences *element-wise* often involves writing loops in the test body, and if a dedicated tool is already in place
  the test body would gain in clarity and expressiveness (including the report in case of failure),
* some comparison methods such as the lexicographical one, have good general behavior (e.g. total ordering,
  defined for collections of different size), but are sometimes inappropriate.

__BOOST_TEST__ provides specific tools for comparing collections:

* using the /native/ [footnote either defined by the container or by the user] operator of the container of the collection,
  which is mentioned as the [link ref_boost_test_coll_default_comp /default behavior/].
* using [link boost_test_coll_perelement element-wise] comparison for which extended failure diagnostic is provided,
* and using [link boost_test_coll_default_lex lexicographical] comparison for which extended failure diagnostic is provided,

More details about the concept of /collection/ in the __UTF__ is given [link what_is_a_collection /here/].

[#ref_boost_test_coll_default_comp][h3 Default comparison]
The default comparison dispatches to the existing overloaded comparison operator. The __UTF__ distinguishes two use cases

# none of the comparison operand is a C-Array, in which case we use the [link ref_boost_test_coll_default_comp_container container default behavior]
# one of the comparison operand is a C-array, in which case we [link ref_boost_test_coll_c_arrays mimic `std::vector`] behavior

[#ref_boost_test_coll_default_comp_container][h4 Container default behavior]
Given two containers `c_a` and `c_b` that are not C-arrays,

``
BOOST_TEST(c_a op c_b)
``

is equivalent, in terms of test success, to

``
auto result = c_a op c_b;
BOOST_TEST(result);
``

In the example below, `operator==` is not defined for `std::vector` of different types, and the program would fail to
compile if the corresponding lines were uncommented (`std::vector` uses lexicographical comparison by default).

[note In the case of default comparison, there is no additional diagnostic provided by the __UTF__. See the section
[link ref_boost_test_coll_special_macro `BOOST_TEST_SPECIALIZED_COLLECTION_COMPARE`] below.]

[bt_example boost_test_container_default..BOOST_TEST containers comparison default..run-fail]

[#ref_boost_test_coll_c_arrays][h4 C-arrays default behavior]
As soon as one of the operands is a C-array, there is no /default behavior/ the __UTF__ can dispatch to.
This is why in that case, the comparison mimics the `std::vector` behavior.

[bt_example boost_test_macro_container_c_array..BOOST_TEST C-arrays..run-fail]

[#boost_test_coll_perelement][h3 Element-wise comparison]
By specifying the manipulator [classref boost::test_tools::per_element], the comparison of the elements of the containers
are performed /element-wise/, in the order given by the forward iterators of the containers. This is a comparison on
the /sequences/ of elements generated by the containers, for which the __UTF__ provides advanced diagnostic.

In more details, let `c_a = (a_1,... a_n)` and `c_b = (b_1,... b_n)` be two sequences of same length, but not necessarily of same type.
Those sequences correspond to the content of the respective containers, in the order given by their iterator. Let
`op` be one of the [link boost_test_statement_overloads binary comparison operators].

``
BOOST_TEST(c_a op c_b, boost::test_tools::per_element() );
``

is equivalent to

``
if(c_a.size() == c_b.size())
{
  for(int i=0; i < c_a.size(); i++)
  {
    __BOOST_TEST_CONTEXT__("index " << i)
    {
      BOOST_TEST(a_i op b_i);
    }
  }
}
else
{
  BOOST_TEST(c_a.size() == c_b.size());
}
``

[warning this is fundamentally different from using the containers' default comparison operators (default behavior).]
[warning this is not an order relationship on containers. As a side effect, it is possible to have
 ``BOOST_TEST(c_a == c_b)`` and ``BOOST_TEST(c_a != c_b)`` failing at the same time]

Sequences are compared using the specified operator `op`, evaluated on the left and right elements of the respective sequences.
The order of the compared elements is given by the iterators of the respective containers [footnote the containers should yield the same
sequences for a fixed set of elements they contain].
In case of failure, the indices of the elements failing `op` are returned.

[bt_example boost_test_sequence_per_element..BOOST_TEST sequence comparison..run-fail]

[h4 Requirements]
For the sequences to be comparable element-wise, the following conditions should be met:

* the containers should meet the [link what_is_a_collection sequence] definition,
* the containers should yield the same number of elements,
* `op` should be one of the comparison operator `==`, `!=`, `<`, `<=`, `>`, `>=`
* the `a_i op b_i` should be defined, where the type of `a_i` and `b_i` are the type returned by the dereference operator
  of the respective collections.

[caution the resulting type of "`c_a == c_b`" is an [classref boost::test_tools::assertion_result assertion_result]: it is not
 possible to compose more that one comparison on the `BOOST_TEST` statement:

 ``
   BOOST_TEST(c_a == c_b == 42, boost::test_tools::per_element() ); // does not compile
 ``]

[#boost_test_coll_default_lex][h3 Lexicographic comparison]
By specifying the manipulator [classref boost::test_tools::lexicographic], the containers are compared using the /lexicographical/
order and for which the __UTF__ provides additional diagnostic in case of failure.

``
BOOST_TEST(c_a op c_b, boost::test_tools::lexicographic() );
``

The comparison is performed in the order given by forward iterators of the containers.

[tip lexicographic comparison yields a total order on the containers: the statements `c_a < c_b` and
 `c_b <= c_a` are mutually exclusive.]

[note The equality `==` and inequality `!=` are not available for this type of comparison.]

[bt_example boost_test_container_lex..BOOST_TEST container comparison using lexicographical order..run-fail]


[#ref_boost_test_coll_special_macro][h3 Extended diagnostic by default for specific containers]

As seen above,

* for testing equality, the `==` relation is either explicit (using `boost::test_tools::per_element()`) or
  implicit when the container overloads/implements this type of comparison,
* for testing inequality, lexicographical comparison for `<` (and derived operations) is either explicit
  (using `boost::test_tools::lexicographic()`) or implicit when
  the container overloads/implements uses this type of comparison.

When the default is to using the container implementation, it is not possible to benefit from an extended failure diagnostic.
The __UTF__ provides a mechanism for performing the same comparisons through the __UTF__ instead of the container operator,
through the macro `BOOST_TEST_SPECIALIZED_COLLECTION_COMPARE` that might be used as follow:

[bt_example boost_test_container_lex_default..Default `std::vector<int>` to lexicographic with extended diagnostic..run-fail]

[h4 Requirements]

* the containers should meet the [link what_is_a_collection sequence] definition,
* the containers should be of the exact same type
* `op` should be one of the comparison operator `==`, `!=`, `<`, `<=`, `>`, `>=`

[note Note that the operation `!=` is in this case not an element-wise comparison, ]


[#what_is_a_collection][h3 What is a sequence?]
A /sequence/ is given by the iteration over a /forward iterable/ container. A forward iterable container is:

* either a C-array,
* or a `class`/`struct` that implements the member functions `begin` and `end`.

For collection comparisons, both sequences are also required to be different than `string` sequences. In that case, the sequences are
dispatched to string [link boost_test.testing_tools.extended_comparison.strings comparison instead].

[warning `string` (or `wstring`) meets the sequence concept by definition, but their handling with __BOOST_TEST__ is done differently.
 See [link boost_test.testing_tools.extended_comparison.strings Strings and C-strings comparison] for more details.]

[tip If the behavior of __BOOST_TEST__ is not the one you expect, you can always use raw comparison. See [link boost_test_statement_limitations this section]
 for details.]

[note Since [link ref_CHANGE_LOG_3_6 Boost.Test 3.6] (Boost 1.65) the requirements for the collection concepts have been relaxed to
 include C-arrays as well]
[note Since [link ref_CHANGE_LOG_3_7 Boost.Test 3.7] (Boost 1.67) the definition of `const_iterator` and `value_type` in the collection
 type is not required anymore (for the compilers properly supporting `decltype`). ]

The detection of the types that meet these requirements containers is delegated to the class [classref boost::unit_test::is_forward_iterable],
which for C++11 detects the required member functions and fields. However for C++03, the types providing the sequences should be explicitly
indicated to the __UTF__ by a specialization of [classref boost::unit_test::is_forward_iterable]
[footnote Standard containers of the `STL` are recognized as /forward iterable/ container.].

[endsect] [/ sequences]