ae01c8387c
- section about command line argument filtering for template test cases - many typos fixes - remove reference to bjam
235 lines
9.9 KiB
Plaintext
235 lines
9.9 KiB
Plaintext
[/
|
|
/ Copyright (c) 2003 Boost.Test contributors
|
|
/
|
|
/ Distributed under the Boost Software License, Version 1.0. (See accompanying
|
|
/ file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
|
|
/]
|
|
|
|
[section:log_formats Log formats]
|
|
|
|
The __UTF__ supports several log formats:
|
|
|
|
* [link boost_test.test_output.log_formats.log_human_readable_format HRF]: human readable format
|
|
* [link boost_test.test_output.log_formats.log_xml_format XML]: an machine interpretable log format
|
|
* [link boost_test.test_output.log_formats.log_junit_format JUNIT]: a standardized log format
|
|
understandable by automated tools such as Continuous Builds
|
|
|
|
[h4 Design]
|
|
|
|
The following functionalities are supported by the logging framework:
|
|
|
|
* each logger manages its own log level. The rationale is that some log format are meant for
|
|
automated processing, and by design need to carry all the information that will later be digested
|
|
by a visualization tool.
|
|
* several log format may be active at the same time. The rationale is that the user might want to
|
|
see a non-exhaustive log in his terminal using a human friendly format, while having a detailed
|
|
full log in a file with a format dedicated to automated processing.
|
|
* each logger /indicates/ its default output stream. some logger may prefer to output to one of the standard
|
|
stream while other may prefer output to a file.
|
|
|
|
[note The logger indicates the default output stream in case the default should be used. ]
|
|
|
|
[h4 Defaults]
|
|
By default the active log level threshold is set to
|
|
|
|
* [link boost_test.test_output.log_formats.test_log_output "non fatal error messages"] and the test log output
|
|
is generated in [link boost_test.test_output.log_formats.log_human_readable_format human readable format].
|
|
* [link boost_test.test_output.log_formats.test_log_output "general information"] for
|
|
[link boost_test.test_output.log_formats.log_junit_format JUNIT] log format
|
|
|
|
The active log level threshold and the output format can be configured
|
|
at runtime during a test module invocation and at compile time from within a test module using the
|
|
[link boost_test.test_output.logging_api test log public interfaces]. The behavior is logger specific though.
|
|
|
|
|
|
[/ -------------------------------------------------------------------------------------------------- ]
|
|
[section Test log output]
|
|
|
|
The test log is produced during the test execution. All entries in the test log are assigned a particular log
|
|
level. Only the entries with level that exceeds the ['active log level threshold] actually
|
|
appear in the test log output. Log levels are arranged by the 'importance' of the log entries. Here is
|
|
the list of all levels in order of increasing 'importance':
|
|
|
|
[#test_log_output_table]
|
|
[table:id_messages Messages
|
|
[
|
|
[Notifications]
|
|
[Meaning]
|
|
]
|
|
[
|
|
[Success]
|
|
[This category includes messages that provide information on successfully passed assertions]
|
|
]
|
|
[
|
|
[Test tree traversal]
|
|
[This category includes messages that are produced by the __UTF__ core and indicate which test suites/cases are currently being executed or skipped]
|
|
]
|
|
[
|
|
[General information]
|
|
[This category includes general information messages produced in most cases by a test module author using the
|
|
macro __BOOST_TEST_MESSAGE__]
|
|
]
|
|
[
|
|
[Warning]
|
|
[This category includes messages produced by failed `WARNING` level assertions]
|
|
]
|
|
[
|
|
[Non fatal error]
|
|
[This category includes messages produced by failed `CHECK` level assertions]
|
|
]
|
|
|
|
[
|
|
[Uncaught C++ exceptions]
|
|
[This category includes messages that are produced by the __UTF__ and provide detailed information on the C++
|
|
exceptions uncaught by the test case body.
|
|
]
|
|
]
|
|
[
|
|
[Non-fatal system error]
|
|
[This category includes messages that are produced by the __UTF__ itself and provides information about caught
|
|
non-fatal system error. For example it includes messages produced in the case of test case timeout or if
|
|
floating point values calculation errors are caught.
|
|
]
|
|
]
|
|
[
|
|
[Fatal system error]
|
|
[This category includes messages produced by failed require level assertions and by the __UTF__ itself in case of
|
|
abnormal test case termination.]
|
|
]
|
|
]
|
|
|
|
[note
|
|
The active log level works namely as threshold, not as selector. For the given active log level threshold, all
|
|
test log entries with ['importance] higher than threshold are enabled and all test log entries with
|
|
['importance] below threshold are disabled.
|
|
]
|
|
|
|
In addition to the levels described above the test log defines two special log levels. The current log level can
|
|
be set to:
|
|
|
|
* All messages[br]
|
|
If active log level threshold is set to this value, all test log entries appear in the output. In practice
|
|
this is equivalent to setting the active log level threshold to ['success information messages]
|
|
* Nothing[br]
|
|
If the active log level threshold is set to this value, none of test log entries appear in the output. This log level
|
|
is used to execute a ['silent] test that doesn't produce any test log and only generates a result code indicating
|
|
whether test failed or passed.
|
|
|
|
[endsect] [/ test log output]
|
|
|
|
|
|
|
|
[section:log_human_readable_format HRF: Human readable log format]
|
|
|
|
The human readable log format is designed to closely match an errors description produced by the Microsoft family
|
|
of C++ compilers. This format allows jumping to the error location, if test module output is redirected into IDE
|
|
output window. The rest of the log messages are designed to produce the most human friendly description of the
|
|
events occurring in test module. This is a default format generated by test modules.
|
|
|
|
Here the list of events along with corresponding message and the condition that has to be satisfied for it to appear
|
|
in the output.
|
|
|
|
[table
|
|
[
|
|
[Event]
|
|
[Condition]
|
|
[Output]
|
|
]
|
|
|
|
[ [On testing start]
|
|
[threshold != log_nothing]
|
|
[`Running <total number of test cases> test case(s) ...`] ]
|
|
|
|
[ [On testing start]
|
|
[threshold != log_nothing, show_build_info is set]
|
|
[
|
|
[pre Platform: $BOOST_PLATFORM
|
|
Compiler: $BOOST_COMPILER
|
|
STL : $BOOST_STDLIB
|
|
Boost : $BOOST_VERSION]] ]
|
|
|
|
[ [On abnormal testing termination]
|
|
[threshold <= log_messages]
|
|
[`Test is aborted`] ]
|
|
|
|
[ [On test unit start]
|
|
[threshold <= log_test_units]
|
|
[`Entering test <test unit type> <test unit name>`] ]
|
|
|
|
[ [On test unit end]
|
|
[threshold <= log_test_units; testing time is reported only if elapsed time is more than 1 us.]
|
|
[`Leaving test <test unit type> <test unit name>; testing time <value>`] ]
|
|
|
|
[ [On skipped test unit]
|
|
[threshold <= log_test_units]
|
|
[`Test <test unit type> <test unit name> is skipped`] ]
|
|
|
|
[ [On uncaught C++ exception]
|
|
[threshold <= log_cpp_exception_errors. Checkpoint message is reported only if provided]
|
|
[`unknown location(0): fatal error in <test case name>: <explanation> <last checkpoint location>: last checkpoint: <checkpoint message>`] ]
|
|
|
|
[ [On resumable system error]
|
|
[threshold <= log_system_errors. Checkpoint message is reported only if provided]
|
|
[`unknown location(0): fatal error in <test case name>: <explanation> <last checkpoint location>: last checkpoint: <checkpoint message>`] ]
|
|
|
|
[ [On fatal system error]
|
|
[threshold <= log_fatal_errors. Checkpoint message is reported only if provided]
|
|
[`unknown location(0): fatal error in <test case name>: <explanation> <last checkpoint location>: last checkpoint: <checkpoint message>`] ]
|
|
|
|
[ [On passed test assertion]
|
|
[threshold <= log_successful_tests]
|
|
[`<assertion location>: info: check<assertion expression> passed`] ]
|
|
|
|
[ [On failed WARNING level test assertion]
|
|
[threshold <= log_warnings]
|
|
[`<assertion location>: warning in <test case name>: condition <assertion description> is not satisfied`]]
|
|
|
|
[ [On failed CHECK level test assertion]
|
|
[threshold <= log_all_errors]
|
|
[`<assertion location>: error in <test case name>: check <assertion description> failed`] ]
|
|
|
|
[ [On failed REQUIRE level test assertion]
|
|
[threshold <= log_fatal_errors]
|
|
[`<assertion location>: fatal error in <test case name>: critical check <assertion description> failed`]]
|
|
|
|
[ [On test log message]
|
|
[threshold <= log_messages]
|
|
[`<Message content>`]]
|
|
]
|
|
|
|
The level of details concerning the error message depends on the [link boost_test.testing_tools testing tool] producing the log entry.
|
|
|
|
[endsect] [/ human readable report]
|
|
|
|
[/ -------------------------------------------------------------------------------------------------- ]
|
|
[section:log_xml_format XML log format]
|
|
|
|
This log format is designed for automated test results processing. The test log output XML schema depends on the
|
|
active log level threshold.
|
|
|
|
[endsect] [/section:log_xml_format ]
|
|
|
|
[/ -------------------------------------------------------------------------------------------------- ]
|
|
[section:log_junit_format JUNIT log format]
|
|
|
|
The [@http://junit.org/ JUNIT format] is log format supported by a wide range of Continuous Build/Integration tools.
|
|
|
|
This format defaults its log level to [link test_log_output_table `General information`] and its default stream to a file named after
|
|
[link boost_test.tests_organization.test_tree.master_test_suite master test suite].
|
|
The logger will attempt to not overwrite any existing output file, which is also usually understood by Continuous Build tools.
|
|
|
|
This format is in fact both a log and a report format: most of the Continuous Build tools will summarize
|
|
the content of a JUNIT file and show an overview of the failing/succeeding tests of a module (report format)
|
|
while letting the user inspect the detailed logs (log format).
|
|
|
|
[caution The minimal log-level for the JUnit logger is [link test_log_output_table `non fatal error`].
|
|
Any set-up to higher log level will default to that minimal log-level.]
|
|
|
|
[note Until Boost 1.64, the log level was previously defaulting to `success` and was causing a heavy load
|
|
on the logging part in some circumstances.]
|
|
|
|
[endsect] [/section:log_junit_format ]
|
|
|
|
|
|
[endsect]
|