1 <sect1 id="manual.intro.setup.test" xreflabel="Testing">
2 <?dbhtml filename="test.html"?>
20 <sect2 id="test.organization" xreflabel="test.organization">
21 <title>Organization</title>
23 The directory <emphasis>libsrcdir/testsuite</emphasis> contains the
24 individual test cases organized in sub-directories corresponding to
25 chapters of the C++ standard (detailed below), the dejagnu test
26 harness support files, and sources to various testsuite utilities
27 that are packaged in a separate testing library.
31 All test cases for functionality required by the runtime components
32 of the C++ standard (ISO 14882) are files within the following
50 In addition, the following directories include test files:
54 tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
55 backward Tests for backwards compatibility and deprecated features.
56 demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler
57 ext Tests for extensions.
58 performance Tests for performance analysis, and performance regressions.
59 thread Tests for threads.
63 Some directories don't have test files, but instead contain
64 auxiliary information (<ulink url="#internals">more information</ulink>):
68 config Files for the dejagnu test harness.
69 lib Files for the dejagnu test harness.
70 libstdc++* Files for the dejagnu test harness.
71 data Sample text files for testing input and output.
72 util Files for libtestc++, utilities and testing routines.
76 Within a directory that includes test files, there may be
77 additional subdirectories, or files. Originally, test cases
78 were appended to one file that represented a particular section
79 of the chapter under test, and was named accordingly. For
80 instance, to test items related to <code> 21.3.6.1 -
81 basic_string::find [lib.string::find]</code> in the standard,
82 the following was used:
88 However, that practice soon became a liability as the test cases
89 became huge and unwieldy, and testing new or extended
90 functionality (like wide characters or named locales) became
91 frustrating, leading to aggressive pruning of test cases on some
92 platforms that covered up implementation errors. Now, the test
93 suite has a policy of one file, one test case, which solves the
94 above issues and gives finer grained results and more manageable
95 error debugging. As an example, the test case quoted above
99 21_strings/basic_string/find/char/1.cc
100 21_strings/basic_string/find/char/2.cc
101 21_strings/basic_string/find/char/3.cc
102 21_strings/basic_string/find/wchar_t/1.cc
103 21_strings/basic_string/find/wchar_t/2.cc
104 21_strings/basic_string/find/wchar_t/3.cc
108 All new tests should be written with the policy of one test
109 case, one file in mind.
114 <sect2 id="test.naming" xreflabel="test.naming">
115 <title>Naming Conventions</title>
120 In addition, there are some special names and suffixes that are
121 used within the testsuite to designate particular kinds of
128 <emphasis>_xin.cc</emphasis>
131 This test case expects some kind of interactive input in order
132 to finish or pass. At the moment, the interactive tests are not
133 run by default. Instead, they are run by hand, like:
136 g++ 27_io/objects/char/3_xin.cc
137 cat 27_io/objects/char/3_xin.in | a.out
142 <emphasis>.in</emphasis>
145 This file contains the expected input for the corresponding <emphasis>
146 _xin.cc</emphasis> test case.
151 <emphasis>_neg.cc</emphasis>
154 This test case is expected to fail: it's a negative test. At the
155 moment, these are almost always compile time errors.
160 <emphasis>char</emphasis>
163 This can either be a directory name or part of a longer file
164 name, and indicates that this file, or the files within this
165 directory are testing the <code>char</code> instantiation of a
171 <emphasis>wchar_t</emphasis>
174 This can either be a directory name or part of a longer file
175 name, and indicates that this file, or the files within this
176 directory are testing the <code>wchar_t</code> instantiation of
177 a template. Some hosts do not support <code>wchar_t</code>
178 functionality, so for these targets, all of these tests will not
184 <emphasis>thread</emphasis>
187 This can either be a directory name or part of a longer file
188 name, and indicates that this file, or the files within this
189 directory are testing situations where multiple threads are
195 <emphasis>performance</emphasis>
198 This can either be an enclosing directory name or part of a
199 specific file name. This indicates a test that is used to
200 analyze runtime performance, for performance regression testing,
201 or for other optimization related analysis. At the moment, these
202 test cases are not run by default.
209 <sect2 id="test.utils" xreflabel="test.utils">
210 <title>Utilities</title>
214 The testsuite directory also contains some files that implement
215 functionality that is intended to make writing test cases easier,
216 or to avoid duplication, or to provide error checking in a way that
217 is consistent across platforms and test harnesses. A stand-alone
218 executable, called <emphasis>abi_check</emphasis>, and a static
219 library called <emphasis>libtestc++</emphasis> are
220 constructed. Both of these items are not installed, and only used
225 These files include the following functionality:
231 <emphasis>testsuite_abi.h</emphasis>,
232 <emphasis>testsuite_abi.cc</emphasis>,
233 <emphasis>testsuite_abi_check.cc</emphasis>
236 Creates the executable <emphasis>abi_check</emphasis>.
237 Used to check correctness of symbol versioning, visibility of
238 exported symbols, and compatibility on symbols in the shared
239 library, for hosts that support this feature. More information
240 can be found in the ABI documentation <ulink url="abi.html">here</ulink>
245 <emphasis>testsuite_allocator.h</emphasis>,
246 <emphasis>testsuite_allocator.cc</emphasis>
249 Contains specialized allocators that keep track of construction
250 and destruction. Also, support for overriding global new and
251 delete operators, including verification that new and delete
252 are called during execution, and that allocation over max_size
258 <emphasis>testsuite_character.h</emphasis>
261 Contains <code>std::char_traits</code> and
262 <code>std::codecvt</code> specializations for a user-defined
268 <emphasis>testsuite_hooks.h</emphasis>,
269 <emphasis>testsuite_hooks.cc</emphasis>
272 A large number of utilities, including:
275 <listitem><para>VERIFY</para></listitem>
276 <listitem><para>set_memory_limits</para></listitem>
277 <listitem><para>verify_demangle</para></listitem>
278 <listitem><para>run_tests_wrapped_locale</para></listitem>
279 <listitem><para>run_tests_wrapped_env</para></listitem>
280 <listitem><para>try_named_locale</para></listitem>
281 <listitem><para>try_mkfifo</para></listitem>
282 <listitem><para>func_callback</para></listitem>
283 <listitem><para>counter</para></listitem>
284 <listitem><para>copy_tracker</para></listitem>
285 <listitem><para>copy_constructor</para></listitem>
286 <listitem><para>assignment_operator</para></listitem>
287 <listitem><para>destructor</para></listitem>
289 <para>pod_char, pod_int and associated char_traits specializations</para>
295 <emphasis>testsuite_io.h</emphasis>
298 Error, exception, and constraint checking for
299 <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
304 <emphasis>testsuite_iterators.h</emphasis>
307 Wrappers for various iterators.
312 <emphasis>testsuite_performance.h</emphasis>
315 A number of class abstractions for performance counters, and
316 reporting functions including:
319 <listitem><para>time_counter</para></listitem>
320 <listitem><para>resource_counter</para></listitem>
321 <listitem><para>report_performance</para></listitem>
328 <sect2 id="test.run" xreflabel="test.run">
329 <title>Running the Testsuite</title>
331 <sect3 id="test.run.basic" xreflabel="test.run.basic">
332 <title>Basic Results</title>
334 There are several options for running tests, including testing
335 the regression tests, testing a subset of the regression tests,
336 testing the performance tests, testing just compilation, testing
337 installed tools, etc. In addition, there is a special rule for
338 checking the exported symbols of the shared library.
342 You can check the status of the build without installing it
343 using the dejagnu harness, much like the rest of the gcc
345 <programlisting> make check</programlisting>
346 <para>in the <emphasis>libbuilddir</emphasis> directory.</para>
348 <programlisting> make check-target-libstdc++-v3</programlisting>
349 <para>in the <emphasis>gccbuilddir</emphasis> directory.
353 These commands are functionally equivalent and will create a
354 'testsuite' directory underneath
355 <emphasis>libbuilddir</emphasis> containing the results of the
356 tests. Two results files will be generated: <emphasis>
357 libstdc++.sum</emphasis>, which is a PASS/FAIL summary for each
358 test, and <emphasis>libstdc++.log</emphasis> which is a log of
359 the exact command line passed to the compiler, the compiler
360 output, and the executable output (if any).
364 Archives of test results for various versions and platforms are
365 available on the GCC website in the <ulink
366 url="http://gcc.gnu.org/gcc-4.1/buildstat.html">build
367 status</ulink> section of each individual release, and are also
368 archived on a daily basis on the <ulink
369 url="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</ulink>
370 mailing list. Please check either of these places for a similar
371 combination of source version, operating system, and host CPU.
375 <sect3 id="test.run.options" xreflabel="test.run.options">
376 <title>Options</title>
378 To debug the dejagnu test harness during runs, try invoking with a
379 specific argument to the variable RUNTESTFLAGS, as below.
383 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
391 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
395 To run a subset of the library tests, you will need to generate
396 the <emphasis>testsuite_files</emphasis> file by running
397 <command>make testsuite_files</command> in the
398 <emphasis>libbuilddir/testsuite</emphasis> directory, described
399 below. Edit the file to remove the tests you don't want and
400 then run the testsuite as normal.
404 There are two ways to run on a simulator: set up DEJAGNU to point to a
405 specially crafted site.exp, or pass down --target_board flags.
409 Example flags to pass down for various embedded builds are as follows:
413 --target=powerpc-eabism (libgloss/sim)
414 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
416 --target=calmrisc32 (libgloss/sid)
417 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
419 --target=xscale-elf (newlib/sim)
420 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
424 Also, here is an example of how to run the libstdc++ testsuite
425 for a multilibed build directory with different ABI settings:
429 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
433 You can run the tests with a compiler and library that have
434 already been installed. Make sure that the compiler (e.g.,
435 <code>g++</code>) is in your <code>PATH</code>. If you are
436 using shared libraries, then you must also ensure that the
437 directory containing the shared version of libstdc++ is in your
438 <code>LD_LIBRARY_PATH</code>, or equivalent. If your GCC source
439 tree is at <code>/path/to/gcc</code>, then you can run the tests
444 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
448 The testsuite will create a number of files in the directory in
449 which you run this command,. Some of those files might use the
450 same name as files created by other testsuites (like the ones
451 for GCC and G++), so you should not try to run all the
452 testsuites in parallel from the same directory.
456 In addition, there are some testing options that are mostly of
457 interest to library maintainers and system integrators. As such,
458 these tests may not work on all cpu and host combinations, and
459 may need to be executed in the
460 <emphasis>libbuilddir/testsuite</emphasis> directory. These
461 options include, but are not necessarily limited to, the
470 Five files are generated that determine what test files
471 are run. These files are:
477 <emphasis>testsuite_files</emphasis>
480 This is a list of all the test cases that will be run. Each
481 test case is on a separate line, given with an absolute path
482 from the <emphasis>libsrcdir/testsuite</emphasis> directory.
488 <emphasis>testsuite_files_interactive</emphasis>
491 This is a list of all the interactive test cases, using the
492 same format as the file list above. These tests are not run
499 <emphasis>testsuite_files_performance</emphasis>
502 This is a list of all the performance test cases, using the
503 same format as the file list above. These tests are not run
510 <emphasis>testsuite_thread</emphasis>
513 This file indicates that the host system can run tests which
514 involved multiple threads.
520 <emphasis>testsuite_wchar_t</emphasis>
523 This file indicates that the host system can run the wchar_t
524 tests, and corresponds to the macro definition <code>
525 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
535 The library ABI can be tested. This involves testing the shared
536 library against an ABI-defining previous version of symbol
545 This rule compiles, but does not link or execute, the
546 <emphasis>testsuite_files</emphasis> test cases and displays the
551 make check-performance
555 This rule runs through the
556 <emphasis>testsuite_files_performance</emphasis> test cases and
557 collects information for performance analysis and can be used to
558 spot performance regressions. Various timing information is
559 collected, as well as number of hard page faults, and memory
560 used. This is not run by default, and the implementation is in
565 We are interested in any strange failures of the testsuite;
566 please email the main libstdc++ mailing list if you see
567 something odd or have questions.
571 <sect3 id="test.run.permutations" xreflabel="test.run.permutations">
572 <title>Test Permutations</title>
574 To run the libstdc++ test suite under the <link
575 linkend="manual.ext.debug_mode">debug mode</link>, edit
576 <filename>libstdc++-v3/scripts/testsuite_flags</filename> to add the
577 compile-time flag <constant>-D_GLIBCXX_DEBUG</constant> to the
578 result printed by the <literal>--build-cxx</literal>
579 option. Additionally, add the
580 <constant>-D_GLIBCXX_DEBUG_PEDANTIC</constant> flag to turn on
581 pedantic checking. The libstdc++ test suite should produce
582 precisely the same results under debug mode that it does under
583 release mode: any deviation indicates an error in either the
584 library or the test suite.
588 Or, just run the testsuites with <constant>CXXFLAGS</constant>
589 set to <constant>-D_GLIBCXX_DEBUG</constant>.
594 <sect2 id="test.new_tests" xreflabel="test.new_tests">
595 <title>New Test Cases</title>
598 The first step in making a new test case is to choose the correct
599 directory and file name, given the organization as previously
604 All files are copyright the FSF, and GPL'd: this is very
605 important. The first copyright year should correspond to the date
606 the file was checked in to SVN.
610 As per the dejagnu instructions, always return 0 from main to
615 A bunch of utility functions and classes have already been
616 abstracted out into the testsuite utility library, <code>
617 libtestc++</code>. To use this functionality, just include the
618 appropriate header file: the library or specific object files will
619 automatically be linked in as part of the testsuite run.
623 For a test that needs to take advantage of the dejagnu test
624 harness, what follows below is a list of special keyword that
625 harness uses. Basically, a test case contains dg-keywords (see
626 dg.exp) indicating what to do and what kinds of behavior are to be
627 expected. New test cases should be written with the new style
628 DejaGnu framework in mind.
632 To ease transition, here is the list of dg-keyword documentation
637 # The currently supported options are:
642 # dg-options "options ..." [{ target selector }]
643 # specify special options to pass to the tool (eg: compiler)
645 # dg-do do-what-keyword [{ target/xfail selector }]
646 # `do-what-keyword' is tool specific and is passed unchanged to
647 # ${tool}-dg-test. An example is gcc where `keyword' can be any of:
648 # preprocess|compile|assemble|link|run
649 # and will do one of: produce a .i, produce a .s, produce a .o,
650 # produce an a.out, or produce an a.out and run it (the default is
653 # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
654 # indicate an error message <regexp> is expected on this line
655 # (the test fails if it doesn't occur)
656 # Linenum=0 for general tool messages (eg: -V arg missing).
657 # "." means the current line.
659 # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
660 # indicate a warning message <regexp> is expected on this line
661 # (the test fails if it doesn't occur)
663 # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
664 # indicate a bogus error message <regexp> use to occur here
665 # (the test fails if it does occur)
667 # dg-build regexp comment [{ target/xfail selector }]
668 # indicate the build use to fail for some reason
669 # (errors covered here include bad assembler generated, tool crashes,
671 # (the test fails if it does occur)
673 # dg-excess-errors comment [{ target/xfail selector }]
674 # indicate excess errors are expected (any line)
675 # (this should only be used sparingly and temporarily)
677 # dg-output regexp [{ target selector }]
678 # indicate the expected output of the program is <regexp>
679 # (there may be multiple occurrences of this, they are concatenated)
681 # dg-final { tcl code }
682 # add some tcl code to be run at the end
683 # (there may be multiple occurrences of this, they are concatenated)
684 # (unbalanced braces must be \-escaped)
686 # "{ target selector }" is a list of expressions that determine whether the
687 # test succeeds or fails for a particular target, or in some cases whether the
688 # option applies for a particular target. If the case of `dg-do' it specifies
689 # whether the test case is even attempted on the specified target.
691 # The target selector is always optional. The format is one of:
693 # { xfail *-*-* ... } - the test is expected to fail for the given targets
694 # { target *-*-* ... } - the option only applies to the given targets
696 # At least one target must be specified, use *-*-* for "all targets".
697 # At present it is not possible to specify both `xfail' and `target'.
698 # "native" may be used in place of "*-*-*".
700 Example 1: Testing compilation only
703 Example 2: Testing for expected warnings on line 36, which all targets fail
704 // { dg-warning "string literals" "" { xfail *-*-* } 36
706 Example 3: Testing for expected warnings on line 36
707 // { dg-warning "string literals" "" { target *-*-* } 36
709 Example 4: Testing for compilation errors on line 41
711 // { dg-error "no match for" "" { target *-*-* } 41 }
713 Example 5: Testing with special command line settings, or without the
714 use of pre-compiled headers, in particular the stdc++.h.gch file. Any
715 options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
716 up in the normal.exp file.
717 // { dg-options "-O0" { target *-*-* } }
721 More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
727 <sect2 id="test.dejagnu" xreflabel="test.dejagnu">
728 <title>Test Harness Details</title>
730 Underlying details of testing are abstracted via the GNU Dejagnu package.
734 <para>This is information for those looking at making changes to the testsuite
735 structure, and/or needing to trace dejagnu's actions with --verbose. This
736 will not be useful to people who are "merely" adding new tests to the existing
740 <para>The first key point when working with dejagnu is the idea of a "tool".
741 Files, directories, and functions are all implicitly used when they are
742 named after the tool in use. Here, the tool will always be "libstdc++".
745 <para>The <code>lib</code> subdir contains support routines. The
746 <code>lib/libstdc++.exp</code> file ("support library") is loaded
747 automagically, and must explicitly load the others. For example, files can
748 be copied from the core compiler's support directory into <code>lib</code>.
751 <para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are
752 our own. Callbacks must be prefixed with the name of the tool. To easily
753 distinguish the others, by convention our own routines are named "v3-*".
756 <para>The next key point when working with dejagnu is "test files". Any
757 directory whose name starts with the tool name will be searched for test files.
758 (We have only one.) In those directories, any <code>.exp</code> file is
759 considered a test file, and will be run in turn. Our main test file is called
760 <code>normal.exp</code>; it runs all the tests in testsuite_files using the
761 callbacks loaded from the support library.
764 <para>The <code>config</code> directory is searched for any particular "target
765 board" information unique to this library. This is currently unused and sets
766 only default variables.
771 <sect2 id="test.future" xreflabel="test.future">
772 <title>Future</title>
778 Shared runs need to be implemented, for targets that support shared libraries.
782 Diffing of expected output to standard streams needs to be finished off.
786 The V3 testing framework supports, or will eventually support,
787 additional keywords for the purpose of easing the job of writing
788 test cases. All V3-keywords are of the form <code>@xxx@</code>.
789 Currently plans for supported keywords include:
793 <varlistentry><term> <code> @require@ <files> </code> </term>
796 The existence of <files> is essential for the test to complete
797 successfully. For example, a test case foo.C using bar.baz as
801 // @require@ bar.baz</programlisting>
803 The special variable % stands for the rootname, e.g. the
804 file-name without its `.C' extension. Example of use (taken
805 verbatim from 27_io/filebuf.cc)
808 // @require@ %-*.tst %-*.txt</programlisting>
809 </listitem></varlistentry>
810 <varlistentry><term> <code> @diff@ <first-list> <second-list> </code> </term>
813 After the test case compiles and ran successfully, diff
814 <first-list> against <second-list>, these lists should
815 have the same length. The test fails if diff returns non-zero a
818 </listitem></varlistentry>