source: trunk/test/README

Last change on this file was 2754, checked in by Peter, 7 years ago

avoid suggesting bug reports to trac, prefer single quote (rather than backtick) as suggest by GCS, and removed a trailing WS.

  • Property svn:eol-style set to native
  • Property svn:keywords set to Author Date Id Revision
File size: 4.6 KB
Line 
1$Id: README 2754 2012-06-28 07:39:57Z peter $
2
3    The yat test suite
4
5
6Usage:
7========
8
9Running the tests
10-----------------
11
12  To run all tests:
13
14    make check
15
16  You can use '-j N' for faster completion on multiprocessor systems.
17
18  To rerun only failed tests:
19
20    make recheck
21
22  To rerun only those tests for which there are newer sources
23
24    make lazycheck
25
26  To run only a subset of tests, use TESTS variable:
27
28    make check TESTS="first_test second_test"
29
30  To enable test output you can use the VERBOSE variable
31
32    make check VERBOSE=1
33
34Interpretation
35--------------
36
37  Successes:
38    PASS  - success
39    XFAIL - expected failure
40
41  Failures:
42    FAIL  - failure
43    XPASS - unexpected success
44
45  Other:
46    SKIP  - skipped test
47
48
49Getting details from failures
50-----------------------------
51
52  You can run the test directly, 'test/foo_test'; it will be verbose. The
53  verbose output is also available in 'test/foo_test.log'. A summary log is
54  created in the file 'test-suite.log'.
55
56
57Reporting failures
58------------------
59
60  If a test fails, please send file 'test-suite.log' to the address
61  given in the failed 'make check' or in './configure --help',
62  together with other relevant info such as version numbers, operating
63  system, etc.
64
65
66Writing tests
67=============
68
69  If you plan to fix a bug, write the test first. This way you will
70  make sure the test catches the bug, and that it succeeds once you
71  have fixed the bug.
72
73  Use the test::Suite class liberally. See other tests for examples of
74  typical usage.
75
76  Tests shall return 0 at success and -1 otherwise (see
77  test::Suite::return_value()). If the test should be skipped for some
78  reason it shall return 77.
79
80  When writing numerical tests, it is typically desired to allow for
81  some small errors in the comparisons. When comparing two doubles
82  computed within the same program, the error bound should scale with
83  the machine precision. See test::Suite::equal(3). In other case, for
84  example, when the comparison includes a double distributed in
85  'test/data' the error bound should be fixed and reflect the
86  precision in the distributed file. See test::Suite::equal_fix(3).
87
88  If a test needs a data file, place the data in directory
89  'test/data'. However, you should feel free to re-use the data files
90  already there. It should be possible to run the test suite also in a
91  VPATH build in which builddir and srcdir are not the same. To avoid
92  problems with files not found in VPATH builds, use the function
93  test::filename() which gives you the absolute path to the file.
94
95  If a test grows large, avoid having several tests in one long
96  main. It creates long correlations and complicates debugging and
97  extending the test. If you don't want to split the test into several
98  files, split the test into several independent sub-functions.
99
100C++ tests
101---------
102
103  Use the test::Suite class liberally. See other tests for examples of
104  typical usage. Add the test to EXTRA_PROGRAMS in Makefile.am. Tests
105  should have the extension '.test' as in 'foo.test' and corresponding
106  source file should be named 'foo.cc'.
107
108Shell tests
109-----------
110
111  Use 'required=...' for setting a space delimited list of required
112  tools (see other tests and common_defs.sh). Include common_defs.sh,
113  with '. ./common_defs.sh || exit 99'.
114
115  Add test to 'DISTRIBUTED_TESTS' in Makefile.am. In order to get the
116  lazycheck to behave as expected, declare accurate dependencies. If
117  test 'foo.sh', for example, depends on file 'bar.txt', create a make
118  rule: 'foo.log: bar.txt'.
119
120  Do not use 'exit' directly, but use functions 'exit_fail', 'exit_skip',
121  'exit_fatal', or 'exit_success'.
122
123  Make sure to set svn property 'svn:executable' on test.
124
125======================================================================
126Copyright (C) 2005 Peter Johansson
127Copyright (C) 2006 Jari Häkkinen
128Copyright (C) 2007, 2008 Jari Häkkinen, Peter Johansson
129Copyright (C) 2009, 2010, 2011, 2012 Peter Johansson
130
131This file is part of yat library, http://dev.thep.lu.se/yat
132
133The yat library is free software; you can redistribute it and/or
134modify it under the terms of the GNU General Public License as
135published by the Free Software Foundation; either version 3 of the
136License, or (at your option) any later version.
137
138The yat library is distributed in the hope that it will be useful, but
139WITHOUT ANY WARRANTY; without even the implied warranty of
140MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
141General Public License for more details.
142
143You should have received a copy of the GNU General Public License
144along with yat. If not, see <http://www.gnu.org/licenses/>.
145======================================================================
Note: See TracBrowser for help on using the repository browser.