source: trunk/test/README @ 2228

Last change on this file since 2228 was 2228, checked in by Peter, 12 years ago

Using the parallel-tests driver. All tests are now by default verbose,
but 'make check' is still silent and the output is collected in log
files. Please refer to test/README or the Automake manual for more
details.

  • Property svn:eol-style set to native
  • Property svn:keywords set to Author Date Id Revision
File size: 3.6 KB
Line 
1$Id: README 2228 2010-03-25 12:28:14Z peter $
2
3    The yat test suite
4
5
6Usage:
7========
8
9Running the tests
10-----------------
11
12  To run all tests:
13
14    make check
15
16  To run only failed tests:
17
18    make recheck
19
20  To run only a subset of tests, use TESTS variable:
21
22    make check TESTS="first_test second_test"
23
24  To enable test output you can use the VERBOSE variable
25
26    make check VERBOSE=1
27
28Interpretation
29--------------
30
31  Successes:
32    PASS  - success
33    XFAIL - expected failure
34
35  Failures:
36    FAIL  - failure
37    XPASS - unexpected success
38
39  Other:
40    SKIP  - skipped test
41
42
43Getting details from failures
44-----------------------------
45
46  You can run the test directly, `./foo_test'; it will be verbose. The
47  verbose output is also available in `foo_test.log'. A summary log is
48  created in the file `test-suite.log'.
49
50
51Reporting failures
52------------------
53
54  Open a 'New Ticket' in the issue tracking system
55  (http://dev.thep.lu.se/yat) with verbose output of failing tests
56  along with which version of yat and which operating system
57  etc. Alternatively, you can send a bug report to the address given in
58  the failed `make check' or in `configure --help'.
59
60
61
62Writing tests
63=============
64
65  If you plan to fix a bug, write the test first. This way you will
66  make sure the test catches the bug, and that it succeeds once you
67  have fixed the bug.
68
69  Use the test::Suite class liberally. See other tests for examples of
70  typical usage.
71
72  Tests shall return 0 at success and -1 otherwise (see
73  test::Suite::return_value()). If the test should be skipped for some
74  reason it shall return 77.
75
76  When writing numerical tests, it is typically desired to allow for
77  some small errors in the comparisons. When comparing two doubles
78  computed within the same program, the error bound should scale with
79  the machine precision. See test::Suite::equal(3). In other case, for
80  example, when the comparison includes a double distributed in
81  'test/data' the error bound should be fixed and reflect the
82  precision in the distributed file. See test::Suite::equal_fix(3).
83
84  If a test needs a data file, place the data in directory
85  'test/data'. However, you should feel free to re-use the data files
86  already there. It should be possible to run the test suite also in a
87  VPATH build in which builddir and srcdir are not the same. To avoid
88  problems with files not found in VPATH builds, use the function
89  test::filename() which gives you the absolute path to the file.
90
91  If a test grows large, avoid having several tests in one long
92  main. It creates long correlations and complicates debugging and
93  extending the test. If you don't want to split the test into several
94  files, split the test into several independent sub-functions.
95
96
97======================================================================
98Copyright (C) 2005 Peter Johansson
99Copyright (C) 2006 Jari Häkkinen
100Copyright (C) 2007, 2008 Jari Häkkinen, Peter Johansson
101Copyright (C) 2009 Peter Johansson
102
103This file is part of yat library, http://dev.thep.lu.se/yat
104
105The yat library is free software; you can redistribute it and/or
106modify it under the terms of the GNU General Public License as
107published by the Free Software Foundation; either version 3 of the
108License, or (at your option) any later version.
109
110The yat library is distributed in the hope that it will be useful, but
111WITHOUT ANY WARRANTY; without even the implied warranty of
112MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
113General Public License for more details.
114
115You should have received a copy of the GNU General Public License
116along with yat. If not, see <http://www.gnu.org/licenses/>.
117======================================================================
118
119
Note: See TracBrowser for help on using the repository browser.