source: trunk/test/README @ 2238

$Id: README 2238 2010-04-04 23:58:40Z peter $

    The yat test suite


Usage:
========

Running the tests
-----------------

  To run all tests:

    make check

  To rerun only failed tests:

    make recheck

  To rerun only those tests for there are newer sources

    make lazycheck

  To run only a subset of tests, use TESTS variable:

    make check TESTS="first_test second_test"

  To enable test output you can use the VERBOSE variable

    make check VERBOSE=1

Interpretation
--------------

  Successes:
    PASS  - success
    XFAIL - expected failure

  Failures:
    FAIL  - failure
    XPASS - unexpected success

  Other:
    SKIP  - skipped test


Getting details from failures
-----------------------------

  You can run the test directly, `./foo_test'; it will be verbose. The
  verbose output is also available in `foo_test.log'. A summary log is
  created in the file `test-suite.log'.


Reporting failures
------------------

  Open a 'New Ticket' in the issue tracking system
  (http://dev.thep.lu.se/yat) with verbose output of failing tests
  along with which version of yat and which operating system
  etc. Alternatively, you can send a bug report to the address given in
  the failed `make check' or in `configure --help'.



Writing tests
=============

  If you plan to fix a bug, write the test first. This way you will
  make sure the test catches the bug, and that it succeeds once you
  have fixed the bug.

  Use the test::Suite class liberally. See other tests for examples of
  typical usage.

  Tests shall return 0 at success and -1 otherwise (see
  test::Suite::return_value()). If the test should be skipped for some
  reason it shall return 77.

  When writing numerical tests, it is typically desired to allow for
  some small errors in the comparisons. When comparing two doubles
  computed within the same program, the error bound should scale with
  the machine precision. See test::Suite::equal(3). In other case, for
  example, when the comparison includes a double distributed in
  'test/data' the error bound should be fixed and reflect the
  precision in the distributed file. See test::Suite::equal_fix(3).

  If a test needs a data file, place the data in directory
  'test/data'. However, you should feel free to re-use the data files
  already there. It should be possible to run the test suite also in a
  VPATH build in which builddir and srcdir are not the same. To avoid
  problems with files not found in VPATH builds, use the function
  test::filename() which gives you the absolute path to the file.

  If a test grows large, avoid having several tests in one long
  main. It creates long correlations and complicates debugging and
  extending the test. If you don't want to split the test into several
  files, split the test into several independent sub-functions.

C++ tests
---------

  Use the test::Suite class liberally. See other tests for examples of
  typical usage. Add the test to EXTRA_PROGRAMS in Makefile.am.

Shell tests
-----------

  Use 'required=...' for setting a space delimited list of required
  tools (see other tests and common_defs.sh). Include common_defs.sh,
  with '. ./common_defs.sh'.

  Add test to 'DISTRIBUTED_TESTS' in Makefile.am. In order to get the
  lazycheck to behave as expected, declare accurate dependencies. If
  test 'foo.sh', for example, depends on file 'bar.txt', create a make
  rule: 'foo.log: bar.txt'.

  Make sure to set svn property 'svn:executable' on test.

======================================================================
Copyright (C) 2005 Peter Johansson
Copyright (C) 2006 Jari Häkkinen
Copyright (C) 2007, 2008 Jari Häkkinen, Peter Johansson
Copyright (C) 2009 Peter Johansson

This file is part of yat library, http://dev.thep.lu.se/yat

The yat library is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 3 of the
License, or (at your option) any later version.

The yat library is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.

You should have received a copy of the GNU General Public License
along with yat. If not, see <http://www.gnu.org/licenses/>.
======================================================================