source: trunk/test/README @ 1092

$Id: README 1092 2010-06-12 23:50:18Z peter $

    The svndigest test suite


Usage:
========

Running the tests
-----------------

  To run all tests:

    make check

  To rerun only failed tests:

    make recheck

  To rerun only those tests for there are newer sources

    make lazycheck

  To run only a subset of tests, use TESTS variable:

    make check TESTS="first_test second_test"

  To enable test output you can use the VERBOSE variable

    make check VERBOSE=1

Interpretation
--------------

  Successes:
    PASS  - success
    XFAIL - expected failure

  Failures:
    FAIL  - failure
    XPASS - unexpected success

  Other:
    SKIP  - skipped test


Getting details from failures
-----------------------------

  You can run the test directly, `./foo_test'; it will be verbose. The
  verbose output is also available in `foo_test.log'. A summary log is
  created in the file `test-suite.log'.


Reporting failures
------------------

  Open a 'New Ticket' in the issue tracking system
  (http://dev.thep.lu.se/svndigest) with verbose output of failing
  tests along with which version of svndigest and which operating
  system etc. Alternatively, you can send a bug report to the address
  given in the failed `make check' or in `configure --help'.



Writing tests
=============

  If you plan to fix a bug, write the test first. This way you will
  make sure the test catches the bug, and that it succeeds once you
  have fixed the bug.

  Add a copyright and license followed by one or two lines describing
  what the test does.

  Tests shall returns 0 at success, 77 if it should be skipped, and
  something else at failure. Tests should be silent also at failure
  unless option `-v` is passed or env variable VERBOSE is set.

  Add the test to XFAIL_TESTS in Makefile.am, if it tests a bug still
  present, i.e., the test is expected to fail.

  Note that that if tests are initialized correctly using either Suite
  class (see section C++ tests) or via init.sh (see shell tests) a
  sub-dir testSubDir/foo is created from which the test is run.

C++ tests
---------

  Use the test::Suite class liberally. See other tests for examples of
  typical usage.

  Set `need_test_repo` to true in Suite constructor if the test uses
  the test repository. That will result in the test being skipped if
  test repo is not available, or if it is available the wc will
  checked out (if needed) and updated.

  If a test needs a data file, place the data in directory
  'test/data'. However, you should feel free to re-use the data files
  already there. It should be possible to run the test suite also in a
  VPATH build in which builddir and srcdir are not the same. To avoid
  problems with files not found in VPATH builds, use the function
  test::src_filename() which gives you the absolute path to the
  file. If the data file is generated during build, e.g., checked out
  in a wc and residing in builddir, use test:filename instead.

  Add the test to check_PROGRAMS in Makefile.am.

Shell tests
-----------

  Use `required=...` for setting a space delimited list of required
  tools (see other tests and init.sh). If `repo` is included in
  `required`, a toy_project wc will be checked out and updated if test
  repo is available, otherwise test is skipped.

  Include init.sh, with `. ./init.sh`.

  Use variables set in init.sh such as $srcdir and $abs_top_builddir, and $SVN.

  Do not run svndigest and svncopyright directly, but use functions
  SVNDIGEST_run and SVNCOPYRIGHT_run.

  Do not use `exit`, but use functions `exit_fail()`, `exit_skip()`,
  or `exit_success()`.

  Make sure to set svn property `svn:executable` on test.

  Add test to `distributed_TESTS` in Makefile.am. In order to get the
  lazycheck to behave as expected, declare accurate dependencies. If
  test 'foo.sh', for example, depends on file 'bar.txt', create a make
  rule: 'foo.log: bar.txt'.

======================================================================
Copyright (C) 2009, 2010 Peter Johansson

This file is part of svndigest, http://dev.thep.lu.se/svndigest

The svndigest is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or (at
your option) any later version.

The svndigest is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.

You should have received a copy of the GNU General Public License
along with svndigest. If not, see <http://www.gnu.org/licenses/>.
======================================================================