# -*- text -*-
# $RCSfile: NOTES,v $
# $Revision: 1.3 $
# $Author: langer $
# $Date: 2010/12/03 22:10:59 $

# This software was produced by NIST, an agency of the U.S. government,
# and by statute is not subject to copyright in the United States.
# Recipients of this software assume all responsibilities associated
# with its operation, modification and maintenance. However, to
# facilitate maintenance we ask that before distributing modified
# versions of this software, you first contact the authors at
# oof_manager@nist.gov. 

------------------------------------------------------------------------------

This file contains notes for developers.  The README file contains
notes for users who would like to test their OOF2 installation.



  Some notes on the regression testing suite for OOF2
  ----------------------------------------------------


The files in this directory make up the regression testing suite for
the OOF2 software.  They form a two-level hierarchy, with the "root"
being the "regression.py" file, and the "leaves" being the various
other "*.py" files.


To run the full regression suite, first ensure that OOF is built on
the system.  The full suite will only run for the "dist" build.  Then,
while in this directory, type "python regression.py".  This will run
all of the tests in the appropriate order, with dependent tests
running after their dependencies.


To run one of the "test modules" contained in one of the leaf *.py
files, the procedure is the same -- type "python <test_module>.py" at
the command prompt in this directory.  All the tests in that module
will then be run.  Of course when run this way, there is no way to
ensure that dependencies of tests in the module have been run, as they
may reside in other modules.


Dependencies
------------

There are two kinds of dependencies that can exist between tests.  The
first of these may be called "weak" dependency, and occurs when the
result of a later, dependent test is only meaningful if the prior
dependency test has succeeded.  For example, an operation on an object
cannot be meaningfully tested if the object cannot be created.  In the
absence of the creation test, the failure of the operation test may
mean that the operation doesn't work, or it may mean that the object
can't be created.  In the presence of the creation test, the meaning
of the failure of the operation test is clearer.

The second kind of dependency may be called "strong" dependency, and
occurs when the dependent test relies on side-effects of the
dependency test.  In several cases in this testing suite, there are,
for example, object deletion test items, which will not run correctly
unless the expected object they want to delete is present in the OOF
system.  In this typical case, the test is strongly dependent on a
corresponding object creation test, which creates the object with the
expected name and attributes.  In the strong-dependency case, the
second test will fail in the absence of the first test, even if there
are no bugs in the OOF code at all.

The testing modules in this directory only have weak dependencies
between them.  Within many of the modules, tests are grouped into test
suite objects, and there are also only weak dependencies between
groups.  There may be strong dependencies within groups.  Anyone
interested in rearranging the tests must of course take this into
account.


Structure of Modules
--------------------

Each test module is in a python file in this directory.  To run the
tests in a module, run
   python regression.py [module names]
in this directory, or
   oof2-test [module names]
in any directory.  If no modules are listed, all modules will be run.
The ".py" suffix is optional in the module name list.

Each module file must contain a list of tests called "test_set".  The
tests are created from subclasses of the TestCase class from the
Python distribution-provided unittest module.  Each TestCase class
declaration defines several functions within a given subclass, each
function corresponding to an individual test.  Tests are created by
giving the function name as an argument to the constructor for the
class, so that each function corresponds to an instance of the
TestCase subclass object.  Besides the individual tests, TestCase
subclass objects also have a setUp and tearDown function, the former
being run before each test, and the latter after.  Tests are thus
naturally grouped roughly according to their preparatory and clean-up
requirements.  In the OOF2 test suite, tests are also grouped
logically according to which sub-hierarchy of the OOF menu tree they
inhabit.

Actual tests are written such that, having done whatever set-up is
required in their class's setUp routine, they run the OOF menu command
being tested, and the examine the results using the TestCase's
"assert" functions.  These functions inspect various conditions, and
if the conditions are not satisfied, will report a test failure.


Notes
-----

Most of the tests which actually exist are rather superficial.  The
most basic thing that is consistently tested is whether or not the
tested menu item runs at all.  Most tests also make a cursory check
that various internal book-keeping devices give answers consistent
with the correct function of the menu item, but beyond this, little is
done to ensure that the menu item really worked as intended.  This is
primarily due to a shortage of time for writing tests.

The most in-depth and sophisticated tests achieve their results by
running menu items on known inputs, saving the result to a file, and
then checking the generated file against a known-good reference file
or known-good result.  This tends to be a feature of later, "all-up"
tests of, for instance, saving a rich mesh or skeleton to a file, or
running the solver on a problem.  These tests suffer from the opposite
problem, in that they are rigorously precise in their requirements,
but their scope is limited to the provided inputs.  While most tests
are shallow, these tests are deep but narrow, so to speak.

Subdirectories of this directory include subsidiary data used by the
tests, either as inputs or for comparison.  This data may need to be
updated from time to time if the OOF behavior changes.
