Description of Testcases for OMK
================================

Each test resides in a separate directory. Every directory can
contains multiple testcases. The directories can be nested. There is
no difference between the tests in different levels of directory
structure.

Structure of a Testcase
-----------------------

A directory contains a testcase if it contains at least one file that
matches runtest* pattern (files ending with "~" or ".rules" are not
counted).

* Testcase Definition:

  Each test is represented by a set of files. Some of these files are
  used as "data" files for the test (e.g. source code, config.omk,
  etc.) and some files controls how is the test run. Here follows the
  description of control files in the test directory.

  - runtest*: is the executable that runs the test(s). It is possible
    to have more files in a test directory, whose name starts with
    `runtest'. In that case all these files will be executed in
    alphabetical order as separate testcases. The goal is to be able
    to run several tests on the same structure of files and
    directories.

  - runtest*.rules: Specifies the set of rules the testcase applies
    to. If there is more runtest files, every testcase can have its
    own .rules file. The syntax of this files is described in section
    `Rules description'.

* Testcase Execution:

  Runtest is executed with OMK_RULES environment variable set to the
  name of the actual rules tested and OMK_TESTSROOT to the directory
  containing the `tester.py' script.

* Testcase Output:

  Each execution of testcase should produce the following outputs:

  - Exit status of make or runtest: 
        zero - the testcase was successfully passed
        other than zero - the testcase failed or cannot be executed
              	       	  (e.g. because the needed compiler is not
              	       	  available)

  - Error message:
    In the case of nonzero exit status, if file '_canttest' exists in
    the same directory as Makefile.rules, it is considered that the
    test cannot be executed (i.e. the compiler is missing) and the
    content of that file is provided as error message in results
    log. Otherwise, the test is considered as failed and file '_error'
    can contain more detailed description of the error.

  - stdout and stderr: The output of testcase execution is captured
    and is included in the results log. It should contain useful
    information to ease the debugging process.

Rules Description
-----------------

If the 'runtest*.rules' files are contained in the test directory,
this file defines a set of Makefile.rules the particular testcase can
be applied to. If there is no rules file, this testcase is applied to
all the rules.

The rules file contains one line of one of the following forms:

* <rules name> - This selects only one rules file

* all: - this selects all the rules files

* snip:<snippet name> - this selects all the rules, which contain the
  given snippet.

* python:<python expression> - The python expression is evaluated. The
  global namespace contains the following variables:

    - rules: the name of tested rules

    - snippets: is of type list and contains the snippet names from
      which the tested rules are composed.

  If the expression evaluates as True, the testcase is executed with
  this rules, otherwise it is not.

  Example: python: "config_h" in snippets and rules != "rtems"