Description of Testcases for OMK
================================
-Each test resides in a separate directory. The directories can be
-nested. There is no difference between the tests in different levels of
-directory structure.
+Each test resides in a separate directory. Every directory can
+contains multiple testcases. The directories can be nested. There is
+no difference between the tests in different levels of directory
+structure.
Structure of a Testcase
-----------------------
-A directory contains the testcase if it contains at least one of these
-files: Makefile.test, runtest.
+A directory contains a testcase if it contains at least one file that
+matches runtest* pattern (files ending with "~" or ".rules" are not
+counted).
* Testcase Definition:
- - Makefile.test: testcase commands are in the form of makefile. The
- test is run by "make -f Makefile.test".
+ Each test is represented by a set of files. Some of these files are
+ used as "data" files for the test (e.g. source code, config.omk,
+ etc.) and some files controls how is the test run. Here follows the
+ description of control files in the test directory.
- - runtest: executable file that runs the test(s).
+ - runtest*: is the executable that runs the test(s). It is possible
+ to have more files in a test directory, whose name starts with
+ `runtest'. In that case all these files will be executed in
+ alphabetical order as separate testcases. The goal is to be able
+ to run several tests on the same structure of files and
+ directories.
- - rules: Specifies the rules this testcase applies to. The syntax of
- this files is describes in section Rules description.
+ - runtest*.rules: Specifies the set of rules the testcase applies
+ to. If there is more runtest files, every testcase can have its
+ own .rules file. The syntax of this files is describes in section
+ `Rules description'.
+
+* Testcase Execution:
+
+ Runtest is executed with OMK_RULES environment variable set to the
+ name of the actual rules tested and OMK_TESTSROOT to the directory
+ containing the `tester.py' script.
* Testcase Output:
- Exit status of make or runtest:
0 - the testcase was successfully passed
- 1 - testcase failed
- 2 - testcase cannot be executed (e.g. because the needed
+ 1 - the testcase failed
+ 2 - the testcase cannot be executed (e.g. because the needed
compiler is not available)
-
- Error message:
- In the case of error, the file 'error' should contain more
+ In the case of error, the file '_error' should contain more
detailed description of the error.
- - stdout and stderr: TODO
+ - stdout and stderr: The output of testcase execution is captured
+ and is included in the results log. It should contain useful
+ information to ease debugging problems.
Rules Description
-----------------
-If the 'rules' file is contained in the testcase, this file defines
-a set of Makefile.rules this testcase can be applied to. If there is
-no rules file, this testcase is applied to all the rules.
+If the 'runtest*.rules' files are contained in the test directory,
+this file defines a set of Makefile.rules the particular testcase can
+be applied to. If there is no rules file, this testcase is applied to
+all the rules.
The rules file contains one line of one of the following forms:
* snip:<snippet name> - this selects all the rules, which contain the
given snippet.
-* python:<python exression> - The python expression is evaluated. The
+* python:<python expression> - The python expression is evaluated. The
global namespace contains the following variables:
- rules: the name of tested rules
projects / omk / sssa.git / blobdiff