Description of Testcases for OMK
================================
-Each test resides in a separate directory. The directories can be
-nested. There is no difference between the tests in different levels of
-directory structure.
+Each test resides in a separate directory. Every directory can
+contains multiple testcases. The directories can be nested. There is
+no difference between the tests in different levels of directory
+structure.
Structure of a Testcase
-----------------------
-A directory contains the testcase if it contains at least one of these
-files: Makefile.test, runtest.
+A directory contains a testcase if it contains at least one file that
+matches runtest* pattern (files ending with "~" or ".rules" are not
+counted).
* Testcase Definition:
- - runtest (mandatory): is the executable that runs the test(s).
+ Each test is represented by a set of files. Some of these files are
+ used as "data" files for the test (e.g. source code, config.omk,
+ etc.) and some files controls how is the test run. Here follows the
+ description of control files in the test directory.
- - rules: Specifies the rules this testcase applies to. The syntax of
- this files is describes in section Rules description.
+ - runtest*: is the executable that runs the test(s). It is possible
+ to have more files in a test directory, whose name starts with
+ `runtest'. In that case all these files will be executed in
+ alphabetical order as separate testcases. The goal is to be able
+ to run several tests on the same structure of files and
+ directories.
+
+ - runtest*.rules: Specifies the set of rules the testcase applies
+ to. If there is more runtest files, every testcase can have its
+ own .rules file. The syntax of this files is described in section
+ `Rules description'.
* Testcase Execution:
Runtest is executed with OMK_RULES environment variable set to the
- name of the actual rules tested.
+ name of the actual rules tested and OMK_TESTSROOT to the directory
+ containing the `tester.py' script.
* Testcase Output:
Each execution of testcase should produce the following outputs:
- Exit status of make or runtest:
- 0 - the testcase was successfully passed
- 1 - testcase failed
- 2 - testcase cannot be executed (e.g. because the needed
- compiler is not available)
-
+ zero - the testcase was successfully passed
+ other than zero - the testcase failed or cannot be executed
+ (e.g. because the needed compiler is not
+ available)
- Error message:
- In the case of error, the file '_error' should contain more
- detailed description of the error.
+ In the case of nonzero exit status, if file '_canttest' exists in
+ the same directory as Makefile.rules, it is considered that the
+ test cannot be executed (i.e. the compiler is missing) and the
+ content of that file is provided as error message in results
+ log. Otherwise, the test is considered as failed and file '_error'
+ can contain more detailed description of the error.
- - stdout and stderr: TODO
+ - stdout and stderr: The output of testcase execution is captured
+ and is included in the results log. It should contain useful
+ information to ease the debugging process.
Rules Description
-----------------
-If the 'rules' file is contained in the testcase, this file defines
-a set of Makefile.rules this testcase can be applied to. If there is
-no rules file, this testcase is applied to all the rules.
+If the 'runtest*.rules' files are contained in the test directory,
+this file defines a set of Makefile.rules the particular testcase can
+be applied to. If there is no rules file, this testcase is applied to
+all the rules.
The rules file contains one line of one of the following forms:
* snip:<snippet name> - this selects all the rules, which contain the
given snippet.
-* python:<python exression> - The python expression is evaluated. The
+* python:<python expression> - The python expression is evaluated. The
global namespace contains the following variables:
- rules: the name of tested rules
projects / omk.git / blobdiff