Description of Testcases for OMK
================================
-Each test resides in a separate directory. The directories can be
-nested. There is no difference between the tests in diffrent levels of
-directory structure.
+Each test resides in a separate directory. Every directory can
+contains multiple testcases. The directories can be nested. There is
+no difference between the tests in different levels of directory
+structure.
Structure of a Testcase
-----------------------
-A directory contain the testcase if it contains at least one of these
-files: Makefile.test, run_test.
+A directory contains a testcase if it contains at least one file that
+matches runtest* pattern (files ending with "~" or ".rules" are not
+counted).
-Definition of a Testcase:
+* Testcase Definition:
-* Makefile.test: testcase commands in the form of makefile. The test is
- run by "make -f Makefile.test". If the exit status is zero, the
- testcase is considered as passed, otherwise as failed.
+ Each test is represented by a set of files. Some of these files are
+ used as "data" files for the test (e.g. source code, config.omk,
+ etc.) and some files controls how is the test run. Here follows the
+ description of control files in the test directory.
-* run_test: executable file that runs the test(s). The exit status has
- the same meaning as in the case of Makefile.test.
+ - runtest*: is the executable that runs the test(s). It is possible
+ to have more files in a test directory, whose name starts with
+ `runtest'. In that case all these files will be executed in
+ alphabetical order as separate testcases. The goal is to be able
+ to run several tests on the same structure of files and
+ directories.
-* rules: Specifies the rules this testcase applies to. The syntax of
- this files is describes in section Rules description.
+ - runtest*.rules: Specifies the set of rules the testcase applies
+ to. If there is more runtest files, every testcase can have its
+ own .rules file. The syntax of this files is described in section
+ `Rules description'.
-Output of testcase:
+* Testcase Execution:
-Besides exit status, the testcases can produce the folowing output.
+ Runtest is executed with OMK_RULES environment variable set to the
+ name of the actual rules tested and OMK_TESTSROOT to the directory
+ containing the `tester.py' script.
-* FIXME: chybove hlaseni - soubor error nebo stderr?
+* Testcase Output:
+
+ Each execution of testcase should produce the following outputs:
+
+ - Exit status of make or runtest:
+ zero - the testcase was successfully passed
+ other than zero - the testcase failed or cannot be executed
+ (e.g. because the needed compiler is not
+ available)
+
+ - Error message:
+ In the case of nonzero exit status, if file '_canttest' exists in
+ the same directory as Makefile.rules, it is considered that the
+ test cannot be executed (i.e. the compiler is missing) and the
+ content of that file is provided as error message in results
+ log. Otherwise, the test is considered as failed and file '_error'
+ can contain more detailed description of the error.
+
+ - stdout and stderr: The output of testcase execution is captured
+ and is included in the results log. It should contain useful
+ information to ease the debugging process.
Rules Description
-----------------
-The rules file contains one or more lines. Each line represents a set
-of zero or more rules. The set of rules to be applied to this testcase
-is union of these sets.
+If the 'runtest*.rules' files are contained in the test directory,
+this file defines a set of Makefile.rules the particular testcase can
+be applied to. If there is no rules file, this testcase is applied to
+all the rules.
+
+The rules file contains one line of one of the following forms:
+
+* <rules name> - This selects only one rules file
+
+* all: - this selects all the rules files
+
+* snip:<snippet name> - this selects all the rules, which contain the
+ given snippet.
+
+* python:<python expression> - The python expression is evaluated. The
+ global namespace contains the following variables:
-Each line can contain:
+ - rules: the name of tested rules
-* <rule name> - This selects rules of this name
+ - snippets: is of type list and contains the snippet names from
+ which the tested rules are composed.
-* all: - this selects all rules
+ If the expression evaluates as True, the testcase is executed with
+ this rules, otherwise it is not.
-* snip:<snippet name> - this selects all the rules, which are composed
- of the given snippet.
+ Example: python: "config_h" in snippets and rules != "rtems"
projects / omk.git / blobdiff