2 WvTest: the dumbest cross-platform test framework that could possibly work
3 ==========================================================================
5 I have a problem with your unit testing framework. Yes, you.
6 The person reading this. No, don't look away guiltily. You
7 know your unit testing framework sucks. You know it has a
8 million features you don't understand. You know you hate it,
9 and it hates you. Don't you?
11 Okay, fine. Let's be honest. Actually, I don't know who you
12 are or how you feel about your unit testing framework, but I've
13 tried a lot of them, and I don't like any of them. WvTest is
14 the first one I don't hate, at least sort of. That might be
15 because I'm crazy and I only like things I design, or it might
16 be because I'm crazy and therefore I'm the only one capable of
17 designing a likable unit testing framework. Who am I to say?
19 Here are the fundamental design goals of WvTest:
21 - Be the stupidest thing that can possibly work. People are
22 way, way too serious about their testing frameworks. Some
23 people build testing frameworks as their *full time job*.
24 This is ridiculous. A test framework, at its core, only does
25 one thing: it runs a program that returns true or false. If
26 it's false, you lose. If it's true, you win. Everything
27 after that is gravy. And WvTest has only a minimal amount of
30 - Be a protocol, not an API. If you don't like my API, you can
31 write your own, and it can still be WvTest and it can still
32 integrate with other WvTest tools. If you're stuck with
33 JUnit or NUnit, you can just make your JUnit/NUnit test
34 produce WvTest-compatible output if you want (although I've
35 never done this, so you'll have to do it yourself). I'll
36 describe the protocol below.
38 - Work with multiple languages on multiple operating systems.
39 I'm a programmer who programs on Linux, MacOS, and Windows,
40 to name just three, and I write in lots of programming
41 languages, including C, C++, C#, Python, Perl, and others.
42 And worse, some of my projects use *multiple* languages and I
43 want to have unit tests for *all* of them. I don't know of
44 any unit testing framework - except maybe some horrendously
45 overdesigned ones - that work with multiple languages at
48 - NO UNNECESSARY OBJECT ORIENTATION. The big unit testing
49 craze seems to have been started by JUnit in Java, which is
50 object-oriented. Now, that's not a misdesign in JUnit; it's
51 a misdesign in Java. You see, you can't *not* encapsulate
52 absolutely everything in Java in a class, so it's perfectly
53 normal for JUnit to require you to encapsulate everything in
54 a class. That's not true of almost any other language
55 (except C#), and yet *every* clone of JUnit in *every*
56 language seems to have copied its classes and objects. Well,
57 that's stupid. WvTest is designed around the simple idea of
58 test *functions*. WvTest runs your function, it checks a
59 bunch of stuff and it returns or else it dies horribly. If
60 your function wants to instantiate some objects while it does
61 that, then that's great; WvTest doesn't care. And yes, you
62 can assert whether two variables are equal even if your
63 function *isn't* in a particular class, just as God intended.
65 - Don't make me name or describe my individual tests. How many
66 times have you seen this?
68 assertTrue(thing.works(), "thing didn't work!");
70 The reasoning there is that if the test fails, we want to be
71 able to print a user-friendly error message that describes
72 why. Right? NO!! That is *awful*. That just *doubled* the
73 amount of work you have to do in order to write a test.
74 Instead, WvTest auto-generates output including the line
75 number of the test and the code on that line. So you get a
78 ! mytest.t.cc:431 thing.works() FAILED
80 and all you have to write is this:
82 WVPASS(thing.works());
84 (WVPASS is all-caps because it's a macro in C++, but also
85 because you want your tests to stand out. That's what
86 you'll be looking for when it fails, after all. And don't
87 even get me started about the 'True' in assertTrue. Come
88 on, *obviously* you're going to assert that the condition is
91 - No setup() and teardown() functions or fixtures. "Ouch!" you
92 say. "I'm going to have so much duplicated code!" No, only
93 if you're an idiot. You know what setup() and teardown() are
94 code names for? Constructor and destructor. Create some
95 objects and give them constructors and destructors, and I
96 think you'll find that, like magic, you've just invented
97 "test fixtures." Nothing any test framework can possibly do
98 will make that any easier. In fact, everything test
99 frameworks *try* to do with test fixtures just makes it
100 harder to write, read, and understand. Forget it.
102 - Big long scary test functions. Some test frameworks are
103 insistent about the rule that "every function should test
104 only one thing." Nobody ever really explains why. I can't
105 understand this; it just causes uncontrolled
106 hormone-imbalance hypergrowth in your test files, and you
107 have to type more stuff... and run test fixtures over and
110 My personal theory for why people hate big long test
111 functions: it's because their assertTrue() implementation
112 doesn't say which test failed, so they'd like the *name of
113 the function* to be the name of the failed test. Well,
114 that's a cute workaround to a problem you shouldn't have had
115 in the first place. With WvTest, WVPASS() actually tells you
116 exactly what passed and what failed, so it's perfectly okay -
117 and totally comprehensible - to have a sequence of five
118 things in a row where only thing number five failed.
124 WvTest is a protocol, not really an API. As it happens, the
125 WvTest project includes several (currently five)
126 implementations of APIs that produce data in the WvTest format,
127 but it's super easy to add your own.
129 The format is really simple too. It looks like this:
131 Testing "my test function" in mytest.t.cc:
132 ! mytest.t.cc:432 thing.works() ok
133 This is just some crap that I printed while counting to 3.
134 ! mytest.t.cc.433 3 < 4 FAILED
136 There are only four kinds of lines in WvTest, and each of the
137 lines above corresponds to one of them:
139 - Test function header. A line that starts with the word
140 Testing (no leading whitespace) and then has a test function
141 name in double quotes, then "in", then the filename, and then
142 colon, marks the beginning of a test function.
144 - A passing assertion. Any line that starts with ! and ends with
145 " ok" (whitespace, the word "ok", and a newline) indicates
146 one assertion that passed. The first "word" on that line is
147 the "name" of that assertion (which can be anything, as long
148 as it doesn't contain any whitespace). Everything between the
149 name and the ok is just some additional user-readable detail
150 about the test that passed.
152 - Random filler. If it doesn't start with an ! and it doesn't
153 look like a header, then it's completely ignored by anything
154 using WvTest. Your program can print all the debug output it
155 wants, and WvTest won't care, except that you can retrieve it
156 later in case you're wondering why a test failed. Naturally,
157 random filler *before* an assertion is considered to be
158 associated with that assertion; the assertion itself is the
161 - A failing assertion. This is just like an 'ok' line, except
162 augmented with extra detail. A more advanced parser could choose to
163 parse the extra string to count partial failures:
165 xfail ok - test was marked as known to fail and failed
166 (i.e. a known breakage)
168 xpass ok - test was marked as known to fail and passed
169 (i.e. previously known breakage is fixed)
171 skip ok - test was skipped.
173 and it could be something else instead, if you invent a new and improved way
177 Reading the WvTest Protocol: wvtestrun
178 --------------------------------------
180 WvTest provides a simple perl script called wvtestrun, which
181 runs a test program and parses its output. It works like this:
184 ../wvtestrun ./wvtest.py t/twvtest.py
186 (Why can't we just pipe the output to wvtestrun, instead of
187 having wvtestrun run the test program? Three reasons: first, a
188 fancier version of wvtestrun could re-run the tests several
189 times or give a GUI that lets you re-run the test when you push
190 a button. Second, it handles stdout and stderr separately.
191 And third, it can kill the test program if it gets stuck
192 without producing test output for too long.)
194 If we put the sample output from the previous section through
195 wvtestrun (and changed the FAILED to ok), it would produce this:
197 $ ./wvtestrun cat sample-ok
199 Testing "all" in cat sample-ok:
200 ! mytest.t.cc my ok test function: ..... 0.010s ok
202 WvTest: 5 tests, 0 failures, total time 0.010s.
203 WvTest: 0 tests skipped, 0 known breakages, 0 fixed breakages.
205 WvTest result code: 0
207 What happened here? Well, wvtestrun took each test header (in
208 this case, there's just one, which said we're testing "my test
209 function" in mytest.t.cc) and turns it into a single test line.
210 Then it prints a dot for each assertion in that test function,
211 tells you the total time to run that function, and prints 'ok'
212 if the entire test function failed.
214 Note that the output of wvtestrun is *also* valid WvTest output.
215 That means you can use wvtestrun in your 'make test' target in a
216 subdirectory, and still use wvtestrun as the 'make test' runner
217 in the parent directory as well. As long as your top-level
218 'make test' runs in wvtestrun, all the WvTest output will be
219 conveniently summarized into a *single* test output.
221 Now, what if the test had failed? Then it would look like this:
223 $ ./wvtestrun cat sample-error
225 Testing "all" in cat sample-error:
226 ! mytest.t.cc my error test function: .
227 ! mytest.t.cc:432 thing.works() ok
228 This is just some crap that I printed while counting to 3.
229 ! mytest.t.cc.433 3 < 4 FAILED
232 WvTest: 5 tests, 1 failure, total time 0.000s.
233 WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.
235 WvTest result code: 0
237 What happened there? Well, because there were failed tests,
238 wvtestrun decided you'd probably want to see the detailed output
239 for that test function, so it expanded it out for you. The line
240 with the dots is still there, but since it doesn't have an 'ok',
241 it's considered a failure too, just in case.
243 Watch what happens if we run a test with both the passing, and
244 then the failing, test functions:
246 $ ./wvtestrun cat sample-ok sample-error
248 Testing "all" in cat sample-ok sample-error:
249 ! mytest.t.cc my ok test function: ..... 0.000s ok
250 ! mytest.t.cc my error test function: .
251 ! mytest.t.cc:432 thing.works() ok
252 This is just some crap that I printed while counting to 3.
253 ! mytest.t.cc.433 3 < 4 FAILED
256 WvTest: 10 tests, 1 failure, total time 0.000s.
257 WvTest: 1 test skipped, 1 known breakage, 1 fixed breakage.
259 WvTest result code: 0
261 Notice how the messages from sample-ok are condensed; only the
262 details from sample-error are expanded out, because only that
263 output is interesting.
266 How do I actually write WvTest tests?
267 -------------------------------------
269 Sample code is provided for these languages:
271 C: try typing "cd c; make test"
272 C++: try typing "cd cpp; make test"
273 C# (mono): try typing "cd dotnet; make test"
274 Python: try typing "cd python; make test"
275 Shell: try typing "cd sh; make test"
277 There's no point explaining the syntax here, because it's really
278 simple. Just look inside the cpp, dotnet, python, and sh
279 directories to learn how the tests are written.
282 How should I embed WvTest into my own program?
283 ----------------------------------------------
285 The easiest way is to just copy the WvTest source files for your
286 favourite language into your project. The WvTest protocol is
287 unlikely to ever change - at least not in a
288 backwards-incompatible way - so it's no big deal if you end up
289 using an "old" version of WvTest in your program. It should
290 still work with updated versions of wvtestrun (or wvtestrun-like
293 Another way is to put the WvTest project in a subdirectory of
294 your project, for example, using 'svn:externals',
295 'git submodule', or 'git subtree'.
298 How do I run just certain tests?
299 --------------------------------
301 Unfortunately, the command-line syntax for running just *some*
302 of your tests varies depending which WvTest language you're using.
303 For C, C++ or C#, you link an executable with wvtestmain.c or
304 wvtestmain.cc or wvtestmain.cs, respectively, and then you can
305 provide strings on the command line. Test functions will run only
306 if they have names that start with one of the provided strings:
309 ../../wvtestrun ./wvtest myfunc otherfunc
311 With python, since there's no linker, you have to just tell it
315 ../wvtestrun ./wvtest.py ...filenames...
318 What else can parse WvTest output?
319 ----------------------------------
321 It's easy to parse WvTest output however you like; for example,
322 you could write a GUI program that does it. We had a tcl/tk
323 program that did it once, but we threw it away since the
324 command-line wvtestrun is better anyway.
326 One other program that can parse WvTest output is gitbuilder
327 (http://github.com/apenwarr/gitbuilder/), an autobuilder tool
328 for git. It reports a build failure automatically if there are
329 any WvTest-style failed tests in the build output.
332 Other Assorted Questions
333 ------------------------
336 What does the "Wv" stand for?
338 Either "Worldvisions" or "Weaver", both of which were part of the
339 name of the Nitix operating system before it was called Nitix, and
340 *long* before it was later purchased by IBM and renamed to Lotus
343 It does *not* stand for World Vision (sigh) or West Virginia.
345 Who owns the copyright?
347 While I (Avery) wrote most of the WvTest framework in C++, C#, and
348 Python, and I also wrote wvtestrunner, the actual code I wrote is
349 owned by whichever company I wrote it for at the time. For the most
352 C++: Net Integration Technologies, Inc. (now part of IBM)
353 C#: Versabanq Innovations Inc.
354 Python: EQL Data Inc.
356 What can I do with it?
358 WvTest is distributed under the terms of the GNU LGPLv2. See the
359 file LICENSE for more information.
361 Basically this means you can use it for whatever you want, but if
362 you change it, you probably need to give your changes back to the
363 world. If you *use* it in your program (which is presumably a test
364 program) you do *not* have to give out your program, only
365 WvTest itself. But read the LICENSE in detail to be sure.
367 Where did you get the awesome idea to use a protocol instead of an API?
369 The perl source code (not to be confused with perlunit)
370 did a similar trick for the perl interpreter's unit
371 test, although in a less general way. Naturally, you
372 shouldn't blame them for how I mangled their ideas, but
373 I never would have thought of it if it weren't for them.
375 Why are xfail/xpass/skip needed at all?
377 Suppose you have a test which you expect to pass, but which is
378 failing (in the usual sense). You have two choices: 1) fix it right
379 now, or 2) defer fixing. Sometimes at the start we have lots of
380 tests failing, and in order to make gradual progress, it makes sense
381 to mark those presently-failing tests as "I know, it fails".
383 Another use case is when tests always pass on e.g. Linux, but some
384 of them fail on Win32 due to differences in environment, and one
385 does not want to concentrate on fixing win32 yet.
387 So when you run tests again, you'd like to differentiate between
388 failing tests marked as xfail (known to fail) and new failing tests.
389 The latter should produce real FAILURE with details. xfail on the
390 other hand should produce just a small warning/reminder (I'm failing
391 here, please don't forget to fix me).
395 So now what happens when a test marked as xfail passes anyway
396 instead of the expected failure? That's not a failure - quite
397 differently, it _was_ a failure, and now it passes, so what should
398 we do? Right, we should tell the user that "hey, a test which used
399 to fail now passes! You probably would want to mark it back as
402 This is how xfail/xpass works.
404 With introduction of xfail tests are no longer strictly true or
405 false, but you can ignore the additional information if you want
406 (the last word is still either "ok" or not).
408 Who should I complain to about WvTest?
410 Email me at: Avery Pennarun <apenwarr@gmail.com>
412 I will be happy to read your complaints, because I actually really
413 like it when people use my programs, especially if they hate them.
414 It fills the loneliness somehow and prevents me from writing bad
417 Testing makes me gouge out my eyes
418 But with WvTest, it takes fewer tries.
419 WvTest is great, wvtest is fun!
420 Don't forget to call wvtestrun.