--- /dev/null
+
+WvTest: the dumbest cross-platform test framework that could possibly work
+==========================================================================
+
+I have a problem with your unit testing framework. Yes, you.
+The person reading this. No, don't look away guiltily. You
+know your unit testing framework sucks. You know it has a
+million features you don't understand. You know you hate it,
+and it hates you. Don't you?
+
+Okay, fine. Let's be honest. Actually, I don't know who you
+are or how you feel about your unit testing framework, but I've
+tried a lot of them, and I don't like any of them. WvTest is
+the first one I don't hate, at least sort of. That might be
+because I'm crazy and I only like things I design, or it might
+be because I'm crazy and therefore I'm the only one capable of
+designing a likable unit testing framework. Who am I to say?
+
+Here are the fundamental design goals of WvTest:
+
+ - Be the stupidest thing that can possibly work. People are
+ way, way too serious about their testing frameworks. Some
+ people build testing frameworks as their *full time job*.
+ This is ridiculous. A test framework, at its core, only does
+ one thing: it runs a program that returns true or false. If
+ it's false, you lose. If it's true, you win. Everything
+ after that is gravy. And WvTest has only a minimal amount of
+ gravy.
+
+ - Be a protocol, not an API. If you don't like my API, you can
+ write your own, and it can still be WvTest and it can still
+ integrate with other WvTest tools. If you're stuck with
+ JUnit or NUnit, you can just make your JUnit/NUnit test
+ produce WvTest-compatible output if you want (although I've
+ never done this, so you'll have to do it yourself). I'll
+ describe the protocol below.
+
+ - Work with multiple languages on multiple operating systems.
+ I'm a programmer who programs on Linux, MacOS, and Windows,
+ to name just three, and I write in lots of programming
+ languages, including C, C++, C#, Python, Perl, and others.
+ And worse, some of my projects use *multiple* languages and I
+ want to have unit tests for *all* of them. I don't know of
+ any unit testing framework - except maybe some horrendously
+ overdesigned ones - that work with multiple languages at
+ once. WvTest does.
+
+ - NO UNNECESSARY OBJECT ORIENTATION. The big unit testing
+ craze seems to have been started by JUnit in Java, which is
+ object-oriented. Now, that's not a misdesign in JUnit; it's
+ a misdesign in Java. You see, you can't *not* encapsulate
+ absolutely everything in Java in a class, so it's perfectly
+ normal for JUnit to require you to encapsulate everything in
+ a class. That's not true of almost any other language
+ (except C#), and yet *every* clone of JUnit in *every*
+ language seems to have copied its classes and objects. Well,
+ that's stupid. WvTest is designed around the simple idea of
+ test *functions*. WvTest runs your function, it checks a
+ bunch of stuff and it returns or else it dies horribly. If
+ your function wants to instantiate some objects while it does
+ that, then that's great; WvTest doesn't care. And yes, you
+ can assert whether two variables are equal even if your
+ function *isn't* in a particular class, just as God intended.
+
+ - Don't make me name or describe my individual tests. How many
+ times have you seen this?
+
+ assertTrue(thing.works(), "thing didn't work!");
+
+ The reasoning there is that if the test fails, we want to be
+ able to print a user-friendly error message that describes
+ why. Right? NO!! That is *awful*. That just *doubled* the
+ amount of work you have to do in order to write a test.
+ Instead, WvTest auto-generates output including the line
+ number of the test and the code on that line. So you get a
+ message like this:
+
+ ! mytest.t.cc:431 thing.works() FAILED
+
+ and all you have to write is this:
+
+ WVPASS(thing.works());
+
+ (WVPASS is all-caps because it's a macro in C++, but also
+ because you want your tests to stand out. That's what
+ you'll be looking for when it fails, after all. And don't
+ even get me started about the 'True' in assertTrue. Come
+ on, *obviously* you're going to assert that the condition is
+ true!)
+
+ - No setup() and teardown() functions or fixtures. "Ouch!" you
+ say. "I'm going to have so much duplicated code!" No, only
+ if you're an idiot. You know what setup() and teardown() are
+ code names for? Constructor and destructor. Create some
+ objects and give them constructors and destructors, and I
+ think you'll find that, like magic, you've just invented
+ "test fixtures." Nothing any test framework can possibly do
+ will make that any easier. In fact, everything test
+ frameworks *try* to do with test fixtures just makes it
+ harder to write, read, and understand. Forget it.
+
+ - Big long scary test functions. Some test frameworks are
+ insistent about the rule that "every function should test
+ only one thing." Nobody every really explains why. I can't
+ understand this; it just causes uncontrolled
+ hormone-imbalance hypergrowth in your test files, and you
+ have to type more stuff... and run test fixtures over and
+ over.
+
+ My personal theory for why people hate big long test
+ functions: it's because their assertTrue() implementation
+ doesn't say which test failed, so they'd like the *name of
+ the function* to be the name of the failed test. Well,
+ that's a cute workaround to a problem you shouldn't have had
+ in the first place. With WvTest, WVPASS() actually tells you
+ exactly what passed and what failed, so it's perfectly okay -
+ and totally comprehensible - to have a sequence of five
+ things in a row where only thing number five failed.
+
+
+The WvTest Protocol
+-------------------
+
+WvTest is a protocol, not really an API. As it happens, the
+WvTest project includes several (currently three)
+implementations of APIs that produce data in the WvTest format,
+but it's super easy to add your own.
+
+The format is really simple too. It looks like this:
+
+ Testing "my test function" in mytest.t.cc:
+ ! mytest.t.cc:432 thing.works() ok
+ This is just some crap that I printed while counting to 3.
+ ! mytest.t.cc.433 3 < 4 FAILED
+
+There are only four kinds of lines in WvTest, and each of the
+lines above corresponds to one of them:
+
+ - Test function header. A line that starts with the word
+ Testing (no leading whitespace) and then has a test function
+ name in double quotes, then "in", then the filename, and then
+ colon, marks the beginning of a test function.
+
+ - A passing assertion. Any line that starts with ! and ends with
+ " ok" (whitespace, the word "ok", and a newline) indicates
+ one assertion that passed. The first "word" on that line is
+ the "name" of that assertion (which can be anything, as long
+ as it doesn't contain any whitespace). Everything between the
+ name and the ok is just some additional user-readable detail
+ about the test that passed.
+
+ - Random filler. If it doesn't start with an ! and it doesn't
+ look like a header, then it's completely ignored by anything
+ using WvTest. Your program can print all the debug output it
+ wants, and WvTest won't care, except that you can retrieve it
+ later in case you're wondering why a test failed. Naturally,
+ random filler *before* an assertion is considered to be
+ associated with that assertion; the assertion itself is the
+ last part of a test.
+
+ - A failing assertion. This is just like an 'ok' line, except
+ the last word is something other than 'ok'. Generally we use
+ FAILED here, but sometimes it's EXCEPTION, and it could be
+ something else instead, if you invent a new and improved way
+ to fail.
+
+
+Reading the WvTest Protocol: wvtestrun
+--------------------------------------
+
+WvTest provides a simple perl script called wvtestrun, which
+runs a test program and parses its output. It works like this:
+
+ cd python
+ ../wvtestrun python ./wvtestmain.py t/twvtest.py
+
+(Why can't we just pipe the output to wvtestrun, instead of
+ having wvtestrun run the test program? Three reasons: first, a
+ fancier version of wvtestrun could re-run the tests several
+ times or give a GUI that lets you re-run the test when you push
+ a button. Second, it handles stdout and stderr separately.
+ And third, it can kill the test program if it gets stuck
+ without producing test output for too long.)
+
+If we put the sample output from the previous section through
+wvtestrun (and changed the FAILED to ok), it would produce this:
+
+ $ ./wvtestrun cat sample-ok
+
+ Testing "all" in cat sample-ok:
+ ! mytest.t.cc my ok test function: .. 0.010s ok
+
+ WvTest: 2 tests, 0 failures, total time 0.010s.
+
+ WvTest result code: 0
+
+What happened here? Well, wvtestrun took each test header (in
+this case, there's just one, which said we're testing "my test
+function" in mytest.t.cc) and turns it into a single test line.
+Then it prints a dot for each assertion in that test function,
+tells you the total time to run that function, and prints 'ok'
+if the entire test function failed.
+
+Note that the output of wvtestrun is *also* valid WvTest output.
+That means you can use wvtestrun in your 'make test' target in a
+subdirectory, and still use wvtestrun as the 'make test' runner
+in the parent directory as well. As long as your top-level
+'make test' runs in wvtestrun, all the WvTest output will be
+conveniently summarized into a *single* test output.
+
+Now, what if the test had failed? Then it would look like this:
+
+ $ ./wvtestrun cat sample-error
+
+ Testing "all" in cat sample-error:
+ ! mytest.t.cc my error test function: .
+ ! mytest.t.cc:432 thing.works() ok
+ This is just some crap that I printed while counting to 3.
+ ! mytest.t.cc.433 3 < 4 FAILED
+ 0.000s ok
+
+ WvTest: 2 tests, 1 failure, total time 0.000s.
+
+ WvTest result code: 0
+
+What happened there? Well, because there were failed tests,
+wvtestrun decided you'd probably want to see the detailed output
+for that test function, so it expanded it out for you. The line
+with the dots is still there, but since it doesn't have an 'ok',
+it's considered a failure too, just in case.
+
+Watch what happens if we run a test with both the passing, and
+then the failing, test functions:
+
+ $ ./wvtestrun cat sample-ok sample-error
+
+ Testing "all" in cat sample-ok sample-error:
+ ! mytest.t.cc my ok test function: .. 0.000s ok
+ ! mytest.t.cc my error test function: .
+ ! mytest.t.cc:432 thing.works() ok
+ This is just some crap that I printed while counting to 3.
+ ! mytest.t.cc.433 3 < 4 FAILED
+ 0.000s ok
+
+ WvTest: 4 tests, 1 failure, total time 0.000s.
+
+ WvTest result code: 0
+
+Notice how the messages from sample-ok are condensed; only the
+details from sample-error are expanded out, because only that
+output is interesting.
+
+
+How do I actually write WvTest tests?
+-------------------------------------
+
+Sample code is provided for these languages:
+
+ C++: try typing "cd cpp; make test"
+ C# (mono): try typing "cd dotnet; make test"
+ Python: try typing "cd python; make test"
+
+There's no point explaining the syntax here, because it's really
+simple. Just look inside the cpp, dotnet, and python
+directories to learn how the tests are written.
+
+
+How should I embed WvTest into my own program?
+----------------------------------------------
+
+The easiest way is to just copy the WvTest source files for your
+favourite language into your project. The WvTest protocol is
+unlikely to ever change - at least not in a
+backwards-incompatible way - so it's no big deal if you end up
+using an "old" version of WvTest in your program. It should
+still work with updated versions of wvtestrun (or wvtestrun-like
+programs).
+
+Another way is to put the WvTest project in a subdirectory of
+your project, for example, using 'svn:externals',
+'git submodule', or 'git subtree'.
+
+
+How do I run just certain tests?
+--------------------------------
+
+Unfortunately, the command-line syntax for running just *some*
+of your tests varies depending which WvTest language you're
+using. For C++ or C#, you link an executable with wvtestmain.cc
+or wvtestmain.cs, respectively, and then you can provide strings
+on the command line. Test functions will run only if they have
+names that start with one of the provided strings:
+
+ cd cpp/t
+ ../../wvtestrun ./wvtest myfunc otherfunc
+
+With python, since there's no linker, you have to just tell it
+which files to run:
+
+ cd python
+ ../wvtestrun ./wvtestmain.py ...filenames...
+
+
+What else can parse WvTest output?
+----------------------------------
+
+It's easy to parse WvTest output however you like; for example,
+you could write a GUI program that does it. We had a tcl/tk
+program that did it once, but we threw it away since the
+command-line wvtestrun is better anyway.
+
+One other program that can parse WvTest output is gitbuilder
+(http://github.com/apenwarr/gitbuilder/), an autobuilder tool
+for git. It reports a build failure automatically if there are
+any WvTest-style failed tests in the build output.
+
+
+Other Assorted Questions
+------------------------
+
+
+What does the "Wv" stand for?
+
+ Either "Worldvisions" or "Weaver", both of which were part of the
+ name of the Nitix operating system before it was called Nitix, and
+ *long* before it was later purchased by IBM and renamed to Lotus
+ Foundations.
+
+ It does *not* stand for World Vision (sigh) or West Virginia.
+
+Who owns the copyright?
+
+ While I (Avery) wrote most of the WvTest framework in C++, C#, and
+ Python, and I also wrote wvtestrunner, the actual code I wrote is
+ owned by whichever company I wrote it for at the time. For the most
+ part, this means:
+
+ C++: Net Integration Technologies, Inc. (now part of IBM)
+ C#: Versabanq Innovations Inc.
+ Python: EQL Data Inc.
+
+What can I do with it?
+
+ WvTest is distributed under the terms of the GNU LGPLv2. See the
+ file LICENSE for more information.
+
+ Basically this means you can use it for whatever you want, but if
+ you change it, you probably need to give your changes back to the
+ world. If you *use* it in your program (which is presumably a test
+ program) you do *not* have to give out your program, only
+ WvTest itself. But read the LICENSE in detail to be sure.
+
+Where did you get the awesome idea to use a protocol instead of an API?
+
+ The perl source code (not to be confused with perlunit)
+ did a similar trick for the perl interpreter's unit
+ test, although in a less general way. Naturally, you
+ shouldn't blame them for how I mangled their ideas, but
+ I never would have thought of it if it weren't for them.
+
+Who should I complain to about WvTest?
+
+ Email me at: Avery Pennarun <apenwarr@gmail.com>
+
+ I will be happy to read your complaints, because I actually really
+ like it when people use my programs, especially if they hate them.
+ It fills the loneliness somehow and prevents me from writing bad
+ poetry like this:
+
+ Testing makes me gouge out my eyes
+ But with WvTest, it takes fewer tries.
+ WvTest is great, wvtest is fun!
+ Don't forget to call wvtestrun.
projects / wvtest.git / commitdiff