From: Avery Pennarun Date: Fri, 1 May 2009 03:33:31 +0000 (-0400) Subject: Added an initial README. X-Git-Url: http://rtime.felk.cvut.cz/gitweb/wvtest.git/commitdiff_plain/080fdc1be7a36f7a31f07bea65a3ad4936d7dbe8 Added an initial README. --- diff --git a/README b/README new file mode 100644 index 0000000..4f6aab1 --- /dev/null +++ b/README @@ -0,0 +1,373 @@ + +WvTest: the dumbest cross-platform test framework that could possibly work +========================================================================== + +I have a problem with your unit testing framework. Yes, you. +The person reading this. No, don't look away guiltily. You +know your unit testing framework sucks. You know it has a +million features you don't understand. You know you hate it, +and it hates you. Don't you? + +Okay, fine. Let's be honest. Actually, I don't know who you +are or how you feel about your unit testing framework, but I've +tried a lot of them, and I don't like any of them. WvTest is +the first one I don't hate, at least sort of. That might be +because I'm crazy and I only like things I design, or it might +be because I'm crazy and therefore I'm the only one capable of +designing a likable unit testing framework. Who am I to say? + +Here are the fundamental design goals of WvTest: + + - Be the stupidest thing that can possibly work. People are + way, way too serious about their testing frameworks. Some + people build testing frameworks as their *full time job*. + This is ridiculous. A test framework, at its core, only does + one thing: it runs a program that returns true or false. If + it's false, you lose. If it's true, you win. Everything + after that is gravy. And WvTest has only a minimal amount of + gravy. + + - Be a protocol, not an API. If you don't like my API, you can + write your own, and it can still be WvTest and it can still + integrate with other WvTest tools. If you're stuck with + JUnit or NUnit, you can just make your JUnit/NUnit test + produce WvTest-compatible output if you want (although I've + never done this, so you'll have to do it yourself). I'll + describe the protocol below. + + - Work with multiple languages on multiple operating systems. + I'm a programmer who programs on Linux, MacOS, and Windows, + to name just three, and I write in lots of programming + languages, including C, C++, C#, Python, Perl, and others. + And worse, some of my projects use *multiple* languages and I + want to have unit tests for *all* of them. I don't know of + any unit testing framework - except maybe some horrendously + overdesigned ones - that work with multiple languages at + once. WvTest does. + + - NO UNNECESSARY OBJECT ORIENTATION. The big unit testing + craze seems to have been started by JUnit in Java, which is + object-oriented. Now, that's not a misdesign in JUnit; it's + a misdesign in Java. You see, you can't *not* encapsulate + absolutely everything in Java in a class, so it's perfectly + normal for JUnit to require you to encapsulate everything in + a class. That's not true of almost any other language + (except C#), and yet *every* clone of JUnit in *every* + language seems to have copied its classes and objects. Well, + that's stupid. WvTest is designed around the simple idea of + test *functions*. WvTest runs your function, it checks a + bunch of stuff and it returns or else it dies horribly. If + your function wants to instantiate some objects while it does + that, then that's great; WvTest doesn't care. And yes, you + can assert whether two variables are equal even if your + function *isn't* in a particular class, just as God intended. + + - Don't make me name or describe my individual tests. How many + times have you seen this? + + assertTrue(thing.works(), "thing didn't work!"); + + The reasoning there is that if the test fails, we want to be + able to print a user-friendly error message that describes + why. Right? NO!! That is *awful*. That just *doubled* the + amount of work you have to do in order to write a test. + Instead, WvTest auto-generates output including the line + number of the test and the code on that line. So you get a + message like this: + + ! mytest.t.cc:431 thing.works() FAILED + + and all you have to write is this: + + WVPASS(thing.works()); + + (WVPASS is all-caps because it's a macro in C++, but also + because you want your tests to stand out. That's what + you'll be looking for when it fails, after all. And don't + even get me started about the 'True' in assertTrue. Come + on, *obviously* you're going to assert that the condition is + true!) + + - No setup() and teardown() functions or fixtures. "Ouch!" you + say. "I'm going to have so much duplicated code!" No, only + if you're an idiot. You know what setup() and teardown() are + code names for? Constructor and destructor. Create some + objects and give them constructors and destructors, and I + think you'll find that, like magic, you've just invented + "test fixtures." Nothing any test framework can possibly do + will make that any easier. In fact, everything test + frameworks *try* to do with test fixtures just makes it + harder to write, read, and understand. Forget it. + + - Big long scary test functions. Some test frameworks are + insistent about the rule that "every function should test + only one thing." Nobody every really explains why. I can't + understand this; it just causes uncontrolled + hormone-imbalance hypergrowth in your test files, and you + have to type more stuff... and run test fixtures over and + over. + + My personal theory for why people hate big long test + functions: it's because their assertTrue() implementation + doesn't say which test failed, so they'd like the *name of + the function* to be the name of the failed test. Well, + that's a cute workaround to a problem you shouldn't have had + in the first place. With WvTest, WVPASS() actually tells you + exactly what passed and what failed, so it's perfectly okay - + and totally comprehensible - to have a sequence of five + things in a row where only thing number five failed. + + +The WvTest Protocol +------------------- + +WvTest is a protocol, not really an API. As it happens, the +WvTest project includes several (currently three) +implementations of APIs that produce data in the WvTest format, +but it's super easy to add your own. + +The format is really simple too. It looks like this: + + Testing "my test function" in mytest.t.cc: + ! mytest.t.cc:432 thing.works() ok + This is just some crap that I printed while counting to 3. + ! mytest.t.cc.433 3 < 4 FAILED + +There are only four kinds of lines in WvTest, and each of the +lines above corresponds to one of them: + + - Test function header. A line that starts with the word + Testing (no leading whitespace) and then has a test function + name in double quotes, then "in", then the filename, and then + colon, marks the beginning of a test function. + + - A passing assertion. Any line that starts with ! and ends with + " ok" (whitespace, the word "ok", and a newline) indicates + one assertion that passed. The first "word" on that line is + the "name" of that assertion (which can be anything, as long + as it doesn't contain any whitespace). Everything between the + name and the ok is just some additional user-readable detail + about the test that passed. + + - Random filler. If it doesn't start with an ! and it doesn't + look like a header, then it's completely ignored by anything + using WvTest. Your program can print all the debug output it + wants, and WvTest won't care, except that you can retrieve it + later in case you're wondering why a test failed. Naturally, + random filler *before* an assertion is considered to be + associated with that assertion; the assertion itself is the + last part of a test. + + - A failing assertion. This is just like an 'ok' line, except + the last word is something other than 'ok'. Generally we use + FAILED here, but sometimes it's EXCEPTION, and it could be + something else instead, if you invent a new and improved way + to fail. + + +Reading the WvTest Protocol: wvtestrun +-------------------------------------- + +WvTest provides a simple perl script called wvtestrun, which +runs a test program and parses its output. It works like this: + + cd python + ../wvtestrun python ./wvtestmain.py t/twvtest.py + +(Why can't we just pipe the output to wvtestrun, instead of + having wvtestrun run the test program? Three reasons: first, a + fancier version of wvtestrun could re-run the tests several + times or give a GUI that lets you re-run the test when you push + a button. Second, it handles stdout and stderr separately. + And third, it can kill the test program if it gets stuck + without producing test output for too long.) + +If we put the sample output from the previous section through +wvtestrun (and changed the FAILED to ok), it would produce this: + + $ ./wvtestrun cat sample-ok + + Testing "all" in cat sample-ok: + ! mytest.t.cc my ok test function: .. 0.010s ok + + WvTest: 2 tests, 0 failures, total time 0.010s. + + WvTest result code: 0 + +What happened here? Well, wvtestrun took each test header (in +this case, there's just one, which said we're testing "my test +function" in mytest.t.cc) and turns it into a single test line. +Then it prints a dot for each assertion in that test function, +tells you the total time to run that function, and prints 'ok' +if the entire test function failed. + +Note that the output of wvtestrun is *also* valid WvTest output. +That means you can use wvtestrun in your 'make test' target in a +subdirectory, and still use wvtestrun as the 'make test' runner +in the parent directory as well. As long as your top-level +'make test' runs in wvtestrun, all the WvTest output will be +conveniently summarized into a *single* test output. + +Now, what if the test had failed? Then it would look like this: + + $ ./wvtestrun cat sample-error + + Testing "all" in cat sample-error: + ! mytest.t.cc my error test function: . + ! mytest.t.cc:432 thing.works() ok + This is just some crap that I printed while counting to 3. + ! mytest.t.cc.433 3 < 4 FAILED + 0.000s ok + + WvTest: 2 tests, 1 failure, total time 0.000s. + + WvTest result code: 0 + +What happened there? Well, because there were failed tests, +wvtestrun decided you'd probably want to see the detailed output +for that test function, so it expanded it out for you. The line +with the dots is still there, but since it doesn't have an 'ok', +it's considered a failure too, just in case. + +Watch what happens if we run a test with both the passing, and +then the failing, test functions: + + $ ./wvtestrun cat sample-ok sample-error + + Testing "all" in cat sample-ok sample-error: + ! mytest.t.cc my ok test function: .. 0.000s ok + ! mytest.t.cc my error test function: . + ! mytest.t.cc:432 thing.works() ok + This is just some crap that I printed while counting to 3. + ! mytest.t.cc.433 3 < 4 FAILED + 0.000s ok + + WvTest: 4 tests, 1 failure, total time 0.000s. + + WvTest result code: 0 + +Notice how the messages from sample-ok are condensed; only the +details from sample-error are expanded out, because only that +output is interesting. + + +How do I actually write WvTest tests? +------------------------------------- + +Sample code is provided for these languages: + + C++: try typing "cd cpp; make test" + C# (mono): try typing "cd dotnet; make test" + Python: try typing "cd python; make test" + +There's no point explaining the syntax here, because it's really +simple. Just look inside the cpp, dotnet, and python +directories to learn how the tests are written. + + +How should I embed WvTest into my own program? +---------------------------------------------- + +The easiest way is to just copy the WvTest source files for your +favourite language into your project. The WvTest protocol is +unlikely to ever change - at least not in a +backwards-incompatible way - so it's no big deal if you end up +using an "old" version of WvTest in your program. It should +still work with updated versions of wvtestrun (or wvtestrun-like +programs). + +Another way is to put the WvTest project in a subdirectory of +your project, for example, using 'svn:externals', +'git submodule', or 'git subtree'. + + +How do I run just certain tests? +-------------------------------- + +Unfortunately, the command-line syntax for running just *some* +of your tests varies depending which WvTest language you're +using. For C++ or C#, you link an executable with wvtestmain.cc +or wvtestmain.cs, respectively, and then you can provide strings +on the command line. Test functions will run only if they have +names that start with one of the provided strings: + + cd cpp/t + ../../wvtestrun ./wvtest myfunc otherfunc + +With python, since there's no linker, you have to just tell it +which files to run: + + cd python + ../wvtestrun ./wvtestmain.py ...filenames... + + +What else can parse WvTest output? +---------------------------------- + +It's easy to parse WvTest output however you like; for example, +you could write a GUI program that does it. We had a tcl/tk +program that did it once, but we threw it away since the +command-line wvtestrun is better anyway. + +One other program that can parse WvTest output is gitbuilder +(http://github.com/apenwarr/gitbuilder/), an autobuilder tool +for git. It reports a build failure automatically if there are +any WvTest-style failed tests in the build output. + + +Other Assorted Questions +------------------------ + + +What does the "Wv" stand for? + + Either "Worldvisions" or "Weaver", both of which were part of the + name of the Nitix operating system before it was called Nitix, and + *long* before it was later purchased by IBM and renamed to Lotus + Foundations. + + It does *not* stand for World Vision (sigh) or West Virginia. + +Who owns the copyright? + + While I (Avery) wrote most of the WvTest framework in C++, C#, and + Python, and I also wrote wvtestrunner, the actual code I wrote is + owned by whichever company I wrote it for at the time. For the most + part, this means: + + C++: Net Integration Technologies, Inc. (now part of IBM) + C#: Versabanq Innovations Inc. + Python: EQL Data Inc. + +What can I do with it? + + WvTest is distributed under the terms of the GNU LGPLv2. See the + file LICENSE for more information. + + Basically this means you can use it for whatever you want, but if + you change it, you probably need to give your changes back to the + world. If you *use* it in your program (which is presumably a test + program) you do *not* have to give out your program, only + WvTest itself. But read the LICENSE in detail to be sure. + +Where did you get the awesome idea to use a protocol instead of an API? + + The perl source code (not to be confused with perlunit) + did a similar trick for the perl interpreter's unit + test, although in a less general way. Naturally, you + shouldn't blame them for how I mangled their ideas, but + I never would have thought of it if it weren't for them. + +Who should I complain to about WvTest? + + Email me at: Avery Pennarun + + I will be happy to read your complaints, because I actually really + like it when people use my programs, especially if they hate them. + It fills the loneliness somehow and prevents me from writing bad + poetry like this: + + Testing makes me gouge out my eyes + But with WvTest, it takes fewer tries. + WvTest is great, wvtest is fun! + Don't forget to call wvtestrun. diff --git a/sample-error b/sample-error new file mode 100644 index 0000000..3890d6d --- /dev/null +++ b/sample-error @@ -0,0 +1,5 @@ +Testing "my error test function" in mytest.t.cc: +! mytest.t.cc:432 thing.works() ok +This is just some crap that I printed while counting to 3. +! mytest.t.cc.433 3 < 4 FAILED + diff --git a/sample-ok b/sample-ok new file mode 100644 index 0000000..82047d7 --- /dev/null +++ b/sample-ok @@ -0,0 +1,5 @@ +Testing "my ok test function" in mytest.t.cc: +! mytest.t.cc:432 thing.works() ok +This is just some crap that I printed while counting to 3. +! mytest.t.cc.433 3 < 4 ok +