While piecing together the parts of my service I realized one topic I haven’t touched is Unit testing.I’m going to use Continuous Deployment as my development/deployment strategy, which makes unit testing a must, but to be honest, nowadays Unit testing is a standard development practice which everyone should adopt.
So what’s available as open source today?
As of today (Oct. 2012), there are two main open source Unit Testing frameworks available for Java development: JUnit and TestNG.
On top of those, there are several toolkits which extend the basic functionality, the most talked about from what I’ve read are EasyMock and Mockito, extended by PowerMock for more advanced functionality (like mocking private and final methods), and JMockit as an alternative to all three (since it gives both the basic as well as advanced functionality).
JUnit has been around for some time now, and is considered to have had the most contribution to unit testing (at least in java). It is mature and feature rich, and seems to have all you need to get started. It is also the most supported framework on the market, and most used, which results in built in support by most relevant tools.
TestNG seems to have many enthusiastic followers, but has not been around as long as JUnit, and seems to have (much) less market penetration. Interestingly though, while JUnit has been losing market share, it was not lost to TestNG, which has been pretty stable in its usage statistics according to a graph I’ve seen.
In 99% of the comparisons I’ve read online between the two frameworks, TestNG was the recommended framework, mainly for its advanced features of groups, parameterized tests, and dependency between tests. While these seem important to the unit testing novice reader (me), some argue that the reason some of these features do not appear in JUnit is because they defy the purpose of keeping a test independent of other tests. Since most comparisons I’ve read were at least 2 years old, it might be that feature differences have changed in later versions, as it seems that JUnit is constantly catching up to what is done in TestNG (meaning that TestNG is leading the way?). Another point which constantly comes up in favor of TestNG is that it is more suited for wide, integration tests, while JUnit is limited to the Unit Testing world. This comes up in the context of integration tools like Selenium.
Complementing toolkits insights:
EasyMock is usually compared to Mockito, which by the way was forked from EasyMock but has been developed to a point in which (according to its developers) it has no common code. It’s usually stated that EasyMock is more strict and breaks easier on code change, while Mockito is more lenient thus letting some code changes slide before raising a red flag. In EasyMock you have to specify exactly what you expect to happen when the code runs, while Mockito lets you concentrate on the results. This is referred to as an advantage by some, and disadvantage by others, since it’s a constant battle between test completeness and coding agility. Both need to be complemented by PowerMock to be able to do advanced stuff like mocking private or final methods, with no clear winner as far as functionality or design is concerned from what I’ve read. On the other side is JMockit, which seems to have the same type of enthusiastic followers as TestNG (though not necessarily the same ones). It is frequently referred to as a power tool, with a nice set of features and highly competitive of the EasyMock/Mockito & PowerMock pairs.
As with TestNG, JMockit seems to get most of the votes when it comes to feature comparison and decision of which toolkit should be chosen by newcomers, but is not the most widely used toolkit (not sure which one of the two has the lead here).
All frameworks/tools discussed in this post are actively developed (as of Oct. 2012). While it sounds like the newcomers (TestNG and JMockit) have the advantage over the older and more widely used projects, I’m inclined to use the more established and popular tools, as I feel that as a new user I would have less issues and better integration support.
Having said that, I have a feeling that once I collect enough milage with unit testing as a concept, it will be a good time to revisit this decision, as the alternatives seem to offer good value over the older tools (according to those that know both), and worth the investment of learning a new tool.
To be continued…
Martin Fowler is always fun to read, in this post he talks about JMockit as a power tool which lets you do great stuff, at the expense of fixing up your smelly code. Worth reading…
This post describes some guy’s experience with both toolkits (Mockito and EasyMock).
This document shows a comparison between writing tests in jMock vs. EasyMock. While it doesn’t show Mockito, and it is rather old (2008), you can still get the hang of what writing tests is all about. There are other posts with such comparisons, I’d suggest finding a few to understand what each toolkit looks like when implementing tests.
Here is an interesting discussion on stackoverflow.