+972 52-548-6969
The tests has no value by themselves: My most successful project didn't have any tests
In DevTeach, we had a panel that Kathleen Dollard has covered in depth, in which we talked about what the bare minimum aspects of Agile project would be. The first thing that was thrown up was testing.
That is quite predictable, and I objected to that. One of the things that bother me about much of the discussions in the agile space is the hard focus of tests. Often to the exclusion of much else.
My most successful (commercial) project was done without tests, and it is a huge success (ongoing now, by the way). A previous project had tests, quite a few of them, and I consider it a big failure (over time, over budget, overly complex code base, a lot of the logic in the UI, etc).
Update: I wanted to make clear the distinction between Agile and TDD. I consider the project without tests to be a fully agile project. The project with tests was a heavy waterfall project.
I think that I can safely say that it would be hard to accuse me of not getting testing, or not getting TDD. Not that I don't expect the accusations anyway, but I am going to try to preempt them.
I want to make it explicit, and understood. What I am riling against isn't testing. I think that they are very valuable, but I think that some people are focusing on that too much. For myself, I have a single metric for creating successful software:
Ship it, often.
There are many reasons for that, from the political ones and monetary ones to feedback, scope and tracer bullets.
Tests are a great tool to aid you in shipping often, but they aren't the only one. Composite architectures and JFHCI are two other ways that allow us to create stable software platform that we can develop on.
Tests are a tool, and its usage should be evaluated against the usual metrics before applying it in a project. There are many reasons not to use tests, but most of them boil down to: "They add friction to the process".
Testing UI, for example, is a common place where it is just not worth the time and effort. Another scenario is a team that is not familiar with testing, introducing testing at this point would hinder my topmost priority, shipping.
Code quality, flexibility and the ability to change are other things that are often attributed to tests. They certainly help, but they are by no mean the only (or even the best) way to approach that.
And finally, just to give an example, Rhino Igloo was developed without tests, using F5 testing style. I applied the same metric, trying to test what it does would have been painful, therefor, I made the decision not to write tests for that. I don't really like that code base, but that is because it is strongly tied to the Web Forms platform, not because it is hard to change or extend, it isn't.
Okay, I am done, feel free to flame.
Comments
I am a huge proponent of TDD and writing tests. I talk about it all the time. But at the same time, I still think you need to evaluate whether it is worth it to write tests in certain cases.
If I have some code that is really simple, is not going to change much, and does not have a lot of other code dependent on it, I don't know that it's worth taking the time to write tests. Like you said, a lot of UI code falls into this category (that is, the UI code that you are actually able to test).
Like anything else, each situation needs to be reviewed individually. There is no magic formula in software development that is the best choice in every situation.
@Jon,
I think you hit it right on the head when you said "There is no magic formula in software development...".
I think in many ways you are right about using testing when needed.
But I think that we need to remember that you are an expert software developer, with much experience.
This aspect is good to remember, since I hav earlier met experienced software developers who could skip testing and make it OK.
As i see it two import aspects of testing are "built in" to expert software developers:
Designing APIs, when you have developed many APis before - you know how to do it.
Affirmation in developing.
@Klaus,
I disagree ... TDD development doesn't magically make "inexperienced" developers "experienced".
If you are worried about needing tests because your developers are "inexperienced" ... shouldn't you be worried that the tests those developers would have to write could be wrong? Tests are simply code and as with any code they can have defects too.
My thoughts on the topic at hand is; If you want tests, write tests, if you don't, don't.
I am a fan of releasing often. Theer is in average, more than one version of NDepend per month: http://www.ndepend.com/ReleaseNotes.aspx
One secrets is code reviewing of changes since last version: If you release often changes are epsilon and easy to review. Moreover you can count on the fact that code not touced is working in production, else users would complain.I wrote about this:
How to avoid regression bugs while adding new features
searchwindevelopment.techtarget.com/.../...00.html
Another secret: make sure that changes are well covered by automatic tests. I wrote about this too:
Are you sure added and refactored code is covered by tests?
codebetter.com/.../...ode-is-covered-by-tests.aspx
These features were developed primarily with the dog-fooding use in-mind.
I actually think you are confusing two things here - agile and "succesful" is not necessarily the same thing. There are lots of succesful software projects with no automated [unit] testing (I guess that's what we're talking about?) what so ever and I guess a lot of people now doing and evengelizing TDD have managed to deliver good software before they discovered TDD as well.
If you do agile software development in a non-trivial domain, with more than a few developers and over a not so short time period, I think that some level of automated testing will be the most cost efficient way to go as it is either that or a lot of manual downstream regression testing that ensures that features checked off in earlier iterations doesn't break.
Joakim,
I wasn't clear. That WAS an agile project.
Oh,
And it was big, extremely non trivial, had a reasonably sized team and went on for a long time
Well, sometimes you're just lucky I guess... :-)
Of course you have a point, but what I also was trying to point out was that it still does not say much about what "agile" is or is not. You will always be able to find exceptions to the rule, but what are the main criterias for deciding if a project is agile or not, "the bare minimum aspects of Agile project" to quote your introduction?
Wether you choose to interview prominent "agile gurus" or survey (succesful) agile projects (like Scott Ambler does, who coincidentally mentions testing as one of perhaps four core tenets of agile) I think you will find that testing is a core practice.
Agile schmagile - why does it matter? I think it matters because most agile teams will probably find it difficult to succeed without focusing on testing. Agreed? :-)
I am using TDD for the last six months and I could see that my softwares are getting much better and lots of problems I am getting on the tests.
Another interesting thing I can see is that when a new developer is allocated in the project, he can understand it better when looking to the tests.
About time, I am not sure if TDD spends more time. All the time you spent creating the tests is saved during the maintenance phase.
I agree with you about your metric. Shipping the software very often is the best way to detect defects.
As people above me said, "there's no magic formula", and TDD WILL NOT save all your projects!
Interesting you say this - all good.
I heard a comment recently 'the whole point of doing asp.net mvc is to unit test - where are the tests...'
To me, that isn't the 'whole point' - mvc is an architecture. Sorta in the same light as people saying 'I use IoC to help in testing'
I use mvc, ioc, etc.. because I consider them to be good proven architectual approaches to building my code, not just because it's easier to test.
I do like tests, but not in the approach of 'I must have 99% coverage' - but more as a tool to help in design and prototyping. Especially with domain/service logic where I want to run it outside of the UI. - Firing up the web page, clicking 3 times to execute Service 'xyz' ends up taking alot of time. Having a test the executes the service is much easier to test and debug. In that regards, it quickly throws out the 'it takes too long to build tests', in those situations I end up saving alot of time and the side benefit is others can see what the service does by looking at the test. So those are the benefits of testing I think. Not a 'every single function in your app needs a test' mentality. Sure that is good if you have resources, I think - but not always possible
Good to see your post on this with that attitude, to me - agile is about the customer and iterative development where they are making decisions, seeing the progress - with many releases. An attitude of 'this is their software, not mine'. This is much better than a waterfall, 'we'll show you when it's done in 6 months, hope it is what you wanted' . I've seen that - it never works.
Currently I'm a project that meets regularly with the customer, many reviews/feedback and we have built and deliver every week what they want.
Seems to be working to their benefit - so ... it's about delivering what they want, and at the same time, they learn that it takes time to develop good quality stuff, and they appreciate it and are able to help make good choices along the way.
Last place I worked, wasted 6 months without writing any code just talking about what the customer wanted - it was a mess. Over analyzed, over thought, never delivered
Joakim,
Bare necessities of agile?
Ship, often, and get customer feedback. That is it.
"And it was big, extremely non trivial, had a reasonably sized team and went on for a long time"
What is "long time" for you ? Can you give us an open-source example ? The example you give, Rhino Igloo, as far as I can see by checking it out, is a very small project that doesn't fall in this category.
In my experience, what prevent a software from being shipped often (with few bugs and without increasing dev costs) is the combinatorial explosion of sub-components features. When new developpers enter the team or when tired developpers change such sub-components in order to implement new features, bugs occur. Some of these bugs may be detected early by unit tests on sub-components.
Long time here is 2+ years.
And no, off the top of my head I can't think of an OSS example.
Your post is going to be interpreted as "All you have to do to succeed in being agile is to get customer feedback and ship often", or in other words, "willing to be agile is sufficient to be agile".
Unfortunately this is simply wrong because you won't be able to ship periodically at sustainable costs and quality without relevant engineering practices like unit tests, refactoring and continuous integration. I tried the two ways, the one with relevant tools is clearly superior in terms of customer satisfaction. But it's far from being the more common one.
And this is rather disapointing because you are read by a lot of developers...
Are you implying that the customer is solely responsible for verification that the software product works?
I completely agree with your post. The FACT is that lots of projects are successful without test. A lot of people prefer to test because we feel it makes our software better. Not all people would agree.
I’m quite certain I’ve read this somewhere… “Individuals and interactions over processes and tools”
If you want to cram testing down people’s throats, you will get shit test. That’s just the way it is.
Denis,
"you won't be able to ship periodically at sustainable costs and quality without relevant engineering practices like unit tests, refactoring and continuous integration"
Yes, I would. I am not talking in the air, I have the practical experience and two years of a running project to back me up on that.
Ah OK ! I just read your post here: ayende.com/.../...ose-principle-as-applied-to.aspx
Growing a project like a tree, by adding layers without changing the core... Zero tests because of zero refactoring because of uncompromising Open-Close Principle... That's an interesting kind of extreme programing.
And by the way, I'm rather happy to be the one that triggered the unveiling of the solution of "the-less-you-write-tests-the-more-successful-your-projects-are" enigma. Billions have beed saved to the whole industry :)
I'm glad, and pleasantly surprised, to hear you say this.
I think you created some waves and thought process in some who, may be, a bit to 'rigid' in their approach and hopefully this can help many see the 'bell curve' in 'methodologies'. This is the type of 'master of the art' blog post I prefer seeing from 'popular names' out there - such as yourself; where we demonstrate that there is no magic formula, that it is not black and white, and success is not measured by things done or not done, but by final results - and how you obtained those results is irrelevant.
But in order to 'get' this concept, you have to be able to see the forest through the trees, and that I'm afraid too few people are capable (or willing in several cases) of doing that. I can see this by responses implying that you were only successful in that project with tests because of the 'process' you followed where 'change was not allowed' per se.
I myself was very successful on a larger waterfall project (nearly 3 years) that had 0 unit tests but a crap load of refactoring along the way; was it successful because our major chunk of refactoring (which was done by cut and paste btw...not ReSharper or similar tool) was 'collapsing similar child methods to parent'? or was it the fact that we did all our design up front before starting code? Or maybe it was just the people? Or maybe it was combination?
Some on the team knew, and were advocates of TDD, but we didn't go that route and still succeeded. And we've all gone on to have successful projects with a variety of processes, and spectacular failures using the same process as the ones that were success. Some had tests, some did not on both ends of the spectrum, and in the end, the projects that were complete failures - tests and whether we used TDD, XP, Waterfall, or PP made little difference. It was the people who did.
But then myself, and the people I know are 'nobodys' in this industry - so who would listen to us? Thanks for being our voice on this one Ayende!
Ayende,
Thanks for saying what some us were thinking even in the face of popular opinion and potential criticism.
Comment preview