The other day we were discussing different techniques aimed at improving code quality. Continuous integration, static code analysis, unit testing with or without applying test-driven development, code review – we’ve went through them all. At some point I sensed someone could feel that once they employ all this fine practices their code will be ready to ship as soon as it is all green after build on a build server. I just had to counter strike:
“I’m not going to ship any code, no matter how many quality-improving techniques we use, unless some human tester puts their hands on it and blesses it as good. Not even a chance.”
I’ve seen the situation quite a number of times: a team of developers who are all into the best engineering practices, but unfortunately at the same time sharing a belief this is enough to build top-notch applications. I have a bad news for you: if the answer for a question about quality assurance team is “we don’t need one” I scream and run away. And don’t buy any of your apps.
It just isn’t possible to build quality code with no manual testing at all. You may build something and throw it against your client hoping they will do the testing for you. Sometimes you can even get away with this approach. Be warned though – you’re going to fail hard way soon. The next customer may not like the idea of doing beta-tests for you, actually most of them won’t, and then you’re screwed.
Manual and automatic testing is fundamentally different. With automated testing, no matter if we’re talking about unit, stress, regression, integration or end-to-end tests, you go through predefined tracks. You had to create the test in the first place so it does exactly what you told it to do.
On the other hand humans pretty often had their own minds which they happen to use in unpredictable ways. They may for example make up some test scenarios on the fly or they may sense a subtle scent of a big scary bug hiding around and follow their instinct to hunt it down. Every now and then they will abandon beaten tracks to do something unexpected, i.e. find a bug which would be missed otherwise.
Note: I don’t say what kind of procedure you should follow in manual testing. Personally I strongly believe in value of exploratory testing, but any good tester will do the job, no matter what your manual testing procedure look like.
Actually this post was triggered by recent discussion between a few bloggers including Lisa Crispin, James Shore, Ron Jeffries, George Dinwiddie and Gojko Adzic. The discussion, while interesting and definitely worth to read, is a bit too academic for me. My approach is that there is no universal set of techniques which yields indisputably best results. While I like James arguments why they dropped acceptance testing and support his affection for exploratory testing I share Lisa point that even in stable environment our “best” approach will evolve and so would techniques we employ.
You may exchange acceptance testing with exploratory testing and that’s perfectly fine. As long as some fellow human put their hand on the code you want your users to touch with mine that I’m good with that. I don’t care much which approach you choose or how you call it. It is way more important who does the job. Experienced quality engineer with no plan at all will do way better job than poor or inexperienced tester with the best plan.
Just remember if you happen to skip the manual testing at all I’m afraid your app may be good only for machines as only machines tested it.
Subscribe RSS feed
Follow on Twitter
Subscribe by email
8 comments… add one
Much as I rely on unit testing, it’s that random element that QA teams provide that makes the difference. Done right it’s a real positive; but then again, some are so regimented, manual testing by a script written in front of them, that it takes away that advantage.
To me it’s no contradiction to do ATDD according to conditions of satisfaction which you have committed automate in CI during iteration, while *on top* (or in exploratory cross-feature test iteration) you do manual expert tests which you automate asa you found a bug to add to regression of that case. btw – conditions of satisfaction for userstories are not only defined by newbees (and if then you learn…)
Phil,
Maybe I haven’t stressed it enough but when I think manual testing I think very good manual testing. If I have testers who are going to work like mindless golems and just blindly follow test scenarios I guess I can exchange them with one developer who would write some automatic tests and I will be a good deal.
However when you think about one of these die-hard tester who can sense a bug in a distance of hundred meters I wouldn’t exchange them pretty much for anything.
Thoralf,
Of course there is no conflict between automatic and manual testing. Of course it is better to have them both and each in at least a few flavors. However we’re often tempted to switch purely to these cool techniques which makes the code self-testing. Sometimes you will even find people advising you to do so.
And this is just stupid. While manual testing is expensive it is also indispensable. At least this is what I believe in.
True, true, manual testing is always a must, 3 times yes! Automated testing is great, can reduce manual testing 10 times, but in the end someone has to click on it.
In fact if you want to be agile and be able to release every 1-2 weeks to the production, you need all of this unit testing (tdd strongly recommended), some automated acceptance testing (including all target platforms), code review and still some manual testing for each feature and moreover some manual testing for regression.
And this is exacly what good agile teams does. Many scrum teams are able to release every sprint banking software or life critical software to medical hardware all over the planet.
Thanks to great automated tests on many levels. But in the end there is always a guy clicking on the machine, checking if things still make sens.
Yes, the more you move onto automatic testing the less effort your testers spend on assuring you keep quality in simple things. For me automatic testing is a king when in comes to regression. If every bug fix consist not only a fix but also a unit test (or a couple of them) which verify problematic situation we can easily to forget the thing during manual testing.
This allows us more to go free ride during manual tests which is more fun for testers and yields much more interesting results, i.e. revealing basically all hard-core bugs we find.
I’m curious about the statement, “Experienced quality engineer with no plan at all will do way better job than poor or inexperienced tester with the best plan. ”
I’m not sure how an good, experienced tester could have “no plan” even if unable to articulate that plan to you.
And I’m not sure how a tester who can come up with “the best plan” can be considered poor.
George,
I should have added “written” or “specified” before “the plan.” Yes, good tester will have some plan in their head but I don’t think they’ll start with writing it down and follow it as a checklist. And from a perspective of a project as a whole plan which can’t be articulated in any way isn’t really a plan. No one can do anything with such thing.
On the other hand there can be detailed plan, i.e. extensive list of test cases, which is given to the testers and they’re expected to check it all, case by case. They don’t come up with it.