Jennifer Bedell wrote recently at PMStudent about testers goldplating projects. Definitely an interesting read. And pretty hot discussion under the post too.
I’d say the post is written in defense of scope. Jennifer points that testers should test against the specification only because when they step out of the specs it becomes goldplating. In other words quality assurance should verify only whether a product is consistent with requirements.
That’s utterly wrong.
It might work if we worked with complete and definite specifications. Unfortunately we don’t. And I mean we don’t ever work one of these. They just don’t exist. What we work with is incomplete vague and ambiguous.
And yes, you can deliver software which sticks perfectly to specs and at the same time is unusable piece of crap.
You should actually encourage testers to get out of the box (specifications) and wander free through the application using every weirdest scenario they can think of. Why? Because your beloved users won’t follow specs. They won’t read it. They won’t even care whether there is something called with the name. This is artificial artifact created to make discussion between product manager and developers easier. Users couldn’t care less about specs.
This means they don’t use application in a way which was specified in requirements, user stories or whatever you happen to create. They use the app using all kinds of approaches, even the weirdest ones. You don’t want to face it unprepared. So forget about damn specs when testing and try everything you can think of (and some of that you can’t).
I’ll go even further: forget about test cases too if you happen to have them. OK, forget about them during the first iteration of tests, because you have more than one, don’t you? The reason is exactly the same as above.
And yes, I can imagine some extreme bugs which will be submitted because of this approach. Bugs which would take years to fix and they are so unlikely to appear in real life it would be plain stupid to fix them. This only means you should monitor incoming stream of bugs, hand-pick these few you aren’t going to fix and throw them out.
This isn’t a reason to consciously decrease quality by sticking to specs during testing.
Subscribe RSS feed
Follow on Twitter
Subscribe by email
4 comments… add one
You make some interesting points and I do agree that testers should not feel constrained by the requirements. But, it is also impossible to test everything. Therefore, testing should be focused on the area of the system that is being changed, along with an appropriate level of regression testing of the system as a whole. Otherwise, the testing cycle could expand infinitely.
Yes, the users will not be sticking to specs. However, the users often are aware of existing defects and are willing to work around them because they have placed a higher priority on the changes being implemented with the current project. Otherwise, the project would not exist. While the defects should be logged (and are probably already logged somewhere), a project resource should be appointed to review all new defects in order to determine which ones will be fixed within the project, and which ones can wait.
So, yes, you are correct. This article was written in defense of scope. Shouldn’t we all be aiming to hit our targets?
It is always a difficult decision: when you should stop testing and deploy application. I don’t really look for a single definite answer. It is impossible to test everything and it is reasonable to push quality improvement only to some point.
This doesn’t change my point of view though. I was a tester myself and my greatest successes at this role were all triggered when I went out of specs. The very same thing I see now. The most painful bugs are find not when our QA folks follow the book, but when they check if something else isn’t screwed by accident. Something which hasn’t been touched in any way for months already.
Yes, it doesn’t come for free but in terms of quality improvement these all were worthy investments. All of issues I think of would hit us very hard once they reached our customers. A couple of them did actually.
Another thing is (lack of) relation between users and specs. In general users aren’t involved in creation of specs and it’s pretty common that most important features from users’ perspective aren’t included or properly described. This may result in users hating your product, even if they’re forced to use it. If you’re out of luck and these are users who are decision-makers they’ll just go away.
After all I’d prefer to have good product late than crappy product on time (at least most of the time). I guess it depends how we define our targets.
I think you hit the nail on the head with the comment about users and specs. Users need to be involved throughout the entire process. And testers should be there right along with them. Quality Assurance begins as soon as the requirements are defined. Testers should begin by testing the requirements. That is, reviewing the Requirements Document.
On that note, let’s not forget the value of the Business Analyst. BA’s are there to help the users define what they really need. Not to simply document what the user thinks they want. With good Requirements Analysis, we can alleviate many of the other problems that pop up during the testing phase.
I fully agree that role of testers and QA in general should start at the beginning of the project. Unfortunately it rarely does. That’s sad because pretty often these are testers who knows best what end-users would expect.
Yes, good BA should be able to put herself in user’s shoes but again, it is common there’s no BA at all, let alone a good one.
Involvement of users is a tricky part here. It depends on a type of project. If I build app addressed directly for end-users I’d like to see them involved as soon as possible. If I build system for some corporation I need to satisfy decision-makers in the first place – otherwise I won’t get a contract at all. In the latter case users involvement may (but don’t have to) end in goldplating we’d like to avoid.