Shortcomings of Test Cases

I’ve already told you that writing test cases is worth the effort. What I haven’t stressed are shortcomings of test cases.

If you take time to prepare test cases you most likely do it for the reason and I guess not for “my manager will fire me if I don’t” one. You want to invite some organization to your testing and improve overall quality of the product at the end of the day. I don’t want to go deeper for motivations since it isn’t the point of this article.

The thing you do is preparing some scenarios up front, definitely before you actually start testing. Generally the earlier the better but either way test cases are before test execution, that’s for sure. Basically when you prepare test cases you imagine how you’d like the application to be tested and you write it down. Do this, verify that, expect something other and proceed to the next step.

Now what’s wrong with that?

1. You suck at predicting how your users would use the products. We all do. Applications are built by developers who don’t even try to imagine how users will interact with software. Then they’re tested by quality engineers who are also far away from average users. Then they’re deployed by people who actually talk with users but can’t influence application development in almost any way. The whole process is driven by product managers whose job is to think about general concepts, not detailed usage flows. Who cares about usage scenarios then? Well, pretty much no one.

2. Test cases are closed scenarios by definition. When designing test cases you don’t have “flow with your gut feeling” block at hand to put it in the diagram. You actually have to make it possible to verify test results thus you’re limited to some specific path – one among hundreds or thousands. This means you won’t go through the vast majority of cases yet you will decide whether a new feature works well or not.

3. It’s not possible to have a complete set of test cases. A typical method which takes only an integer as an argument can be called in possibly 2^32 different ways which is basically infinity. If you add that you may need to call another function as a prerequisite you end up with a number which blows my head up. And we’re talking only about a single function, not about whole application. Test case is always a dramatic simplification in terms of possible usage scenarios.

4. It’s easy to complete test cases and call it a day. Hey, after all test cases are for quality engineers, aren’t they? QA people should base their activities on test cases. Except they shouldn’t limit their work just to test cases. And you want it or not there’s always a bit of incentive to turn off your thinking and just follow the plan instead of going an extra mile. I’ve already said that a couple of times, but QA team is no place for follow-the-book people. Creativity is a key trait for QA engineers, yet we don’t live in an ideal world and you don’t have only the best and the most creative people in QA team, no matter what your company website says.

With all these shortcomings I still strongly advise you to write test cases. You may try to do them as precise as possible, or you may just use general flows leaving some autonomy for testers but either way it will pay off. Especially if you understand where test cases fall short and find a way to deal with that using other tools.