Pawel Brodzinski on Leadership in Technology

Whatever it takes to lead a team, build a product, and run a business

Tag: test cases

  • Why We Don’t Write Test Cases Anymore

    Almost a year ago I shared an advice to use test cases. Not because they are crucial during testing or dramatically improve the quality of the product (they are not), but because of value you get when you create test cases.

    A confession (and yes, you’d guess it anyway if you read the title): we don’t write test cases anymore.

    We stopped using them and the reason isn’t on the list of shortcomings of test cases. Actually I was aware of these shortcomings a year ago and I were all “test cases are great” anyway. What have changed then?

    We dropped test cases as a side effect of implementing Kanban, but you can perfectly use both if you like. In our case one of effects of switching to Kanban was making our pieces of functionality pushed (pulled actually) to development smaller. Before the switch we had pretty big features which were split into several (8-15) detailed user stories. After the switch we have much smaller features which would make 2 or 3 detailed user stories if we didn’t drop writing user stories at all.

    And the reason for making features smaller was simple – smaller features, smoother and more flexible workflow.

    Initially we were connecting test cases to features, not user stories. It was so because pretty often one testing flow was going through a few different user stories. I told you they were detailed. When standard feature-size went down we realized there’s much less value in preparing test cases.

    Once again: main value of creating test cases is thinking about specific usage scenarios, looking for places forgotten during design. The more complex feature the bigger are chances there is something screwed up. With small features test cases lost much of their value for us since most problems we were able to locate instantly and fix problems without additional technique. And yet the effort needed to create and maintain test cases was still significant. So we dropped the practice.

    It looks like my advice is: make your features smaller, and then you’ll be able to drop user stories and test cases. And I must say it does make sense, at least for me. What do you think?

  • Shortcomings of Test Cases

    I’ve already told you that writing test cases is worth the effort. What I haven’t stressed are shortcomings of test cases.

    If you take time to prepare test cases you most likely do it for the reason and I guess not for “my manager will fire me if I don’t” one. You want to invite some organization to your testing and improve overall quality of the product at the end of the day. I don’t want to go deeper for motivations since it isn’t the point of this article.

    The thing you do is preparing some scenarios up front, definitely before you actually start testing. Generally the earlier the better but either way test cases are before test execution, that’s for sure. Basically when you prepare test cases you imagine how you’d like the application to be tested and you write it down. Do this, verify that, expect something other and proceed to the next step.

    Now what’s wrong with that?

    1. You suck at predicting how your users would use the products. We all do. Applications are built by developers who don’t even try to imagine how users will interact with software. Then they’re tested by quality engineers who are also far away from average users. Then they’re deployed by people who actually talk with users but can’t influence application development in almost any way. The whole process is driven by product managers whose job is to think about general concepts, not detailed usage flows. Who cares about usage scenarios then? Well, pretty much no one.

    2. Test cases are closed scenarios by definition. When designing test cases you don’t have “flow with your gut feeling” block at hand to put it in the diagram. You actually have to make it possible to verify test results thus you’re limited to some specific path – one among hundreds or thousands. This means you won’t go through the vast majority of cases yet you will decide whether a new feature works well or not.

    3. It’s not possible to have a complete set of test cases. A typical method which takes only an integer as an argument can be called in possibly 2^32 different ways which is basically infinity. If you add that you may need to call another function as a prerequisite you end up with a number which blows my head up. And we’re talking only about a single function, not about whole application. Test case is always a dramatic simplification in terms of possible usage scenarios.

    4. It’s easy to complete test cases and call it a day. Hey, after all test cases are for quality engineers, aren’t they? QA people should base their activities on test cases. Except they shouldn’t limit their work just to test cases. And you want it or not there’s always a bit of incentive to turn off your thinking and just follow the plan instead of going an extra mile. I’ve already said that a couple of times, but QA team is no place for follow-the-book people. Creativity is a key trait for QA engineers, yet we don’t live in an ideal world and you don’t have only the best and the most creative people in QA team, no matter what your company website says.

    With all these shortcomings I still strongly advise you to write test cases. You may try to do them as precise as possible, or you may just use general flows leaving some autonomy for testers but either way it will pay off. Especially if you understand where test cases fall short and find a way to deal with that using other tools.

  • What Is the Main Benefit of Writing Test Cases?

    Test cases are here for a long time. Used equally willingly by these who work with heavy-weight processes and those who choose the agile way. They can be stored in an appendix to documentation in BDUF (Big Design Up Front) project or be written on yellow cards along with user stories. Where’s their main value then?

    Think about the comparison: test cases for application functionality are the same as unit tests for application code. Except of one thing: it is way harder to write good and complete list of test cases. You may safely assume that’s impossible. This means the main value of test cases isn’t in covering full application functionality because you simply won’t achieve that, no matter how hard you try.

    Well, maybe it is a valuable tool for QA engineers? You know, they have some kind of walkthrough for testing sessions. Hm… show me quality engineers who prefer just to follow test cases instead of explore the application by themselves and I’ll tell who should be fired. Testing is such a creative task that there should be no place for follow-the-manual people in QA teams.

    OK, test cases don’t ensure completeness of tests and don’t help QA people much. So? Show me the money, as one of my customers used to say. Where’s value?

    The main benefit of test cases lies in the activity of writing them down. Or, to be more precise, in the process of thinking about them before they’re written down.

    Test case is not much more than short scenario of application usage. Sometimes pretty weird scenario but still. Unfortunately we often forget about this perspective. We tend think in terms of code or requirements and forget about the very basic thing – no class in the code and no requirement alone makes any sense for the end-user.

    Looking at application while being in end-user’s shoes benefits in two ways:

    – You find black holes of your design which aren’t covered with user stories, or requirements

    – You find usability issues since you see whole usage flow, not just a bunch of atomic functions

    The main value of writing unit tests is actually within the writing part.