Almost a year ago I shared an advice to use test cases. Not because they are crucial during testing or dramatically improve the quality of the product (they are not), but because of value you get when you create test cases.
A confession (and yes, you’d guess it anyway if you read the title): we don’t write test cases anymore.
We stopped using them and the reason isn’t on the list of shortcomings of test cases. Actually I was aware of these shortcomings a year ago and I were all “test cases are great” anyway. What have changed then?
We dropped test cases as a side effect of implementing Kanban, but you can perfectly use both if you like. In our case one of effects of switching to Kanban was making our pieces of functionality pushed (pulled actually) to development smaller. Before the switch we had pretty big features which were split into several (8-15) detailed user stories. After the switch we have much smaller features which would make 2 or 3 detailed user stories if we didn’t drop writing user stories at all.
And the reason for making features smaller was simple – smaller features, smoother and more flexible workflow.
Initially we were connecting test cases to features, not user stories. It was so because pretty often one testing flow was going through a few different user stories. I told you they were detailed. When standard feature-size went down we realized there’s much less value in preparing test cases.
Once again: main value of creating test cases is thinking about specific usage scenarios, looking for places forgotten during design. The more complex feature the bigger are chances there is something screwed up. With small features test cases lost much of their value for us since most problems we were able to locate instantly and fix problems without additional technique. And yet the effort needed to create and maintain test cases was still significant. So we dropped the practice.
It looks like my advice is: make your features smaller, and then you’ll be able to drop user stories and test cases. And I must say it does make sense, at least for me. What do you think?
Subscribe RSS feed
Follow on Twitter
Subscribe by email
9 comments… add one
Seems like this sort of solution requires a highly skilled QA team and that QA engineers are in fact are subject matter experts. Well, it depends on the software of course (testing games is not the same as testing banking systems or missile guidance system).
Just because a feature is smaller (read simpler) to implement for software engineers it does not necessarily mean that test scenarios are all of a sudden obvious for the QA team.
I can accept that it worked for you but I cannot see (smaller features = simpler testing) being a general law.
You’re right about one thing. I seamlessly made one assumption – testers/QA engineers have to be subject matter experts. Actually if they aren’t your tests would likely to suck, no matter what.
Even the best possible set of test cases executed by engineers who don’t really know what’s happening in the application would have very little value. You will just go through paths you’ve already designed (writing test cases) and which are likely covered well by development team.
I never believed in testers who don’t live with the application as it is built. Even if you feed them with the best possible tools they won’t stand comparison with a couple of clever guys who were with the product from the very beginning.
Pawel, this may look like a good idea today but it may not be so good in the long run.
People move, they get different jobs, move to a different country, retire or simply immerse themselves in a new project from whole different domain.
The QA may be now subject matter experts (and I totally agree with you that they have to be) but the knowledge they hold today is volatile. If you have no written memory of what you’re application should do, be it use cases, user stories or test cases, years from now
maintenance will become a nightmare.
I think that in an Agile environment Test Cases can be merged into User Stories. A well written User Story is like a mini-spec – it describes the design of the piece of functionality being implemented. I should also describe entry points and outcomes of that particular piece, unless they’re obvious.
This way test cases don’t need to be maintained, as marking a user story as completed ticks off that same test case too.
What you wrote is true – the smaller the pieces of functionality you’re implementing, the less documentation you need for them, as their scope becomes pretty much obvious. However moving in this direction carries the danger of the whole system becoming inconsistent, unless the small pieces follow a higher-level design.
Madalina,
That’s why we have documentation – to avoid storing knowledge just in someone’s head. Test cases aren’t the best way of documenting software anyway.
Now we may have problem only if something very unexpected happens which makes our QA folks instantly unavailable. And even then we would lose only very small part of our knowledge – one which is under development. Luckily enough developers should be pretty much updated on this part.
I’m aware that the bigger product grows the harder it is to learn it when a newbie joins the team. But it works the same way no matter if your documentation have form of test cases or some other.
Michał,
Well, we don’t write user stories either.
You’re right about the risk connected with scaling features down. This is job of product manager/product owner and architect to keep new functionality consistent with the old and with application architecture.
In our case it is a part of design – this is the time when we discuss what the feature exactly means in terms of usage scenarios, changes in code, alignment with architecture etc. Some potential traps we are able to name (and avoid) but still sometimes we end up with new feature realizing that some important part is missing.
But overall I strongly prefer small features.
Pawel,
Curious: do your MMFs have acceptance criteria or state conditions of success? If so, are those not test cases.
Pawel,
I’ve been following your comments about no longer using stories and about the size of MMFs/features/user-stories. Now that you’ve stopped using stories and are using smaller MMFs, are you not still doing exactly the same thing in any event (namely, specifying what needs to be built)? What you seem to be really about, though I don’t think you’ve explicitly stated it this way, is finding the sweet spot for MMF/user-story size _for_your_team_. I would argue that the discussion is getting somewhat muddled by changing the name of what you are doing (user-story -> MMF) when you are doing the same thing in either case. No?
Andrew,
We don’t literally write acceptance criteria for MMFs (or user stories if you prefer). Features are small and simple enough we’re able to intuitively understand what application should do when the feature is finished. And yes, it meas we have from time to time discussions like:
Tester: Should the feature foo have the bar option?
Product Owner: Well, you tell me. Should it?
Developer: It doesn’t really make sense for me because [something].
Tester: You’re right, forget it.
At the end of the day it ends having whole team on the same page even if we decide to add the bar option to the feature foo.
You’re right about two things – we probably have found a sweet spot for our team in terms of feature size, even though we happen to have significantly bigger ones from time to time too. And yes, naming may add some mess into the discussion. I don’t actually care if we call it a feature or story or sticky note or whatever.
I started using feature term because it is used in Kanban, which we follow, and because content of our sticky notes doesn’t really suit the hard-core definition of user story. We focus more of pieces of application functionality than finished usage scenarios, but I guess there are many teams which do the same and still call it user stories.