Pawel Brodzinski on Leadership in Technology

Whatever it takes to lead a team, build a product, and run a business

Tag: estimation

  • Story Points and Velocity: The Good Bits

    You get what you measure. The old truth we keep forgetting about so often.

    Story Points and Velocity

    One relevant context to remember this is when we measure progress of project teams. The set that was wildly popularized along with Scrum is story point estimation, most typically with Planning Poker as a method to come up with the estimates, and measuring velocity. In such a set velocity, which simply is a number of story points completed in an iteration, is a primary measurement of pace.

    I don’t say the whole set is evil. What is evil though is how it is frequently used. Story point is pretty much meaningless – the same story can be estimated 2 or 8 and both are perfectly valid sizes. This means that the moment someone starts expecting specific velocity they will get it. In fact, continuous improvement in velocity is as easy as pie. It’s known as story point inflation.

    The same thing will happen when someone starts comparing teams basing on velocity.

    And then there’s expectation for velocity to be predictable, which translates to low variability. If that’s the goal story point estimates will be gamed so velocity looks predictable.

    How much does it have to do with any real sense of sizing?

    OK, I hear the argument that these all are dysfunctions related to velocity. Fair enough. Let’s assume for the rest of this article that we are doing it right.

    Measuring Progress

    The problem is, that the whole activity of estimating story points doesn’t provide much value, if any. What Larry Maccherone’s research shows is that the best measure of throughput is simply counting the stories or features done.

    Let me stress that: it doesn’t matter what size we initially though a story or a feature would be. What matters is that it’s either completed or not. That’s it.

    Larry knows what he’s talking about. The data sample he had was from ten thousand agile teams, vast majority of them being Scrum teams. If there had been a quality signal in story point estimations and measuring velocity he would have seen it. He didn’t.

    So even if we do the whole thing right it’s just a complete waste of time. Or is it?

    Estimation

    One part of Planning Poker, or any other discussion about story point estimates, is validating all sorts assumptions about a feature or a story. Another is addressing gaps in knowledge about the task.

    These discussions can provide two valuable bits of data. Sometimes we realize that the chunk of work hidden behind a story is simply too big and should be split it. Typically it’s after someone provides input about complexity of a scenario.

    The outcome of such a scenario would be simply splitting a story.

    A different case is when we realize we simply don’t know enough to come up with any meaningful sizing. It may be either because we need more input or simply we are discovering a new land and thus our level of uncertainty is higher. The reason doesn’t matter.

    What matters is that in such a case we deal with more uncertainty than normally thus we introduce more risk.

    In both cases we get additional valuable information. Beyond that a discussion whether something is worth 5 or 8 story points is irrelevant.

    No Bullshit Estimation Cards

    That’s basically rationale for no bullshit estimation cards. I like to think of it as of “Story Points and Velocity: The Good Bits.”

    Instead of focusing on futile discussion and potentially driving dysfunctional behaviors we get a neat tool that keeps few valuable parts of the approach. And you get a little bit of sense of humor for free.

    By the way, there’s a more serious a.k.a. politically correct version too.

    It saves time. It provides value. And most of all, it makes you focus on the right challenges.

    And in case you wondered, you can get your own deck (or as many as you want really).

    esimtaion cards

  • Evolutionary Change of Portfolio Management

    There’s a fundamental flaw in how we manage project portfolios. What typically happens is we focus on estimated cost of work and expected profit. After all these are parameters that decide whether a project is successful or not.

    Estimated Cost and Expected Profit

    There’s whole ongoing discussion on how to estimate, when estimates give us value and whether we should estimate at all. A bottom line is that getting estimates right is extremely hard.

    A problem is that, even if we got our estimate right, the scope of a project will change anyway thus it will affect the original estimate. Few companies would bother to change the estimates but even for these few there already are some numbers in contracts and budgets that won’t change anyway.

    It seems that one of the critical numbers we use to make decisions about our portfolios is vague.

    By the way if we are in the context of fixed price contracts it automatically means that the other number—expected profit—has changed as well.

    It’s not a rare case to see these values changing drastically. I can come up with examples from my experience when, because of misestimate or increasing costs instead of margin of 30-60% the project ended up costing a few times more than the revenue a company got for doing it. Not only wasn’t it even close to breakeven but the company paid a huge toll in terms of both direct costs and lost opportunities.

    A partial solution to that is transiting to time and material contracts where the revenue is a derivative of effort invested in building a project. While that definitely helps to avoid direct loses it doesn’t really fix the variables. The time span of a project can be changed thus the absolute numbers will be altered.

    What You See Is All There Is

    Even if we potentially could get our number right it wouldn’t really help. A root cause for that is a concept described by Dan Kahneman: What You See Is All There Is (WYSIATI). Human brain doesn’t look for data points other than those readily available. In other words if we define existing and potential projects from a perspective of estimated cost and expected profit these will be the main, or even the only, inputs for decisions on starting new projects.

    This has very sad consequences. In neither case we assume that in normal circumstances we will be working on unprofitable ventures. If we are in the world of fixed price contracts the final price will directly depend on an initial estimate.

    We’ve already discovered one painful dependency between an estimate and a profit. While costs are changing the contract terms remain the same affecting profits. It is even worse than that. The price was calculated based on the same estimate most likely by multiplying it be a factor of expected margin. Unfortunately humans are crappy at estimation.

    And don’t even get me started about all those dysfunctional situations when there’s a pressure to reduce an estimate because a client has a limited budget or something.

    In either case, a picture of all the potential projects will be rosy. We won’t allow it to be different than that. We will tweak the numbers back and forth until they fit the rosiness of the picture.

    The ultimate issue of this is that, because WYSIATI kicks in, our decisions about starting new projects will be overly optimistic and heavily biased toward perceived attractiveness of our endeavors. A simple fact that it has very little to do with the real outcomes of projects or our appalling track record with assessing attractiveness doesn’t change it. What you see is all there is.

    Overcommitment and Multitasking

    The end result of this bias is easy to predict. Since all things seem to be overly attractive we tend to start too many projects. Eventually, and pretty soon, we are overcommitted beyond the point where we can comfortably finish everything before agreed deadlines.

    Even in the cases when a flash of sanity keeps us from jumping off the cliff and stop one step before there’s usually not nearly enough slack in the system to account for any emergency that would happen.

    Of course there’s no plan whatsoever in case our initial estimate was wrong and we need more effort to finish a project, because we’ve proven to be perfect estimators in the past and never got anything wrong. Oh, wait…

    What does happen next? Expediting of course. When heroics within a team is not enough we pull people from other teams just like there was no tomorrow and they weren’t doing meaningful work on other committed projects. Anyone wants to guess what happens in the projects that we cannibalized to save the other one?

    More of the same vicious cycle.

    Not only such work is ultimately inefficient but it also hits sustainability of the business as it damages relationships with clients.

    Informed Decisions

    To reverse that cycle we need to go to the first place, the point of commitment, and change the drivers that influence our decisions. Bearing WYSIATI in mind, what we need is access to meaningful bits of information.

    We need to know how our current commitments affect our capabilities over time. In other words we want to know which teams are working on which projects. Not only now but also in reasonable future. What reasonable future means will of course vary depending on a context.

    How would a new project fit available capabilities? Do we have people with the right skills and knowledge available? When are they available? How much slack do we need to be able to respond to uncertainty in existing endeavors?

    We need to know, at least roughly, what is Cost of Delay for new projects. Is it possible to delay start of this project for a month? How about longer? What kind of costs, or lost future gains will it incur?

    What is value of a new endeavor? Is it money that earn only? How about long-term relationships with a client or bringing more financial safety thanks to a steady flow of work?

    By the way, when we are talking about value, it is crucial to remember that value for our organization doesn’t equal value for a client. Of course, the more aligned these two are the better, but it’s almost never an ideal fit.

    What are the risks attached to a project? How sure can we be about all the assumptions we make?

    These are all crucial factors that we should be taking into consideration when making a decisions about starting any new project. If we don’t have them at hand, as a result of WYSIATI, they will be ignored and we’ll be back to our rosy, overly optimistic and utterly harmful view of portfolio.

    Bad news is that to do better we need pretty broad understanding of the mechanisms that drive us sideways. Good news is that once we know that the change isn’t that painful. Focusing on different kind of data will alter decision making process on a portfolio level.

  • Portfolio Management: The Search for Value

    I am talking about the cost of multitasking pretty frequently. This discussion gets even more interesting when we are in portfolio management context. Why? It is so because we are dealing with extremes more frequently on that level.

    Cost of Multitasking

    Let’s imagine a situation when a software developer is dealing with three tasks concurrently. We know that it isn’t efficient. It may be based on our knowledge how multitasking affects our work but it may as well be our intuition.

    Now, would we have the same intuition if we changed the context and we were discussing a team working concurrently on three different projects? Interestingly enough, we’d be looking for arguments why in such a context it isn’t that much of a problem. Stuff like: part of the team would work on one project and another part on another project. Sounds familiar, doesn’t it?

    Thing we typically forget about is how team members would interact with each other. They wouldn’t think of themselves as of isolated sub-teams. They will be frequently looking for colleagues’ help thus they will make other people switch projects for a while every now and then. Finally, we have the coordination effort that has to done. Who is working on what, what is the status of everything, etc.

    This gets even worse when the distribution of people across the projects isn’t fixed. Then people would be thrown to one or the other initiative depending on the current situation. Why is it costly? Zeigarnik Effect describes that we, as humans, have intrusive thoughts about stuff that we left unfinished.

    In other words, if I change a project but I haven’t wrapped up my part in the current initiative I will likely be interrupting myself thinking about the old tasks. In fact, I don’t need any external factors to incur the context switching cost to my work.

    There’s more than just efficiency penalty though. The teams that are working on more than one project deliver lower quality results, as Larry Maccherone points in the results of the research run across thousands of agile teams.

    Cluttered Portfolios

    Things get even more interesting when look at the big picture – not a single team but all the teams within one organization. It’s enough to ask two questions. How many teams are there? How many active projects or initiatives are run concurrently?

    The answers with the ratio in a range about 10:1 (ten projects per team on average) aren’t uncommon. It means that our forces are spread very thin, the coordination effort is significant and the frequency of people switching projects is fairly high.

    Another question may be: what is the percentage of people assigned to more than a single project?

    Either way, in some cases we’d find that not only do we have more projects than teams but also we have more projects than people working on them. Some extreme examples I know of would show that there are 4 times more ongoing projects than there are people available to work on these projects.

    How efficient is that?

    No reasonable person would do such a thing on purpose. Yet, this is happening pretty often.

    Project Attractiveness

    If we assume the goodwill of people managing project or product portfolio, there must be something wrong with the approach we typically use to manage the portfolio. Let’s think for a while data we use to inform our decisions on starting new initiatives.

    Obviously we know the client and, at least roughly, the scope of the project. On that basis we come up with the idea how much the project would cost and what revenue we can get out of it. How do we do this? Well, we estimate.

    One problem is that, as Daniel Kahneman points, we are biased toward the most optimistic scenarios. Another one is when we do these estimates. Johanna Rothman phrases it:

    If you fall for estimation as your way of valuing projects in the portfolio, you are doomed to fail. Why? Because you are trying to predict the cost or the date when you know the least about the project.

    Typically, we base on insufficient data and use our flawed estimation skills to come up with the measures that are supposed to help us assess value of projects. That doesn’t sound like an extremely useful approach if you ask me.

    The problem is that pretty frequently that’s all we have in our toolboxes: an unreliable total cost estimate and expected income based on that very estimate. If we are really lucky we can say a bit about high-level risks too. However, if nothing else is in place the only application of risks assessment on this stage would be tweaking the income expectations so it includes potential screw ups.

    All in all we end up looking at projects through attractiveness glasses. It’s all about how much profit we can potentially earn doing this or that project. Given that we model income on expected total cost almost all of the projects will look reasonably profitable, thus all of them will be considered attractive.

    There’s a pattern our brain follows, called What You See Is All There Is (WYSIATI). It basically says that whenever we are judging something we take into consideration only evidence that we have at hand and omit data that we could potentially gather to inform our decisions better. In other words, if what we see is cost and profit we won’t automatically be looking for other data points that can tell us something about value of the project. We will just do our judgment on information we have at hand.

    If we put WYSIATI on the top of our limited and unreliable data almost every initiative would look appealing. It is no surprise that we end up with heavily overloaded portfolios.

    Where Value Is

    The question we should be addressing is not the one about expected profit, but about expected value. The problem is that the latter is more difficult to answer. On different occasions, during my presentations, I’ve had a chance to ask audiences whether they know what the value of features they build for their clients is. Few people do.

    We simply don’t think about our products and projects in such terms. Even when we do though, it solves only a part of the puzzle. One oversimplification I often notice is that we boil down the discussion about value to the users or clients only. The problem is that both parties will define value differently. Let me give you an example.

    At Lunar Logic we build web applications for our clients. We can build a product for a client that delivers valuable features in a predictable fashion. We can even help the client shape the product so we avoid the non-value adding functionality. The client would be happy – they’d got what they define as value. At the same time though, the client may simply be a toxic person who would make the team’s life miserable. Now, on our end, depending on a situation, it means that overall value of such a project may as well be negative: we would have gotten the money (one aspect of value) but lose a part of the team (another aspect of value). It is likely that the former wouldn’t compensate for the latter.

    I don’t have all the answers for the questions about value. The cases when I’d disagree with how our clients define value aren’t rare. On my end, while I can say what we value as the company, sometimes it is very difficult to assess value for initiatives we undertake. It’s not as bad as it looks though.

    We don’t need to define value in absolute numbers. If the goal is to improve the way manage portfolio most of the time we will need to answer only relative questions, e.g. which project is more valuable given the criteria that are important in a given context?

    In fact, the most valuable outcome of the discussion will not be the exact assessment. It will be a discussion itself as it will likely cover the areas we typically ignore, thus will yield in the better understanding of the situation.

    It won’t happen though unless we understand the limitations of our decision-making process on a portfolio level. Otherwise we will keep doing even more of the same thing make our teams even more inefficient and miserable.

  • Estimation Quality

    OK, so I am on yet another agile event. I’m sitting there in the last row and a guy on the stage starts covering estimation. That’s interesting, I think, maybe I’ll learn something. After all estimation is something that bothers me these days.

    Planning poker, here it comes. Yawn. People would pull the card with their estimate, yadda, yadda, yadda, they’d discuss and agree on a story point value. Yawn. The distribution of estimates would be normal.

    Wait, now the guy has my attention.

    So apparently he knows the distribution of the estimates. Good. How about checking what the distribution of lead times for historical data is. Because, you know, there are people who have done that, like Troy Magennis, and it seems that this distribution isn’t normal (Troy proposes Weibull distribution as the closest approximation).

    In short: if your estimates have a different distribution than historical data for the tasks that the team has completed the estimates are crap.

    Back to the presentation. The guy points how estimates for big tasks are useless in his context. Well, yes, they are. All the estimates in his context are crap from what I can tell. Then he points that we should be splitting tasks to smaller ones so planning poker makes sense and produce more reasonable estimates.

    Um, sorry sir, but you got it wrong. If you are using estimates that are distributed differently than historical lead times you are overly optimistic, ignorant, or both.

    How about this: I will not poke you in the eye with my pen but you will check the distribution of the estimates and the past lead times. Then you will recall some basic stuff from statistics course and stop selling crap that we can’t possibly answer the question: “when will it be done?”

    OK, rant aside. I don’t say that coming up with the idea how long it will take to build something is easy. Far from that. I just say that there are pretty simple things that can be done to verify and improve the quality of the estimates.

    Once you got the estimates look at their distribution. If it is different than what you know of historical data (because you’ve checked historical data, haven’t you?) the estimates don’t bear much of value.

    In fact, you can do better. Once you know the distribution of historical lead times you may use that to model estimates for a new project. You don’t really need much more than simply basic work break down. You can take the worst case scenario from the past data and assume the best case scenario is 0 days and let the math do its magic. No guesswork whatsoever needed.

    The credit for this post should go to Troy Magennis, who opened my eyes to how much use we can make out of limited data we have at our hands.

  • No Estimates Middle Ground

    The idea of no estimates (or #NoEstimates) is all hot these days. People would choose different parties and fight a hell of fight just to prove their arguments are valid, they are right and the other party got it all wrong. I’d occasionally get into the crossfire by leaving a general comment on a thread on estimation in general, i.e. not steering a discussion for or against #NoEstimates.

    And that’s totally not my intention. I mean, who likes to get into crossfire?

    What No Estimates Mean

    A major problem of no estimates is that everyone seems to have their own freaking idea what that is. Seriously. If you follow the discussions on the subject you will find pretty much anything you want. There are crusaders willing to ban all the estimates forever as they clearly are the source of all evil in the software world. There also are folks who find it a useful tool to track or monitor health of projects throughout their lifecycles. You definitely can find people who bring to the table statistical methods that are supposed to substitute more commonly used approaches to estimation.

    And, of course, anything in between.

    So which kind of #NoEstimates you support, or diss for that matter? Because there are many of them, it seems.

    Once we know this I have another question: what is your context? You know, it is sort of important whether you work on multimillion dollar worth endeavor, an MVP for a startup or an increment of established application.

    My wild-ass guess is this: if every party getting involved in #NoEstimates discussions answered above questions they’d easily find that they’re talking about different things. Less drama. More value. Less cluttered twitter stream (yeah, I’m just being selfish here).

    Is this post supposed to be a rant against discussion on no estimates?

    No, not really. One thing is that, despite all the drama, I believe that the discussion is valuable and helps to pull our industry forward. In fact, I see the value in the act of discussing as I don’t expect absolute answers.

    Another thing is that I think there is #NoEstimates middle ground, which seems to be cozy and nice place. At least for me.

    Why Estimating Sucks

    Let me start with a confession: I hate estimating. Whoa, that’s quite a confession, isn’t it? I guess it is easily true for more than 90% of population. Anyway, as long as I can get out avoiding estimation I’d totally go for that.

    I have good reasons. In vast majority of cases estimates I’ve seen were so crappy that a drunken monkey could have come up with something on par or only slightly worse. And last time I checked we were paying drunken monkeys way less than we do developers and project managers. Oh, and it was in peanuts, not dollars.

    It’s not only that. Given that kind of quality of the estimates the time spent on them was basically waste, right?

    It’s even worse. It was common when these estimates were used against the team. “You promised that it will be ready by the end of the month. It isn’t. It’s your fault.” Do I sense a blame game? Oh, well…

    And don’t even get me started with all the cases when a team was under pressure to give “better” estimates as the original ones weren’t good enough.

    Why We Estimate Then

    At the same time, working closely with clients for years I perfectly understand why they need estimates. In the case of a fixed-price contract we have to come up with the price somehow. That’s where estimates come handy, don’t they? There also is a million dollar question: so how much will I spend on this thingamajig? I guess sometimes it is a million dollar question literally…

    So as much as I would prefer not to estimate at all I don’t hide in a hole and pretend that I’m not there when I’m asked for an estimate.

    All Sorts of Estimates

    Another story is how I approach the estimation process when I do it.

    I would always use a range. Most of the time pretty broad one, e.g. the worst case scenario may mean twice the cost / time than the best case scenario. And that’s still only an estimate meaning that odds are that we would end beyond the range.

    Whenever appropriate I’d use historical data to come up with an estimate. In fact, I would even use historical data from a different setup, e.g. different team, different project. Yes, I am aware that it may be tricky. Tricky as in “it may bite you in the butt pretty badly.” Anyway, if, basing on our judgment, team setup and feature sizing is roughly similar I would use the data. This approach requires much of understanding the dynamics of different teams and can be difficult to scale up. In my case though, it seems to work pretty fine.

    I’m a huge fan of Troy Magennis and his work. By the way, despite the fact that Troy goes under #NoEstimates banner, he couldn’t possibly be farther from folks advising just to build the stuff with no estimation whatsoever. One of most valuable lessons we can get from Troy is how to use simulations to improve the quality of estimates, especially in a case where little data is available.

    Finally, I’m also fine with good old guesstimation. I would use it on a rather general level and wouldn’t invest much time into it. Nevertheless, it works for me as a nice calibration mechanism. If the historical data or a simulation shows something very different than an expert guess we are likely missing something.

    Interestingly enough, with such an approach having more details in specifications doesn’t really help, but that’s another story.

    On the top of that, whenever it is relevant, I would track how we’re doing against initial estimates. This way I get early warnings whenever we’re going out of track. I guess this is where you think “who, on planet Earth, wouldn’t do that?” The trick is that you need to have quite a few things in place to be able to do this in a meaningful way.

    A continuous flow of work gives us a steady outcome of delivered features. An end-to-end value stream means that what is done is really done. At the same time without continuous delivery and a fully operational staging environment end-to-end value stream is simply wishful thinking. Limiting work in progress helps to improve lead time, shortens feedback loops and helps to build up pace early on. And of course good set of engineering practices allows us to build the whole thing feature by feature without breaking it.

    Quite a lot of stuff just to make tracking progress sensible, isn’t it? Luckily they help with other stuff too.

    Nevertheless, I still hate estimation.

    And I’m lucky enough to be able to avoid it pretty frequently. It’s not a rare case when we have incremental funding and budgets so the only thing we need is keeping our pace rather steady. And I’m not talking here about particularly small projects only. Another context where estimation is not that important is when money burn-out rate is so slow (relatively) that we can afford learning what the real pace is instead of investing a significant effort into estimating what it might have been.

    No Estimates Middle Ground

    To summarize the whole post I guess my message is rather straightforward. There’s value in different approaches to estimation so instead of barking one at another we might as well learn how others approach this complex subject. For some reasons it works for them pretty well. If we understand their context, even if ours is different, we might be able to adapt and adopt these methods to improve our estimation process.

    That’s why I think the discussion is valuable. However, in terms of learning and improving our estimation toolbox #NoEstimates notion doesn’t seem to be very helpful. I guess I’ll stay aside in the middle ground for the time being.

    By the way, if we are able to improve our cooperation with the clients on the estimation I couldn’t care less whether we call it no estimates or something different.

  • Making Use of Estimates

    I’m not a fan of estimation. I try to avoid it when I can. However, I’m not orthodox. I won’t be telling everyone that refusing to estimate is the way to go.

    I admit that I’m now in a comfortable land of time and material contracts that give more flexibility to the clients and more safety to the vendors at the price of mutual trust. At the same time Lunar Logic is the first of my jobs that works this way, so I’m more than familiar with fixed everything sort of contracts too. In that world estimate is a king. A lousy king though.

    Despite the fact, that fixed price projects are a nightmare from my past life, we can’t forget about estimation at all. Even those awesome folks we started working with recently started with asking us to deliver estimate in story points or other abstract measure of our choice.

    I intended to track progress in the project with Cumulative Flow Diagram (CFD) that counts only a number of tasks completed. However, having individual estimates for all the stories in backlog it’s easy to have two CFDs: one tracking a number of stories and tasks and the other that refers to initial estimates.

    One might suppose that it would be basically the same data on both charts. And one would be wrong.

    Let’s start with the charts. Here’s standard CFD for the first few weeks.

    CFD

    And here’s CFD that bases on estimates in story points.

    CFD

    What’s the difference? In fact, there are quite a few of them.

    For the starters, the scope changed only on the first chart. What does it mean? Simply that despite no change in total points there were new tasks. In this case almost all of them were non-functional tasks like setting up the environment, design pre-processing and adjustments in existing code that enables integration with the new part of an application.

    There was also a split of one of stories to a couple of them for the convenience. The total estimate didn’t change although there was a new story to build.

    Another thing you may notice is that on the second chart we’ve finished anything pretty late in the project, while the standard CFD shows completion of a couple of features early. Again it is connected with non-value-adding and non-functional features that had to be completed to get the team running.

    Finally, and most interestingly, try to judge amount of work in progress and size of stories in progress looking at the charts. A hint: you will find the answer to the former in the first and to the latter on the second chart.

    Standard CFD tells you that we care about limiting WIP and avoid starting too much. The story point-driven chart shares a different story. Steepness of the coding curve indicates that we’ve started with heavy features. In this case heaviness was mostly driven by risks – the heaviest features were most risky ones. We decided to start with them to address risks pretty early in the process so if there’s a screw up on the way we know it as soon as possible.

    By the way, I don’t focus here on the data that you specifically get from CFD. You could draw exactly the same conclusions if you used burnup charts, which are gateway drug to Cumulative Flow Diagrams.

    Obviously, CFD can tell you a bit more, e.g. about the awesomeness of the client, as they don’t ever have anything stacked in approval. Whatever is ready it gets verified and accepted.

    Anyway, the point is that despite I don’t feel that detailed estimates in this project really pushed us further (maybe despite building trust relationship, but that’s a different story) we can still use the work invested in estimation to learn something.

    And, as you see, you can learn quite a bit using very simple tools.

  • Trap of Estimation

    So we had this project which was supposed to end by the end of July. Unfortunately a simple burnup chart, which we used to track progress, seemed rather grim – it was consistently showing very beginning of September as a completion date. A month late.

    Suddenly, one day it started looking almost perfectly – end of July! Yay!

    Wait, wait, wait. What? I mean, what the hell happened on a single day that we suddenly recovered from a month-long slip? Something stinks here.

    After a while of digging we came up with the answer. Actually team invested some time to re-estimate all the work, including the work which was already done. Now, how the heck does it affect burnup chart?

    It seems the chart on the y axis, where the work items were shown, had a sum of estimates and not just a number of tasks. It means that changing estimates in retrospect affects the scope and percent complete of the project. It actually means that such changes can affect predicted completion date as well.

    This is called creative accounting and some people went to jail because of that, you know.

    My first question is whether such re-estimation changes real status of a project in terms of how much functionality is ready, what is done, how many bugs are fixed or lines of code written or any other creative, crazy or dumb measure you can come up with to say how much work has been done. Or does it change how much more work is there to be done?

    No! Double no, actually. It’s just a trick to tell us we aren’t screwed up that much. Actually, I accept the fact that we might have been OK in the first place and the chart was wrong. That would be awesome. But fixing the chart in such way, one, doesn’t change the status of work in any way and, two, just covers the real issue so it is harder to address it.

    What is the real issue then? Well, there are a couple of them. First, using time-based estimates to show how much work is to be done is asking for troubles. Unless you are a freaking magician and you can make your estimates right 5 months before you even start working on the task, that is. If you’re just a plain human, like me, and you assume your estimates are wrong using them as a basis for tracing project progress seems sort of dumb to me.

    It would be much better to count features or, if they vary much in size, count weight of features. Say S size is 3 times smaller than M and this is 3 times smaller than L or something like this. By the way, as you gather historical data you can pretty much fix these factors learning from past facts.

    Second, even if you decided to go with estimates to judge how much work is to be done, what makes you thinking that fixing estimates in retrospect pushes you forward for an inch in terms of the next project you’re going to run? Do you expect to know exactly, in advance how much time will take you to build features in future projects? Because this kind of knowledge you’re applying now to “fix” your estimates in current project.

    I would rather prefer a discussion on how to judge the scope better at the beginning of projects because this is going to be your benchmark. In this case precise estimates are almost useless. I will likely be pretty close in terms of telling how many features we have to build. It’s going to be trickier to say which of them will be small, medium or large. But I refuse to guess how many freaking hours each and every feature will take to build because such effort is utterly futile. It just so happens that I’ve forgotten to take my damn crystal ball with me, so sorry, that’s not going to work.

    In this case estimation brings us to a trap. Knowing exactly how much time each work item has taken it is easy to track the progress in retrospect. Average 8-year old child would be able to connect the dots. However, unless you’re a bloody superhero and you’re going to have such data at the beginning of your next project don’t treat it as viable method of tracking progress.

    Use any data that will be available in high quality at the beginning of a project. Number of features, maybe sized in some way if your team have some experience in sizing and you understand variability of work.

    Anyway, whatever you do, just don’t change the benchmark in retrospect as it’s going to mess your data and cover a real problem, which is that you should improve the way you set the benchmark in the first place.

    By the way: if you happen to work on time and material basis you can perfectly ignore the whole post, you lucky bastard. Actually, I doubt you even reached to this point of the post anyways.

  • Get Rid of Estimation

    Software estimation. Ah, a never-ending story. Chances are good that, whenever you’re talking about building software, this subject will pop up soon. You can be pretty sure that basically everyone around has problems with estimation or simply struggles with it. And that’s virtually obvious that there would be a new sexy method of estimation every year or so. The method which is claimed to solve an unsolvable puzzle.

    Recently there was yet another question on estimation on Project Management StackExchange. This sole fact isn’t probably worth writing a whole blog post about that, but there was one side thread which is worth focusing at.

    One of advices I shared was that whenever you can you should avoid estimation at all. This sprung sort of objection. OK, so the subject definitely is worth wider discussion.

    First things first. Why do we estimate at all in the first place? Well, we usually want to know how much time it’s going to take to build this damn thing, don’t we? Um, have I just said “usually?” Meaning, “not always?” Actually yes. It’s not that rare when we either don’t need any estimate at all or we just want to have a general insight whether the project will be built in hours, days, weeks, months or years. In either of these cases just a coarse-grained wild-ass guess should be fine. If it’s needed at all.

    OK, but what about majority of cases when we need some kind of real estimate? For example all those fixed price projects where estimates are basically a part of risk management, as the better the estimate is the smaller are chances that the project goes under water. I can’t deny that we need to have something better than wild-ass guess then.

    Yet, we still can avoid estimating quite often.

    Let me start with one of obvious things about estimation: if you base on historical data, and you apply them in a reasonable way of course, you can significantly improve your estimates. In other words, no matter the method, if you are just guessing how much something is going to take, you will likely to end up with way worse results when compared to a method, which uses your track record. And yes, I just dared to name planning poker “guessing.” It is collective, involves discussion, etc but usually it is just this: guessing.

    Cool, let’s use historical data then. What’s next? My next question would be: how precise must your estimates be? Seriously, what kind of precision you aim for? My point is that we need very precise estimates very rarely. This is by the way the reason why I don’t use Evidence Based Scheduling anymore.

    Anyway, ask yourself a question: how much you would pay for bringing your estimates to the next level of precision. Think of it like being correct in terms of estimating in years, months, weeks, days, hours, etc. Let’s take just an average several-month-long, fixed-priced type of project.

    If I’m wrong with years I’m totally screwed, thus I’d pay significant part of project budget to be correct on such level. If I’m wrong with months it might be a hit on our reputation and also it may consume our whole profit we get of the project, so I’d be ready to invest something around the profit to be correct with months. Now weeks. Well, a week here, a week there, does it make such a difference in this kind of project? Can’t I just assume there is some variability here? Unless of course our deadlines are written in stone, e.g. you adjust your software to law changes. In most cases I’d invest just a handful of bucks here at best. Days? Hours? Are you kidding? Does it even make a difference that I spend a day more on such project?

    Now you know what kind of precision you expect from your estimates. Would it be possible for you to come up with estimates of such precision basing purely on historical data? I mean, can’t you just come up with a simple algorithm which automatically produces results reasonable enough that you can forget about all the effort spent on estimation?

    Whenever we come to discussing estimation I like to share the story from one of my teams: basing on a fact that we were collecting data on our cycle times and we reduced variability in task sizes coming up with the idea of standard-sized features we were able to do a very good job with estimates not estimating at all. We were simply breaking work down so we could learn how many features there are to build and then we were using very simple metrics basing on our track record to come up with the numbers our sales department requested. By the way: a funny thing is, almost all of that appeared as an emergent behavior – something we started doing as a part of continuous improvement initiative.

    Either way, even though we were capable of providing reasonably precise and reliable estimates we didn’t really estimate. I was surprised how easy it was to get rid of estimation, but then I can come back to the point from the beginning of the article: what is the point of estimation? You don’t do it just for the sake of performing the task; you do it for a reason. As long as you achieve your goal, does the method really matter? It means that you can get rid of estimating even if you do need some kind of estimates.

  • The Kanban Story: Coarse-Grained Estimation

    Recently I told you what a screwed way we chose to measure lead time in our team. Unfortunately I’ve also promised to share some insight how we use lead times to get some (hopefully reliable) estimates. So here it goes.

    Simplifying things a bit (but only a bit) we measure development and deployment time and call it lead time (and don’t feel bad at all about that). So how do I answer to our customers when they ask me when something will be ready?

    That’s pretty easy. If we talk about a single feature and there is no other absolutely top priority task, I take a look at the board to track down a developer who will be first to complete his current task and discuss with him when he expects to finish it. Then I know when we can start working on the new, super-important feature ordered by the client. Then it’s enough to add two weeks (which is our average lead time) and make some effort to look oh, so tired as I’ve just completed such a difficult task of estimation. Oh, I need to tell the customer when they’re going to get the darn feature too, of course.

    This however happens pretty rarely. We try to keep our MMFs (Minimal Marketable Features) stick to their name which means they are usually small. This also means that described situations, when client wants just one feature from us, are pretty much non-existent. That’s why you might have noticed I didn’t take into consideration a size of a feature in described scenario. In the real life we usually talk about bigger pieces of functionality. What we do then is we split the scope into small MMFs and then use two magic parameters to get our result.

    One of parameters you already know – it is an average lead time. However there’s one more needed. It takes 13 days for the feature to get from the left to the right side of the board, but how many features are on the board at the same time? I took a crystal orb and it told me that on average we have 4,5 MMFs on the board. OK, I actually did a simple analysis of a longer period of time checking our completed MMFs and I got this result, but doesn’t a crystal orb sound so much better?

    Now, the trick I call math. On average we can do 4,5 MMFs in 13 days. Since I have, let’s say, 10 features on my plate I have to queue them. If I worked with iterations it would take 2,2 iterations (10/4,5) which basically means 3 iterations. But since we don’t use time-boxing I can use 2,2 and multiply it by 13 days of my lead time and get something about 30 days. Now I don’t start with empty board so I have to add some time to allow my team to finish current tasks. On average it would be half of lead time so we should be ready in, say, 37 days.

    And yes, this is rough estimate so I’d probably go with 40 days to avoid being blamed for delivering estimate which looked precise (37 looks very precise) but was just coarse-grained estimate.

    That’s basically what I went through before telling my CEO we’re going to complete management site for the product we work on in 3 months. Well, actually I didn’t told him yet, but 3 months there are. No less, no more.

    Although this post may look as it was loosely coupled with the Kanban Story it is definitely belongs there. So go, read the whole story.

  • Random Thoughts on Estimation

    A lot of discussion on estimation recently. A lot of great arguments but a lot of good old mistakes we’ve already went through. This brought me to a few random thoughts on estimating techniques.

    1. Estimation technique which involves discussion between different people is better than one which just simply uses their estimates as input.
    2. Using factual recorded historical data is better than basing just on experience.
    3. Smaller tasks can be estimated way better than big chunks of work.
    4. Every estimate starts with some kind of guess at the very beginning.

    I know these should be obvious yet I’m surprised how often people:
    – forget about them
    – deny these are true

    Then they head towards wild guesses with some magic number applied, which may have some sense but not when used instead of real estimation.