OK, this one is controversial. Software estimating. How to do it good, or good enough? First of all, when talking about the subject we all can learn a lot from Glen Alleman who bring quite an uncommon perspective to the area as he’s based in industries which tolerate missed deadlines with much less patience than the average.
Now, what’s the point? If you asked me how software estimating should be done I’d direct you to Steve McConnell book on the subject. Believe me or not, he didn’t write anything good (if anything at all) about role of wild guessing in software estimation. Which doesn’t make me avoiding wild guesses at all times.
Actually sometimes I make good use of them.
Now, go flame me.
What’s the use of wild guesses instead of proper estimation? Wild guesses don’t have to be so wild if you ask a right person. If I ask my development manager how long something would take and it’s a piece of software he’s fairly familiar with in vast majority of cases he’ll come with some guess. It should be reasonable enough to judge whether it’d take 9 months to do the job or rather just a few weeks.
I don’t consider that as a full-blown estimate. It’s hard to use it as a base to create a schedule but still it’s a valuable feedback to judge whether and when your team is capable to overtake the project or decide if the whole thing is worth further presales effort. If you need to develop a bunch of tools which your competitors already have in their product portfolio chances are good you won’t match their schedules and/or prices. If you add that your team is backed up for the next half of the year it’s probably not worth investing effort to do more than just getting first coarse-grained approximation.
Yes, every wild guess should be considered very carefully since after all it’s, well, just a wild guess not an estimate done according to rules. However doing proper estimation is quite an effort and sometimes you need something rough but quick instead of exact but demanding.
And remember: don’t ever, ever allow your wild guesses to be sent to sales department.
Subscribe RSS feed
Follow on Twitter
Subscribe by email
17 comments… add one
I know sometimes you have to give a quick answer for software stimation. If you find yourself in this kind of situations and you have to use wild guesses, use it but with wide-band delphi, don’t use the wild guess of just one expert. Greetings
Thanks for a comment Jesus and welcome on Software Project Management.
For more exact result I’d probably ask many people. However if I just want to know the order of magnitude I prefer to:
– interrupt only one person instead of the whole team
– find a person who’s familiar with a kind of software we’re talking about
I believe either Eric Brown (Aligning Technology) or Craig Brown (Better Projects) suggested the PDOOMA methodology.
I’ve taken to quoting it when I give estimates….
“Oh yea, I used to POODMA methodology to arrive at my estimate of…”
PDOOMA = Pulled Directly Out Of My Ass
Andrew,
Being trained (and having practiced for a short time) as a physicist, making estimates is a critical skill.
But “wild guesses” need to have a unit of measure for the boundaries.
“What is the mass of the Sun (our nearest star)?
• 1,000 KG?
• ~2×10^30 KG?
“How long will it take our team to stand up the SAP Supply Chain Module in 10 locations around the world?”
• 3 weeks?
• 20 years?
• 18 months?
You get the point.
PDOOMA has no place in the “profesion” of software development. Providing asn estimate with at least the first STD DEV is the minimal acceptable approach. Higher order statistics add more value up to the 3rd moment.
I can associate with the comment about telling the sales people. I have been on both sides of the fence and neither is easy.
“Wild Guesses” or “Rough Order of Magnitude – ROM” estimates are valid tools to do a quick assessment if something is worth investigating or not. As you said, not good enough for a schedule though!
When I was on the sales side I always qualified these with an escape clause saying that they are ROM and further work needs to be done to validate them. Also – use ranges where you can – it will be between x and y. However, they stick around.
As a sales person I would also always ask at least two people for an ROM. if they are close to one another – good! If not, seek more opinion.
You surely cannot avoid these but tread carefully.
Andrew,
If you came and told me you’ve used PDOOMA method to make your estimate I’d throw it into the trash and probably won’t come any more. Which wouldn’t mean it would be any good for you.
When people ask you about estimates they most likely don’t do that without a reason. They want to know rough time span of a project or they want to create detailed schedule or they need to do some budgeting or you-name-it. Depending on the goal different techniques should be made and different precision should be made. I think PDOOMA doesn’t suit in any of above situation and as such shouldn’t be used unless you personally hate a person who asks you for an estimate and you’re 100% sure you won’t do the project anyway. Which still doesn’t make it a good method.
If someone gives me a quarter to estimate I can give them a wild guess. If someone gives me a quarter and expects exact numbers I tell them I’m not really a miracle specialist.
Glen,
I was talking about the kind of situations you bring on the table.
“What is a time span for implementing such solution? Is it counted in days, weeks, months or years? Ar we closer to a single month or to 10 months?”
Schalk,
The trick is there aren’t many salespeople with as reasonable approach as yours. I’ve seen so many times when a wild guess was taken as reliable date and was treated as commitment. I’ve seen so many times when salespeople who were given a range of dates have taken the earlier and instantly forgot the other one.
Of course it depends on people but I worked with just a few salespeople who understood what is rough estimate, what is reliable schedule and saw a difference between these two.
Pawel,
Yes “qualified” and “calibrated” estimates are the starting point. But there is a fundamental issues (theological actually). In Bayesian statistics, if you do not know the underlying distribution of the random variables in your populaiton, the confidence interval on the sample is 50/50. That’s pretty poor odds when “managing” a project.
This is a complex problem and no addressing the issues is the source of much grief.
We’ve worked on many ERP projects where the kind of guessing we speak of here is common. The result is usually an 100% overrun. No an issue is the project is $10,000 but when it is $100M then we’ve got a real problem.
One recent book is “Software Sizing, Estimation and Risk Management” Dan Galorath and Michael Evans, 2007.
This may be a useful source for complex software estimating problem along with McConnel’s previous book.
All,
here’s a sample of the depth of the issue
http://www.pmforum.org/library/papers/2007/PDFs/Galorath-4-07.pdf from Dan Galortah.
This is one of many approaches to software sizing problem.
The notion that agile does not have to dela with this is coreect ONLY if you consider the project Level of Effort within a Time Boces schedule.
We’ve found the best use of agile is in the maintence or extension of existing software code bases. If you go looking at example you see most are like this. David Anderson’s examples, PayPal and Google are all extensions of operating platforms.
For “green field” development, some bounds are needed for cost and schedule and therefore a credible estimate – with upper and lower bounds and a statistical correlation between duration and effort – is needed.
Glen,
When I was working on ERP projects (for small to medium companies) we’ve been pretty successful with estimates when implementations were close to standard. Expert judgement worked really well then.
Of course experts were using their knowledge about past project but it wasn’t formalized in any way. People didn’t think in SLOC or function points categories, at least not consciously, but their experience allowed to bring pretty exact estimate ranges.
On the other hand lack of formal methods of analysis brought some serious problems when it came to plan a couple of big, non-standard implementations which didn’t aligned well with previous projects.
good article!
“Regards,
SBL – custom software
http://www.sblsoftware.com“
Pawel,
I hate to beat a dead horse, especially one where I got beaten up for making a joke about estimates.
However, I stubbornly stand by my belief that however much you dress them up with training as a physicist, PMI or any other methodology, estimates are nothing more than wild ass guesses. I do not believe there is any methodology that will produce estimates more accurate than PDOOMA.
Before I get flamed again, please consider this.
A study of 214 IT projects from 1998 to 2005 found that only 1 in 8 could be considered successful, meaning they met original time, cost and quality estimates.
http://www.bcs.org/server.php?show=ConWebDoc.19584
Others may have been aware of this study previously, I just came across it today, which is why I’m venturing back into the dangerous land of reopening this debate.
These were large, well funded projects. It is reasonable to assume that the people working on and running them were competent, well trained and motivated to make their projects succeed. However, with a success record of 1 in 8, it is reasonable to question the estimates and methodologies use.
The reason for my scorning estimating techniques and adopting PDOOMA is that I have been dragged through too many exercises in software estimation that quickly deteriorated and deviated so far from reality that we might as well have been discussing how many angels can dance on the head of a pin.
Why don’t we just admit that we don’t know how long something is going to take?
As a business owner who contracts with customers to deliver functionality. Whether my business is profitable and my customers happy really depends if I can propose a reasonable approach that is beneficial for both sides.
I don’t know how long something’s going to take. I don’t know what system is going to have bugs in it no one knew about before hand. I don’t know what companies I’m depending upon are going to get sucked into the black hole that is the current global economic crisis.
There are too many things I do not know and have no way of knowing that are going to affect my projects. One could spend huge amounts of time, energy and brain power trying to predict and estimate what’s going to happen in the future, and if you use enough scientific sounding mumbo-jumbo, you might even convince someone that you know. But the fact is, you do not know.
So yes, PDOOMA was meant to be a joke and it is one I’ve used with several clients, because at least I’m honest enough to admit I don’t know. That honest allows me to write contracts that neither suffocate my clients nor my company without wasting time on estimation exercises.
If you think you have a better, more accurate or more sophisticated system for estimating projects, I suggest looking at the controversy surrounding the Black-Scholes Model for estimating options. There are fewer variables to consider options and far more money and brain power have been applied to it. And guess what, it doesn’t work.
As Nils Bohr famously said: Prediction is very difficult, especially if it’s about the future.
http://www.duke.edu/~rnau/411quote.htm
I’m not going to flame you. You say you don’t know how much time some development would take. On the other hand you set up a deal, most likely with fixed fee. Now if you don’t know how much it would take how do you the deal will be profitable?
Personally I don’t expect to have estimates exactly on target. If the slip is about 10% or 20% in vast majority of cases that’s fine for both a vendor and a customer. I would like to avoid situations when project is slipped by 100% or more.
When you look at a project which was planned for 4-5 months and it’s already 20th month of a schedule something is terribly wrong. I’ve seen that and that was exactly a result of PMDOOMA approach. “Wanna deadline? Here it goes.“.
If I have a tool which doesn’t bring much additional hassle and it helps me either to improve initial estimates or post-process them to bring the result closer to reality I don’t consider that a waste of time.
And yes, I know I won’t be 100% precise.
Actually, our fixed price deals tend to be very short; rarely more than a month. Most of our deals are much longer and priced around improving metrics, usually accounting metrics (Aging Accounts Receivables, Profit Margins, Sales, Productivity etc.)
We do fixed price contracts when finding a metric isn’t worth the effort/cost. I won’t say these contracts aren’t important, but they are not why our customers are working with to us.
Pricing based around improving metrics aligns your incentives with your customers for both the long and the short term. Discussions about estimates, time lines and scope are really about risk and who has to bare its cost.
If you operate in an environment where the demand for skills is far greater than the supply, the person buying the skills pays for all the risk. For a long time, IT companies could get their customers to pay for the risk upfront and then if the estimates/scope/time lines/risks/etc were wrong, then the customer had the choice of either walking away and losing their previous investment or they could keep paying. Many software companies, consulting firms and consultants, myself included, made a lot of money in this environment.
Estimates are a “head I win, tails you lose” game. If my estimates are right, I make money. If my estimates are wrong, you pay more money. Forgive me if I’m cynical about people who claim estimates are a critical skill. How often do they go back and evaluate if their estimates are accurate?
There’s an old saying in Finance. “Forecasts tell you more about the forecaster than they do about the future.” Estimates are the same, they tell you more about the estimator then about how a project will go.
If you have a PhD in physics, you’re probably going to have some complex formula that no one other than you can understand. There’s another saying in Finance that’s becoming popular. “Beware of geeks bearing formulas.”
I’ll give you two big advantages of PDOOMA.
1. It’s funny.
2. They are my estimates. And my interests are aligned with my customers interests in seeing that we succeed, both long term and short term.
Andrew, you’re pretty lucky if you work on fixed price contracts only in small engagements. In my last few jobs we worked almost purely on fixed price. And it was on a few different markets (enterprises, mobile operators, banking). Each time we missed our estimates it was more a problem of us, not of the customer.
We had to add some work which wasn’t planned and we had to deal somehow with forfeits from the formal agreement (which usually meant more work).
From my perspective it’s definitively not “I win or you lose” type of scenario.
If you work in that kind of environment PMDOOMA may work for you. At least as far as someone else comes in and present a bit different approach.
Pawel, I think the global economic situation is changing the equation. Software and integration companies used to be able to play the “heads I win, tails you lose” game, but times they are a changing and customers are now in a much stronger position than they were a couple years ago.
For larger projects, think about changing your billing method. ROI projects with a small upfront fee shift the risks and relationship. You take on a bit more of the risk, but if you make the project succeed, you share in more of the gains.
It doesn’t work for every project and it doesn’t work if someone can game the metric. But it inherently leads to multi-year contracts and it aligns your interests closely with your customers interests.
It won’t work for every project, but I have found that just putting it out there as a business method changes working relationship.
Almost everyone doing consulting work, currently does so on a fixed fee, hourly basis. The environment in which that system worked (I believe) is starting to change and pain associate with estimating, scope management, requirements management etc. is symptomatic of the economic environment changing.
I don’t think ROI based pricing is the eventual solution, but I do think it is a stepping stone if companies are going to build long term relationships with consulting firms/outsourcing firms etc.
Fee + ROI pricing means companies are protected on the downside and consulting firms share on the upside.
I’d be interested to know your thoughts – though I’d rather discuss it through private email than a blog. (ameyer at go2incent dot com)