Comments on: What’s the Use of Wild Guesses Instead of Proper Estimation? http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html Dealing with software projects in real life Fri, 10 Feb 2012 19:07:34 +0000 hourly 1 http://wordpress.org/?v=3.3 By: Andrew Meyer http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2172 Andrew Meyer Sat, 28 Mar 2009 18:21:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2172 Pawel, I think the global economic situation is changing the equation. Software and integration companies used to be able to play the "heads I win, tails you lose" game, but times they are a changing and customers are now in a much stronger position than they were a couple years ago.<br/><br/>For larger projects, think about changing your billing method. ROI projects with a small upfront fee shift the risks and relationship. You take on a bit more of the risk, but if you make the project succeed, you share in more of the gains.<br/><br/>It doesn't work for every project and it doesn't work if someone can game the metric. But it inherently leads to multi-year contracts and it aligns your interests closely with your customers interests.<br/><br/>It won't work for every project, but I have found that just putting it out there as a business method changes working relationship.<br/><br/>Almost everyone doing consulting work, currently does so on a fixed fee, hourly basis. The environment in which that system worked (I believe) is starting to change and pain associate with estimating, scope management, requirements management etc. is symptomatic of the economic environment changing.<br/><br/>I don't think ROI based pricing is the eventual solution, but I do think it is a stepping stone if companies are going to build long term relationships with consulting firms/outsourcing firms etc.<br/><br/>Fee + ROI pricing means companies are protected on the downside and consulting firms share on the upside.<br/><br/>I'd be interested to know your thoughts - though I'd rather discuss it through private email than a blog. (ameyer at go2incent dot com) Pawel, I think the global economic situation is changing the equation. Software and integration companies used to be able to play the “heads I win, tails you lose” game, but times they are a changing and customers are now in a much stronger position than they were a couple years ago.

For larger projects, think about changing your billing method. ROI projects with a small upfront fee shift the risks and relationship. You take on a bit more of the risk, but if you make the project succeed, you share in more of the gains.

It doesn’t work for every project and it doesn’t work if someone can game the metric. But it inherently leads to multi-year contracts and it aligns your interests closely with your customers interests.

It won’t work for every project, but I have found that just putting it out there as a business method changes working relationship.

Almost everyone doing consulting work, currently does so on a fixed fee, hourly basis. The environment in which that system worked (I believe) is starting to change and pain associate with estimating, scope management, requirements management etc. is symptomatic of the economic environment changing.

I don’t think ROI based pricing is the eventual solution, but I do think it is a stepping stone if companies are going to build long term relationships with consulting firms/outsourcing firms etc.

Fee + ROI pricing means companies are protected on the downside and consulting firms share on the upside.

I’d be interested to know your thoughts – though I’d rather discuss it through private email than a blog. (ameyer at go2incent dot com)

]]> By: Pawel Brodzinski http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2171 Pawel Brodzinski Sat, 28 Mar 2009 14:30:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2171 Andrew, you're pretty lucky if you work on fixed price contracts only in small engagements. In my last few jobs we worked almost purely on fixed price. And it was on a few different markets (enterprises, mobile operators, banking). Each time we missed our estimates it was more a problem of us, not of the customer.<br/><br/>We had to add some work which wasn't planned and we had to deal somehow with forfeits from the formal agreement (which usually meant more work).<br/><br/>From my perspective it's definitively not "I win or you lose" type of scenario.<br/><br/>If you work in that kind of environment PMDOOMA may work for you. At least as far as someone else comes in and present a bit different approach. Andrew, you’re pretty lucky if you work on fixed price contracts only in small engagements. In my last few jobs we worked almost purely on fixed price. And it was on a few different markets (enterprises, mobile operators, banking). Each time we missed our estimates it was more a problem of us, not of the customer.

We had to add some work which wasn’t planned and we had to deal somehow with forfeits from the formal agreement (which usually meant more work).

From my perspective it’s definitively not “I win or you lose” type of scenario.

If you work in that kind of environment PMDOOMA may work for you. At least as far as someone else comes in and present a bit different approach.

]]> By: Andrew Meyer http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2170 Andrew Meyer Fri, 27 Mar 2009 03:32:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2170 Actually, our fixed price deals tend to be very short; rarely more than a month. Most of our deals are much longer and priced around improving metrics, usually accounting metrics (Aging Accounts Receivables, Profit Margins, Sales, Productivity etc.) <br/><br/>We do fixed price contracts when finding a metric isn't worth the effort/cost. I won't say these contracts aren't important, but they are not why our customers are working with to us.<br/><br/>Pricing based around improving metrics aligns your incentives with your customers for both the long and the short term. Discussions about estimates, time lines and scope are really about risk and who has to bare its cost. <br/><br/>If you operate in an environment where the demand for skills is far greater than the supply, the person buying the skills pays for all the risk. For a long time, IT companies could get their customers to pay for the risk upfront and then if the estimates/scope/time lines/risks/etc were wrong, then the customer had the choice of either walking away and losing their previous investment or they could keep paying. Many software companies, consulting firms and consultants, myself included, made a lot of money in this environment.<br/><br/>Estimates are a "head I win, tails you lose" game. If my estimates are right, I make money. If my estimates are wrong, you pay more money. Forgive me if I'm cynical about people who claim estimates are a critical skill. How often do they go back and evaluate if their estimates are accurate?<br/><br/>There's an old saying in Finance. "Forecasts tell you more about the forecaster than they do about the future." Estimates are the same, they tell you more about the estimator then about how a project will go. <br/><br/>If you have a PhD in physics, you're probably going to have some complex formula that no one other than you can understand. There's another saying in Finance that's becoming popular. "Beware of geeks bearing formulas."<br/><br/>I'll give you two big advantages of PDOOMA.<br/>1. It's funny.<br/>2. They are my estimates. And my interests are aligned with my customers interests in seeing that we succeed, both long term and short term. Actually, our fixed price deals tend to be very short; rarely more than a month. Most of our deals are much longer and priced around improving metrics, usually accounting metrics (Aging Accounts Receivables, Profit Margins, Sales, Productivity etc.)

We do fixed price contracts when finding a metric isn’t worth the effort/cost. I won’t say these contracts aren’t important, but they are not why our customers are working with to us.

Pricing based around improving metrics aligns your incentives with your customers for both the long and the short term. Discussions about estimates, time lines and scope are really about risk and who has to bare its cost.

If you operate in an environment where the demand for skills is far greater than the supply, the person buying the skills pays for all the risk. For a long time, IT companies could get their customers to pay for the risk upfront and then if the estimates/scope/time lines/risks/etc were wrong, then the customer had the choice of either walking away and losing their previous investment or they could keep paying. Many software companies, consulting firms and consultants, myself included, made a lot of money in this environment.

Estimates are a “head I win, tails you lose” game. If my estimates are right, I make money. If my estimates are wrong, you pay more money. Forgive me if I’m cynical about people who claim estimates are a critical skill. How often do they go back and evaluate if their estimates are accurate?

There’s an old saying in Finance. “Forecasts tell you more about the forecaster than they do about the future.” Estimates are the same, they tell you more about the estimator then about how a project will go.

If you have a PhD in physics, you’re probably going to have some complex formula that no one other than you can understand. There’s another saying in Finance that’s becoming popular. “Beware of geeks bearing formulas.”

I’ll give you two big advantages of PDOOMA.
1. It’s funny.
2. They are my estimates. And my interests are aligned with my customers interests in seeing that we succeed, both long term and short term.

]]> By: Pawel Brodzinski http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2169 Pawel Brodzinski Wed, 25 Mar 2009 07:48:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2169 I'm not going to flame you. You say you don't know how much time some development would take. On the other hand you set up a deal, most likely with fixed fee. Now if you don't know how much it would take how do you the deal will be profitable?<br/><br/>Personally I don't expect to have estimates exactly on target. If the slip is about 10% or 20% in vast majority of cases that's fine for both a vendor and a customer. I would like to avoid situations when project is slipped by 100% or more.<br/><br/>When you look at a project which was planned for 4-5 months and it's already 20th month of a schedule something is terribly wrong. I've seen that and that was exactly a result of PMDOOMA approach. "<i>Wanna deadline? Here it goes.</i>".<br/><br/>If I have a tool which doesn't bring much additional hassle and it helps me either to improve initial estimates or post-process them to bring the result closer to reality I don't consider that a waste of time.<br/><br/>And yes, I know I won't be 100% precise. I’m not going to flame you. You say you don’t know how much time some development would take. On the other hand you set up a deal, most likely with fixed fee. Now if you don’t know how much it would take how do you the deal will be profitable?

Personally I don’t expect to have estimates exactly on target. If the slip is about 10% or 20% in vast majority of cases that’s fine for both a vendor and a customer. I would like to avoid situations when project is slipped by 100% or more.

When you look at a project which was planned for 4-5 months and it’s already 20th month of a schedule something is terribly wrong. I’ve seen that and that was exactly a result of PMDOOMA approach. “Wanna deadline? Here it goes.“.

If I have a tool which doesn’t bring much additional hassle and it helps me either to improve initial estimates or post-process them to bring the result closer to reality I don’t consider that a waste of time.

And yes, I know I won’t be 100% precise.

]]> By: Andrew Meyer http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2168 Andrew Meyer Tue, 24 Mar 2009 18:26:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2168 Pawel,<br/><br/>I hate to beat a dead horse, especially one where I got beaten up for making a joke about estimates.<br/><br/>However, I stubbornly stand by my belief that however much you dress them up with training as a physicist, PMI or any other methodology, estimates are nothing more than wild ass guesses. I do not believe there is any methodology that will produce estimates more accurate than PDOOMA.<br/><br/>Before I get flamed again, please consider this.<br/><br/>A study of 214 IT projects from 1998 to 2005 found that only 1 in 8 could be considered successful, meaning they met original time, cost and quality estimates.<br/>http://www.bcs.org/server.php?show=ConWebDoc.19584<br/><br/>Others may have been aware of this study previously, I just came across it today, which is why I'm venturing back into the dangerous land of reopening this debate.<br/><br/>These were large, well funded projects. It is reasonable to assume that the people working on and running them were competent, well trained and motivated to make their projects succeed. However, with a success record of 1 in 8, it is reasonable to question the estimates and methodologies use. <br/><br/><br/>The reason for my scorning estimating techniques and adopting PDOOMA is that I have been dragged through too many exercises in software estimation that quickly deteriorated and deviated so far from reality that we might as well have been discussing how many angels can dance on the head of a pin.<br/><br/>Why don't we just admit that we don't know how long something is going to take? <br/><br/>As a business owner who contracts with customers to deliver functionality. Whether my business is profitable and my customers happy really depends if I can propose a reasonable approach that is beneficial for both sides.<br/><br/>I don't know how long something's going to take. I don't know what system is going to have bugs in it no one knew about before hand. I don't know what companies I'm depending upon are going to get sucked into the black hole that is the current global economic crisis.<br/><br/>There are too many things I do not know and have no way of knowing that are going to affect my projects. One could spend huge amounts of time, energy and brain power trying to predict and estimate what's going to happen in the future, and if you use enough scientific sounding mumbo-jumbo, you might even convince someone that you know. But the fact is, you do not know.<br/><br/>So yes, PDOOMA was meant to be a joke and it is one I've used with several clients, because at least I'm honest enough to admit I don't know. That honest allows me to write contracts that neither suffocate my clients nor my company without wasting time on estimation exercises.<br/><br/>If you think you have a better, more accurate or more sophisticated system for estimating projects, I suggest looking at the controversy surrounding the Black-Scholes Model for estimating options. There are fewer variables to consider options and far more money and brain power have been applied to it. And guess what, it doesn't work.<br/><br/>As Nils Bohr famously said: Prediction is very difficult, especially if it's about the future.<br/>http://www.duke.edu/~rnau/411quote.htm Pawel,

I hate to beat a dead horse, especially one where I got beaten up for making a joke about estimates.

However, I stubbornly stand by my belief that however much you dress them up with training as a physicist, PMI or any other methodology, estimates are nothing more than wild ass guesses. I do not believe there is any methodology that will produce estimates more accurate than PDOOMA.

Before I get flamed again, please consider this.

A study of 214 IT projects from 1998 to 2005 found that only 1 in 8 could be considered successful, meaning they met original time, cost and quality estimates.
http://www.bcs.org/server.php?show=ConWebDoc.19584

Others may have been aware of this study previously, I just came across it today, which is why I’m venturing back into the dangerous land of reopening this debate.

These were large, well funded projects. It is reasonable to assume that the people working on and running them were competent, well trained and motivated to make their projects succeed. However, with a success record of 1 in 8, it is reasonable to question the estimates and methodologies use.

The reason for my scorning estimating techniques and adopting PDOOMA is that I have been dragged through too many exercises in software estimation that quickly deteriorated and deviated so far from reality that we might as well have been discussing how many angels can dance on the head of a pin.

Why don’t we just admit that we don’t know how long something is going to take?

As a business owner who contracts with customers to deliver functionality. Whether my business is profitable and my customers happy really depends if I can propose a reasonable approach that is beneficial for both sides.

I don’t know how long something’s going to take. I don’t know what system is going to have bugs in it no one knew about before hand. I don’t know what companies I’m depending upon are going to get sucked into the black hole that is the current global economic crisis.

There are too many things I do not know and have no way of knowing that are going to affect my projects. One could spend huge amounts of time, energy and brain power trying to predict and estimate what’s going to happen in the future, and if you use enough scientific sounding mumbo-jumbo, you might even convince someone that you know. But the fact is, you do not know.

So yes, PDOOMA was meant to be a joke and it is one I’ve used with several clients, because at least I’m honest enough to admit I don’t know. That honest allows me to write contracts that neither suffocate my clients nor my company without wasting time on estimation exercises.

If you think you have a better, more accurate or more sophisticated system for estimating projects, I suggest looking at the controversy surrounding the Black-Scholes Model for estimating options. There are fewer variables to consider options and far more money and brain power have been applied to it. And guess what, it doesn’t work.

As Nils Bohr famously said: Prediction is very difficult, especially if it’s about the future.
http://www.duke.edu/~rnau/411quote.htm

]]> By: SBL Software Solutions http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2167 SBL Software Solutions Thu, 19 Feb 2009 08:45:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2167 good article!<br/><br/>"Regards,<br/><a HREF="" REL="nofollow"> SBL - custom software </a><br/>http://www.sblsoftware.com" good article!

“Regards,
SBL – custom software
http://www.sblsoftware.com“

]]> By: Pawel Brodzinski http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2166 Pawel Brodzinski Tue, 17 Feb 2009 18:06:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2166 Glen,<br/><br/>When I was working on ERP projects (for small to medium companies) we've been pretty successful with estimates when implementations were close to standard. Expert judgement worked really well then.<br/><br/>Of course experts were using their knowledge about past project but it wasn't formalized in any way. People didn't think in SLOC or function points categories, at least not consciously, but their experience allowed to bring pretty exact estimate ranges.<br/><br/>On the other hand lack of formal methods of analysis brought some serious problems when it came to plan a couple of big, non-standard implementations which didn't aligned well with previous projects. Glen,

When I was working on ERP projects (for small to medium companies) we’ve been pretty successful with estimates when implementations were close to standard. Expert judgement worked really well then.

Of course experts were using their knowledge about past project but it wasn’t formalized in any way. People didn’t think in SLOC or function points categories, at least not consciously, but their experience allowed to bring pretty exact estimate ranges.

On the other hand lack of formal methods of analysis brought some serious problems when it came to plan a couple of big, non-standard implementations which didn’t aligned well with previous projects.

]]> By: galleman http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2165 galleman Tue, 17 Feb 2009 17:35:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2165 All,<br/>here's a sample of the depth of the issue<br/>http://www.pmforum.org/library/papers/2007/PDFs/Galorath-4-07.pdf from Dan Galortah. <br/>This is one of many approaches to software sizing problem.<br/>The notion that agile does not have to dela with this is coreect ONLY if you consider the project Level of Effort within a Time Boces schedule.<br/>We've found the best use of agile is in the maintence or extension of existing software code bases. If you go looking at example you see most are like this. David Anderson's examples, PayPal and Google are all extensions of operating platforms.<br/>For "green field" development, some bounds are needed for cost and schedule and therefore a credible estimate - with upper and lower bounds and a statistical correlation between duration and effort - is needed. All,
here’s a sample of the depth of the issue
http://www.pmforum.org/library/papers/2007/PDFs/Galorath-4-07.pdf from Dan Galortah.
This is one of many approaches to software sizing problem.
The notion that agile does not have to dela with this is coreect ONLY if you consider the project Level of Effort within a Time Boces schedule.
We’ve found the best use of agile is in the maintence or extension of existing software code bases. If you go looking at example you see most are like this. David Anderson’s examples, PayPal and Google are all extensions of operating platforms.
For “green field” development, some bounds are needed for cost and schedule and therefore a credible estimate – with upper and lower bounds and a statistical correlation between duration and effort – is needed.

]]> By: galleman http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2164 galleman Tue, 17 Feb 2009 17:28:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2164 Pawel,<br/>Yes "qualified" and "calibrated" estimates are the starting point. But there is a fundamental issues (theological actually). In Bayesian statistics, if you do not know the underlying distribution of the random variables in your populaiton, the confidence interval on the sample is 50/50. That's pretty poor odds when "managing" a project.<br/>This is a complex problem and no addressing the issues is the source of much grief.<br/>We've worked on many ERP projects where the kind of guessing we speak of here is common. The result is usually an 100% overrun. No an issue is the project is $10,000 but when it is $100M then we've got a real problem.<br/>One recent book is "Software Sizing, Estimation and Risk Management” Dan Galorath and Michael Evans, 2007. <br/>This may be a useful source for complex software estimating problem along with McConnel's previous book. Pawel,
Yes “qualified” and “calibrated” estimates are the starting point. But there is a fundamental issues (theological actually). In Bayesian statistics, if you do not know the underlying distribution of the random variables in your populaiton, the confidence interval on the sample is 50/50. That’s pretty poor odds when “managing” a project.
This is a complex problem and no addressing the issues is the source of much grief.
We’ve worked on many ERP projects where the kind of guessing we speak of here is common. The result is usually an 100% overrun. No an issue is the project is $10,000 but when it is $100M then we’ve got a real problem.
One recent book is “Software Sizing, Estimation and Risk Management” Dan Galorath and Michael Evans, 2007.
This may be a useful source for complex software estimating problem along with McConnel’s previous book.

]]> By: Pawel Brodzinski http://blog.brodzinski.com/2009/02/whats-use-of-wild-guesses-instead-of.html#comment-2163 Pawel Brodzinski Tue, 17 Feb 2009 10:20:00 +0000 http://blog.brodzinski.com/2009/02/what%e2%80%99s-the-use-of-wild-guesses-instead-of-proper-estimation.html#comment-2163 Schalk,<br/><br/>The trick is there aren't many salespeople with as reasonable approach as yours. I've seen so many times when a wild guess was taken as reliable date and was treated as commitment. I've seen so many times when salespeople who were given a range of dates have taken the earlier and instantly forgot the other one.<br/><br/>Of course it depends on people but I worked with just a few salespeople who understood what is rough estimate, what is reliable schedule and saw a difference between these two. Schalk,

The trick is there aren’t many salespeople with as reasonable approach as yours. I’ve seen so many times when a wild guess was taken as reliable date and was treated as commitment. I’ve seen so many times when salespeople who were given a range of dates have taken the earlier and instantly forgot the other one.

Of course it depends on people but I worked with just a few salespeople who understood what is rough estimate, what is reliable schedule and saw a difference between these two.

]]>