Estimation Using Probability

Earlier this year, I read Forecasting by Combining Expert Opinion, which presented a method for using probability to generate a forecast based on inputs from multiple expert parties. To do this, the author created a triangular distribution for each expert based on that person's minimum, maximum, and most-likely estimates. They then ran a Monte Carlo simulation to arrive at the final forecast in the form of a probability distribution.

Source: http://blog.revolutionanalytics.com/2014/01/forecasting-by-combining-expert-opinion.html

I thought this was a pretty smart and elegant way to arrive at a forecast, and started thinking of other areas where I've seen forecasts--project effort estimates, revenue forecasts, etc.--use much simpler means of prediction. Sometimes these are accurate, sometimes not at all.

In the world of agile software development, teams estimating the effort required to deliver product scope often use multi-point estimates as an input to their project-level estimate. For each story in their backlog, the team will provide an Aggressive But Possible (ABP) estimate, representing the effort estimate they expect to hit 50% of the time, and a Highly Probably (HP) estimate, representing an estimate they will hit 90% of the time.

Teams will then use any of a variety of methods to arrive at the product estimate; typically they will sum the ABP and then add buffer in the form of half the difference between ABP and HP, or the root sum of squares of the difference between the story-level ABP and HP.

A number of different resources point to the probability density function of agile estimates showing a right skewed normal distribution, like the one below from the Kanban Way. The ABP is the mode of the distribution, with the HP following further to the right on the curve.

Source: http://www.kanbanway.com/on-estimating-project-tasks

So now, I'm wondering about a couple things:

1. How would a Monte Carlo simulation of project effort based on a distribution like the one above compare to the project estimates based on simpler estimation techniques?
2. How would the estimate from a technique like the one I referenced at the beginning of the post compare to that coming out of a Wideband Delphi-based estimate?

I will note that I am a big proponent of Wideband Delphi, as I have used it with great success on a number of projects. The transfer of knowledge that comes during the estimation discussions can be extremely valuable. But that doesn't mean it wouldn't be fun to compare outputs of the two processes.