The Standish Group recently published their 2015 Chaos Report, which evaluated data gathered on 50,000 projects across the world. While the report showed that agile projects are more successful than waterfall projects, it also revealed that extremely large projects of either type consistently fail to meet the redefined project success criteria of onTime, onBudget with a satisfactory result.
Estimation is a small part of the effort needed to deliver a successful project, but it produces 2 out of 3 data points used to measure success. Relative or analogous estimation techniques like StoryPoints and T-shirt sizes help to simplify the effort but fail to consider that people just aren't good at determining how much we can get done in a specific time period. I see evidence with the people I work with and even within my role as a coach. A really difficult problem, like finding a JIRA workflow that will meet the needs of 2 organizations who have used 4 different agile tools for years has taken much more time than I imagined. Building a complex system is no different.
That makes me wonder why we bother to estimate at all? It does make sense if the organization needs an idea of the cost to deliver versus the expected revenue to decide whether they should invest, but if estimating a large project keeps us from quickly drawing revenue on those features and could also lead to an unsuccessful project, could it be that it's time for a change?
How could your organization benefit if you divide those large waterfall projects into small agile ones, spend your effort defining and delivering them as quickly as possible, evaluate the results, and then make that product or feature better or deliver something else in your next small project?
If you believe the Chaos Report data, it could help you increase your project success from 3% to 58%.
Source data: http://www.infoq.com/articles/standish-chaos-2015