Humans are notoriously bad at estimating how long it will take to complete a task. Psychologists call it a planning fallacy and this was first discussed by Daniel Kahneman and Amos Tversky way back in 1977. All these years later and regardless of us knowing of the problem, it still exists much the same as back in 1977.
We have more historical data thanks to the expanding collection abilities and yet we still make the same error. That is just the first part of the problem. The second is we think we live in a perfect world where there won’t be any complications. Regardless of what historically happened, it won’t happen to us this time. We won’t need to take as long. This is called the optimism bias. Of course it’s good to have a healthy level of optimism, however, not when it is over emphasised and especially when planning.
Come up with a best case, worst case and most likely scenario.
Unfortunately it is not as simple as just adding hours, days, weeks or months to the project timeline. That adds in another human trait of working to the deadline. Meaning it takes exactly the same amount of time as estimated. Ever sat at work knowing you had to have something done by first thing the following day? Effectively before you leave for the day. I bet on the days where you have an essential appointment at 6pm that can’t be cancelled, you will get it done by 6pm. Days where you don’t have that appointment, the same task might take you though to 10pm or later.
Successful people who still find time for family, friends or other outside work commitments often set a time to finish work each day. They treat it as non-negotiable. That is their self imposed deadline. They make others aware of the deadline. I worked with a person who taught me this and and it changed the way I worked.
It can also work in the reverse. Treat the time before work exactly the same. Don’t fall for the trap of, “Oh well I can just come in early tomorrow morning and get it done before everyone else arrives.” As a morning person this was my go to strategy as far back as my student days. Up early to finish the assignment on the day it was due.
A way to minimise these problems is to adopt a similar philosophy as they do at Basecamp. When setting deadlines, they think in terms of budgets not estimates. If they have a feature they wish to add to their software product their deadline is based on a budget, not an estimate of time. It is not a case of looking at a feature and asking the developers how long it will take to complete. Remember, humans love to under-estimate the time and over-estimate ability.
If you are able to calculate the return on investment, this process helps to eliminate some projects that might appear viable, however, from a financial sense, they are not worthwhile. Perhaps a scaled down project is though. So allocating a time and associated cost helps to avoid going too far into the development process and then not wanting to scale back. Not changing course regardless of what the data tells you is a whole other psychological issue called continuation bias. The scope reduction can be difficult for some people to stomach. Just remember, the work will fill the allotted time.
There’s a few things you can do to help set the budget. Before speaking with the people who will be actually doing the work, study historical data. Ensure this is not done by the people working on the project. Break it down into smaller ranges or sub-projects so you can build in time for delays and not just add an arbitrary number at the end. To reduce the optimism bias, consider using three point optimism. Come up with a best case, worst case and most likely scenario.
Take it a step further and work on this when you are at your low point of the day. Studies have found people are most productive in the first couple of hours, so best to avoid dong it then. People tend to hit a wall around half way through their work day. After lunch or after what ever is the mid point of your way day is ideal.