TLDR: Estimates should be used as a vital sign for the health of the project, not as a deadline.

When starting a new project, one of the first questions that executives and senior managers always ask is, “How long is this going to take?” 

Usually, that question is trickled down through the team’s hierarchy to the engineers tasked with building the thing before the estimate is then bubbled back up to the stakeholders. The stakeholders then treat it like a development schedule and use it to set a rigid deadline. By the end of the project, the estimate is, more often than not, wildly inaccurate and the team embarks on a journey of self-reflection to figure out what went wrong.

Why do we underestimate how long projects will take?

Every team building a new product runs into this issue at some point in the product development process, so why does it still occur so frequently? 

Naivety

One problem with the initial estimate is they’re done when you know the least about the project, its trajectory, and what hurdles you might run into. 

That’s like trying to estimate how long it will take a tree to reach its full height when it’s only a seedling. Things that you can’t predict might affect the growth of that tree (wildfires, pests, drought, disease, thermonuclear war, etc.), so it’s best to use that estimate as a fluid target that could (and should) change when new information about the project becomes available to you.

Hopeless Optimism

When you ask a kid what they want to be when they grow up, they’re going to give you their pipe dream job. Most people like to plan for only the best-case scenario. Engineers are no different.

Software engineers are usually quite accurate at estimating the best-case scenario for how long a project should take. 

The problem is that unforeseen issues that add to the overall development time aren’t usually considered for that initial project estimate. Some examples of those issues might be 

• Bugs
• Dependencies on the schedules of 3rd parties
• Changing project requirements
• Researching and learning a new technology
• Developing a new technology
• Team member turnover and holidays/vacations.

How can you estimate software development more accurately?

Estimating accurately is incredibly hard for a reason: you’re trying to predict the future with limited information. There simply isn't a reliable formula for development. However, here are some things we routinely do to get as close as possible:

More discovery time will lead to a better-defined problem

Ironically, the best way to improve the accuracy of a development timeline is to spend more time. Of course, not just time, but valuable time via a comprehensive discovery or architecture sprint. 

Understand and test the perceived problem against technology restraints and user feedback.

To identify problems, interview current and prospective users, conduct research, and look at what already exists in the market. To map solutions, you’ll need to consider various factors, including market opportunity, cost to build, risks, time to market, and value. 

On top of these factors, it can be hard for engineering teams to objectively assess concepts because teams tend to fall in love with their ideas. To know what to build, an outside perspective is always helpful. 

If you don’t have the luxury of extended discovery time, here are some slightly more clunky ways to create estimates: 

High, middle, and low 

It can be useful to come up with estimates for best-case, worst-case, and most-likely scenarios. This is helpful in a few ways:

1. It gets you actively thinking about issues that might derail a project, and helps you to consider how likely it is that they are to occur.
2. When the project is complete, you can look back at these estimates to see which was the most accurate, giving you some hard data to use for your next estimate. 

Double your estimate

Yes, we’re serious. Come up with a best-case estimate and then multiply that number by two to account for all of the hiccups. If the project includes a lot of features or technologies that your team or vendor does not have experience with, multiply by three. Estimating this way, more often than not, comes close to how long the project will actually take.

What if you have a hard deadline?

For most projects, you have a list of features that need to be completed, and your engineering team estimates how long it will take to complete them all. 

But sometimes, you’re instead given an immovable deadline. In those cases, it’s a matter of determining how many features your team can complete in a given amount of time. The same rules for estimating apply to this scenario, but you’ll estimate each individual feature and then implement them in order of highest priority until you’ve reached your time budget.

If you have a hard deadline and a hard list of features that need to be completed, make an estimate for the list and be ruthless about feasibility. If it looks challenging, your team and stakeholders need to have honest conversations (as soon as possible) about where time can be gained:

• Can you add more designers and developers?
• Can any of the features be simplified?
• Can you forgo manually building any features by pulling in 3rd party SDKs?

Don’t become too attached to your estimate 

Estimates are most valuable when used as a vital sign for the overall health of the project. You shouldn’t treat them as a one-and-done way to set an immovable deadline, especially at the beginning of the project when you know the least. That will only cause tension and resentment between the engineers and the stakeholders when things invariably don’t go according to plan.

At the end of the day, estimates are just a snapshot in time. When you gain more information about the project, you should be using that to reevaluate your estimate. Your estimate should be continually updated to be a reflection of reality. Furthermore, you should be recording that information for use with future estimates. That way, you won’t have to guess, you’ll know.