Why small projects succeed and big ones don’t
By Nick Butler
9 June 2021
Research shows that small projects are much more likely to succeed than big ones.
Analysing a database of over 50,000 in-depth project profiles, the 2018 Standish Group report shows that small software projects succeed 4 times more often than large ones.
Small projects are 4x more likely to succeed than large ones
To help more people deliver more successful projects, we look at how small projects reduce the risk and impact of failure and how to deliver big impacts through small projects.
Research on project size and risk
Research shows that small projects reduce:
- the likelihood your project fails
- the impact if your project does fail.
That’s because big projects fail more often, and frequently fail catastrophically.
Bigger projects fail more often
The Standish Group report shows that big projects succeed less than 15% of the time. That means that over 85% of big projects fail.
Over 85% of big projects fail
Bigger projects fail bigger
The bigger the project, the bigger the failure is likely to be.
Lean expert Don Reinertsen found that the longer the project, the more the delivery dates slip. Worse still, he found that bigger projects increase slippage exponentially. One organisation told him that doubling project duration caused 16 times the slippage.
It gets worse. An Oxford University study written up in the Harvard Business Review found that 17% of large IT projects fail catastrophically — with a cost overrun of 200%, on average, and a schedule overrun of almost 70%.
Big projects are more likely to fail catastrophically
The article lists companies and careers brought down by these “Black Swan” failures. As well as being home to flesh-and-blood black swans, New Zealand has its own history of giant software disasters, with Novopay and INCIS now infamous examples.
Why do people attempt big projects?
There are a number of interconnected reasons people keep on initiating big projects despite the risks.
- Many people are unaware of the risks.
- Others don’t realise there are alternatives.
- People have big ambitions, and they believe bigger projects deliver bigger benefits.
- They have big issues, such as replacing legacy monoliths.
- They can’t see how they can make their particular problem or project any smaller.
- Small projects get big as requirements balloon and scope creeps.
- The sunk-cost fallacy sees organisations continue with projects that are failing.
- People feel one big change will be easier for users than a series of smaller ones.
- They think they’ll get economies of scale.
- They want to futureproof the software.
- Big ambitions attract big budgets — paradoxically, it can be easier to get budget for big projects because they are seen as transformative and so get buy-in from the C-Suite.
In fact, research shows that our brains are wired to solve problems by adding not subtracting, according to this recent paper published in Nature.
If any of these sound like you, read on to see how you can minimise risk by minimising the size of your project.
How to have a big impact by keeping projects small
You can still deliver big transformations while keeping projects small. Here’s how:
- Make keeping the project small an explicit goal.
- Prioritise regularly and ruthlessly.
- Deliver working software early and often.
Make keeping the project small an explicit goal
Keep a constant and explicit focus on minimising the project’s size. From day one, ensure that all team members and stakeholders know that you aim to maximise your return on investment by minimising the size of the project.
Each time you make a decision about the project, explicitly ask which option will best help keep the project small.
Make keeping the project small an explicit goal at each stage in the project’s lifecycle, from initiation and planning, through budgeting and procurement, discovery and requirements gathering, development, testing, integration, deployment, close-out and maintenance.
Within each phase, you’ll have individual pieces of work. Keep these small too. Minimise the scope of your use cases, user stories or however you specify your requirements.
Prioritise regularly and ruthlessly
Throughout the project lifecycle, ruthlessly prioritise the work that delivers the most value.
This goes hand in hand with keeping the project small. At every stage, you want to identify the work that creates the most value for the least effort.
Work is valuable if it:
- delivers benefits to your users
- manages risks
- cuts costs
- produces learnings or tests assumptions about your users, solution options and your operating environment or marketplace.
Identifying the priorities will be a key part of your initial discovery work and requirements gathering. We recommend running a discovery workshop with key stakeholders to gain a shared understanding of your priority user groups, priority outcomes for these users, and the features and functionality that can deliver these outcomes. These priorities will often change, as you gather your learnings and test your assumptions.
Deliver working software early and often
You learn how well your software is achieving project outcomes by putting it in front of your customers. To do this you need to iteratively develop and test working software.
You’re aiming to deliver the smallest, simplest useful implementation of your top priority outcome or feature by the end of each iteration.
Keep your iterations under a month. This gives you fast feedback, so your project can’t go too far off track. It also helps keep each iteration small, because it limits the amount of work you can expect to complete.
Minimising each iteration
To do this you need to prioritise both:
- what you deliver, and
- who you deliver it for.
Focus on your top priority user group and, for that user group, minimise things like:
- steps in the workflow
- options available to the users
- automation of manual processes
- interface refinement
One way you don’t break down your iterations is by building in horizontal architectural layers (e.g. starting with a database, then developing the logic, then the interface). In order to deliver working software, you need to deliver a vertical slice through all layers.
What happens when you have big goals? Breaking big projects into prioritised phases
Often organisations and teams have big changes they need to bring about. You might have to implement a transformative new initiative or replace a monolithic legacy system for example.
In this case, map out the high-level outcomes for the full change that you need to make, prioritise these, then tackle them in phases.
Only do detailed planning and requirements-gathering needed for the current phase. Keeping a constrained focus on each phase enables effective monitoring and control. Outcomes are clearer, it’s easier to determine user needs, and you can regularly check whether you are meeting these user needs and business outcomes.
You can then base later phases on what you’ve learned about your users, product and the operating environment from that first phase. This means you’re using empirical data not guesswork to guide your later phases.
Effectively you’re running a series of linked mini-projects, with close-out of each completed phase driving the initiation of the next.
Some techniques for breaking big projects into prioritised phases
You’ll also want to:
Develop the minimum viable product (MVP) for each phase: Identify the smallest, cohesive, standalone solution that lets customers complete enough of their goals that they’ll choose to use the product or service.
Minimise impact on existing users: Beta test the MVP with a subset of your users, and, if you’re replacing an existing system, consider keeping the existing system in operation until you have a workable replacement.
Build with integration in mind: Break monoliths into loosely coupled chunks via microservices and APIs.
Pilot new approaches: The more novel your approach, the sooner you need to find out if it works.
Keep stakeholders informed and engaged: Demo your working software to your key stakeholders at the end of each iteration.
Collaborate: The people guiding the project and the developers and designers building the product should work together as a single team, with a shared vision, goals and culture.
Why small projects succeed more that big ones
Small projects reduce:
- the likelihood your project fails
- the impact if your project does fail.
They do this through:
- better monitoring and control
- better communication
- ongoing process improvement
- fewer overloads
- lower stakes.
Better monitoring and control
Monitoring and control is crucial to project success.
Large projects have lots of moving parts. The interconnection of these parts makes it much harder to monitor for progress and problems. This in turn makes it much harder to identify causes when you need to fix problems. In short, big projects reduce transparency and accountability.
Keeping projects small at all stages and scales lets you clearly define your requirements and means you get fast feedback on:
- user outcomes achieved and features delivered
- any bugs, usability or performance issues
- any problems within the team.
It also stops you wasting money on unneeded features. Analytics company Pendo found that 80% of features in the average software product are rarely or never used. Check out their Feature adoption report (PDF) to learn more.
Smaller projects need smaller teams which makes communication easier. It’s also easier to collocate the team. This means you can take advantage of the real-time feedback you get from face-to-face communication.
Delivering a number of small projects increases your team’s expertise at each stage. It also means it makes economic sense to invest in automating processes such as testing and integration.
As large projects progress they cause periodic overloads. Don Reinertsen compares it to a restaurant. If 100 customers arrive at once your staff won’t cope. You’ll get queues, delays and dissatisfied customers. But if diners arrive in smaller groups, you’ll have enough resource to complete each order in good time.
The same applies to software development. Big projects overload teams as the project moves through each stage.
Smaller projects inherently reduce risk by cutting the investment at stake. If one phase fails, you’ve still got the return on investment from the earlier phases.
Big projects fail significantly more often. Worse still, when they fail they often do so catastrophically. They’re just not worth the risk.
Luckily, you can still have a big impact with small projects. Developing and testing valuable working software in short iterations delivers compound impacts. Not only do you get value to your users sooner, you get the information about your users, product and operating environment that you need to make evidence-based decisions on your next iteration or your next phase.
When it comes to software development projects, less is definitely more.
Contact us to learn more
If you’d like help keeping an upcoming project small, contact James on +64 4 939 0062 or at [email protected].
Don Reinertsen — The principles of product development flow
Vitality Chicago Chaos report data summary