On Failure

Even the best organizations fail sometimes. Here's how.

This is my first post for “The Internal”, and I want start with something we all try to avoid: Failure.

Even the smartest people and teams from time to time build products with mediocre impact. Even the best organizations fail sometimes. It’s normal. A company where no failure happens can’t be very agile, and it likely doesn’t innovate much.

After having seen hundreds of failures over the past 20 years, generally speaking the two biggest categories for challenges in scaled software development seem to be (servant) leaders who do not really lead and self-organizing people doing whatever the f**k they want. Well actually, it’s a bit more nuanced than that. First of all, there’s rarely ever just one reason for a project’s failure. It’s usually a combination of a few things. Looking back at them, they often seem to form a logical lead-up to the resulting failure. But when you’re in the trenches, it’s quite easy to miss them. Second, “who” usually is a lot less interesting than “how” and “why”, especially when you are fortunate enough to work with extremely smart and talented people (I consider myself very lucky here).

Here are all the reasons for failing projects and products I’ve personally come across over the years. It’s pulled from a text file I had sitting around on my Desktop for roughly 10 years now, that I kept adding to from time to time. So most items on this list have played a role in some failure at some point in our history at sipgate, but do not any more (a few may still be “active”, but that’s not the point). So when I say “we”, take it as “people in software companies”. I’m publishing this list of patterns now because I hope it will help others avoid some of the failures we encountered.

The List

Missing Single Product Mindset

Especially for larger products, one issue is failing to conceive our software as a single product rather than as a set of different projects or features. The quality of decisions erodes as the complexity and scale of the product increases.

Overvaluing Consensus

Unanimous agreement can lead to very bad decisions. Too much agreement tends to signal systematic errors in the decision making process. For example, we sometimes value consensus over truth. Although the thing that will make us the most money may stare us right in the eye, we are unable to build it. We discuss and discuss, in workshop after workshop, and elaborate on the things we can get consensus on. Our plans do get more detailed over time - but in some cases we never actually build anything that finds its way into production.

Avoidance of important issues

The amount of time we spend discussing an item can be inversely proportional to the importance of that issue. Trivial matters are discussed at length. Complicated, costly or difficult decisions are kept very short, or are being avoided entirely. Where things are uncertain or we don’t know the answers, this can slow down development, or make us build something that doesn’t fulfill the needs of the business. Our efforts fall short, likely because we avoid exposing our lack of understanding to others.

Prioritisation trap

We sometimes obsess too much about prioritizing our future efforts and don’t even get to the point where we try to validate if any of these ideas are any good. When we discuss them, they are so broad, they can mean anything. Instead of validating and invalidating concrete ideas, we resort to a decision making pattern that is based on if people feel confident “it will work”. The result can be a roadmap of untested and unclear features.

Powerpoint fallacy

To get your project approved, it is tempting to attach a large economic value to a potential user, and multiply it by a large number of potential users. Obviously, these numbers are completely made up, but they do look great on a Powerpoint slide. Some decisions at the portfolio level are based on flimsy economic justifications, and if they are, I’ve never witnessed them pan out. Other red flags for powerpoint fallacy are the use of a lot of buzzwords (“artificial intelligence”, “blockchain”), a claim the product “is for everyone”, a slide explaining we just need to convince a tiny fraction of a gigantic market (“we just need to convince 1000 users out of 100 million”), or an inability to describe a feature so someone without technical knowledge understands.

Not enough leadership buy-in

When an idea is shiny and new to someone, they are likely to proceed with great speed. That speed can wane off, as other things become more interesting. When leadership moves on to different problems, that can feel like a lack of buy-in for the initial idea, regardless of its value to the company. It could still be the most valuable idea in the history of the company, but without leadership demonstrating they truly care about it on a daily basis, to everyone else it can feel like a dud. Be very aware when leadership just “supports something” - there’s a big chance it will fail.

Missed the planning sweet spot

In retrospect, the best opportunities in terms of value creation were often the ones that came with a more detailed plan. But because there were tiny details in them someone would object to, or the presentations contained questions and showed complexity, some of these opportunities made little sense to the group making the decision at the time we discussed them. Occasionally, until it was too late to profit from them. The better an idea had been prepared at the time we discussed it on the portfolio level, the more likely it gets shot down.

To work around this problem, someone might leave out the part where they prepare a detailed presentation of their ideas. For example, someone might present their plan for an entire year on three post-it-notes. When pressed for more details and validation, they might answer they feel confident the team will come up with the optimal solution, and get their “plan” approved.

There’s too much planning, and there is too little. Hitting the sweet spot is difficult.

Shortsightedness & Low Agency 

We craft our product strategies and make product decisions based on what we know we can easily execute today. We prioritize things that seem easy to get approval for.

This can be taken to the extreme, when we do not even take the time to thoroughly understand the problem, domain or competitors just because we feel we have to stay in constant motion. We just start building the easiest tiny thing we can find. Sometimes we get trapped in insignificance, and churn out low level iterations for an astonishingly long period of time.

No accountability

Without clear lines of accountability, an Agile organization can produce redundant communication, and very little progress. We sometimes do not bother to define ownership, for example because we don’t know which person holds specific knowledge, or who would give a project the most leverage. At other times we avoid assigning accountability, fearing it would add hierarchy.

Low complexity fallacy

Humans naturally prefer solutions that appear less complex. The harder something is to do, the fewer people are willing to do it. Sometimes we make a decision without having done the necessary research, and just stick with something that feels easy to start with.

Things that seem complex are always a great learning opportunity. But sometimes we are not ready to learn.

High complexity fallacy

We overlook ideas that appear too simple to us. We search for better ways, because we assume a simple solution can’t be the best one. For example, when we create a new feature, we design it in a way that prompts the user for a large number of parameters during setup, instead of suggesting the best way for the most common task at hand.

Lean Startup fallacy

We base our decisions on misunderstood Lean Startup bon mots, the most prominent being the Minimum Viable Product (MVP). Contrary to what the word “viable” suggests, it is not enough for a product to merely survive, it needs to be extraordinary in some fundamental way. When the product we built isn’t better than the competitors product in at least one regard, customers show little interest.

Reinventing the wheel

We try to avoid code another team has created in the past, simply because it annoys us. In a multi-product environment the same can be true for entire concepts for features another team has worked on. We dramatically underestimate the effort it takes to come up with something great in a creative environment. So much so, that we sometimes don’t even look at the previous work. And if we do, we often discard it. Either it lacks functionality for our purpose, or we find some little difference in a requirement and therefore believe it to be an impossible solution to our problem. We falsely believe only a completely novel concept would enable us to deliver value.

New architecture thinking

In every successful larger company, a surprisingly small number of architecture decisions from the early days are responsible for the vast majority of value created over time. These are also the decisions that are being questioned the most.

We often falsely believe only a move to a new architecture will enable us to ship features faster.

For example, a team reports they encountered technical debt and just had to start from scratch on their favorite new programming language and framework. Or since a well-known public API exists, it should replace some of our legacy code, and thereby save us some time down the road. Or if a service hasn’t been touched by anyone in quite a while, and since we don’t understand what it was made for, we think it’s probably easier to just build a second system from scratch.

Because we underestimate the complexity of the problems our legacy code solves (the code that is actually generating money), we tend to prefer new architecture. But without understanding the full scope of our legacy code, we can’t really claim that new architecture will be easier. The truth is, we often simply don’t know the long term effects of new architecture, because we haven’t put in the effort to fully understand all the “old” problems it would eventually have to solve over the long term. When our decisions are skewed by new architecture thinking, we can get very technical up to the point that we forget we were originally tasked with building or enhancing a product.

Compartmentalization

We tend to overvalue the differences of user groups, and do not focus enough on what unites them. This leads to a compartmentalization of our product in the sense that we build differences into features for each group. Taken to the extreme, this can lead to multiple products that are hard to tell apart as a customer, but with hundreds of tiny differences built in. This added complexity leads to a lot of confusion. It dilutes the core value proposition of our positioning instead of doubling down on it.

Role fungibility fallacy

Instead of making the effort to identify the best people for a project, we staff with whoever is available right now, whom we personally know, or with people picked from teams that voiced the least objections to the latest reorg. We tend to see people that hold the same role as fungible (i.e. Product Owners, Designers, Engineers). Projects fail simply because of lack of domain expertise, talent, and focus.

Too much technology

Being the nerds we are, we tend to focus too much on technology and not enough on value. Our backlogs sometimes look a lot like feature wish lists. Combined with a tendency to seek the lowest effort solution we churn out a stream of half-assed features no one actually wants.

Sunk cost fallacy

Because we are too attached to the time and money we have already invested, we sometimes base today’s product decisions on past product execution. We do not want our past efforts to have been for nothing. Although no-one buys our product, we might throw more people at it, and build more and more features.

Not being vocal enough

We build something and then forget to tell any potential customers about it. We tell ourselves we’re missing “a marketing person” on the team or we just haven’t gotten around to writing text for a great website yet. Sometimes, building a feature and making sure customers know about it are split into two different projects. Work on the latter may start only months after the technical implementation, or never.

Too much Process

Sometimes artificial constraints, perceived timelines, compromises, bureaucracy, and more get in the way of creating good products. A team needs to be able to say no to these things. Too much process often stands in the way of building a successful product the team would want to use themselves.

Too much focus on “done”

Once a team finishes a project, they immediately move to the next. The next one is based on a completely unrelated idea and technology. We forget to act on feedback, and we do not iteratively improve what we’ve built. After a project is finished, a team has little to none contact with customer service, and sales.

Project mindset

Approaching problems with a “project mindset” instead of a “product mindset”, we optimize for the optimal utilization of our teams and our engineers on the portfolio level, not for the optimal long term outcome. We justify this by telling ourselves it’s the only way to avoid “local optimization”. Because the teams aren’t taking responsibility for an outcome, but are merely there to deliver a project (and quickly move on to something else), the results are meager. There’s just no way engineers, designers and product owners can easily switch between projects in a complex software development environment. By trying to scale through staffing and project juggling, we risk losing our most important success factor from the earlier days: ownership over the customer outcome.

Too much Leadership

Product decisions are being made based on what people think is going to confirm the beliefs held by those in a position of authority. Or worse, founders direct the work themselves, instead of directing the passion. Sometimes leadership on multiple levels can fall back into “industrial age tendencies”, pursuing efficiency, becoming too involved in the details. Thereby we diminish team performance and obstruct meaningful progress.

Lack of Leadership

Especially when multiple teams work on a single product, a lack of properly executed Product Leadership can have dire consequences: Our Sales and marketing may become disconnected from the strategy and product development. One team removes a feature that another team just implemented a few months ago. The overhead of communication and coordination swamps the impact of individuals, leading to mediocre results. Decisions take forever and a lack of focus leaves many people clueless about the product’s future path. Lack of leadership often propagates to other problems, like insufficient project planning, continuous changes in the requirements, and backlog scope creep.

Not updating our positioning

Positioning is all about how you market and sell a product today, while Strategy is a clearly defined path towards a future state of the company. As a product matures and a market changes, the positioning of a product should be updated constantly (at least twice per year). When we don’t put in this effort, our marketing and sales pitches fall short. A potential customer’s curiosity can’t be piqued with last year’s value propositions (if there is competition and the market is moving fast).

Even worse, sometimes our Strategy spills over into Positioning to fill the void. We lose deals when our marketing and sales pitches confuse customers with value propositions that don’t exist yet. Customers will simply delay their purchasing decisions if we can’t prove there is value today.

Avoidance of negative feedback

Things can get confusing when strategic work takes place on multiple flight levels within the same organization. From time to time there will be stretches of ambiguity before a clear path presents itself. Sometimes leaders try to skip this stretch by labeling decisions as “strategic”, and asking everyone not to question them. When negative feedback is shut off like that, it’s usually either because leaders don’t want to talk about them any more (good, if they have been discussed at length and in broad detail already), or because they don’t have any good supporting arguments (bad, for obvious reasons).

No measurement

Some teams do not measure the impact of their work. Even if they do, they might measure the wrong stuff, for example focussing on vanity metrics like new signups, NPS, or aggregated users. They have no idea if the efforts they put in actually lead to value creation. Instead of maximizing customer outcome and trying to capture as much of it as possible, a team might celebrate everything they touched in the bi-weekly all-hands demos, regardless of impact.  

Too much shuffling

Teams, projects and people get shuffled around too often. From an individual’s perspective there is little consistency in the mission they try to accomplish. It just changes too often. Each time someone switches to a new project, a lot of relearning has to take place.

No glory in refactoring

Tackling technical debt is unattractive because it has low visibility within the company. Sometimes the refactoring of code is being skipped entirely. Huge parts of a system could be left unchanged for years. It is way more attractive to build the next new shiny feature, and to just implement it wherever the team feels most comfortable, than to look for the optimal point of implementation, and to improve maintainability and extensibility of what already exists in the codebase along the way. 

We claim the “new code” that gets shipped is of higher quality than ever before. That always seems to be true, but it might be a very local view. Taking the whole platform into account, it might be getting more tangled and scattered across all over the place.

Leadership can be a contributor to the lack of refactoring, if their choice of words leaves everyone else under the impression that code refactoring of existing features is a somewhat less valuable activity than building new features. Here’s a hint: If your legacy code is generating all of this month’s revenue, it clearly is not less valuable.

Lack of documentation

Experienced developers plan their coding. They put their understanding and the intended flow on paper first, they try to visualize it end to end. Only after they are clear about the changes, they start implementing things. The thing is: With a large codebase, it becomes impossible to know all the necessary details for this to work. You need a minimum of documentation and domain experts on standby helping people navigate through dependencies. But sometimes no documentation exists, or no domain experts are known - or at least knowledge about them might be scattered.

As a result, planning meetings can rely on a lot of guesswork, and implementation can start before the team has grasped the whole flow of all use cases involved. Instead of thinking hard about the simplest technical solution for all relevant use cases, we just add in another exceptions just for our new use case, and treat the rest of the system like a black box.

That’s all for now

So that’s the list. Please handle with care, since symptoms and causes are easily conflated. And sometimes it can be very difficult to tell if you are successful because of something you did or in spite of it. I recommend quickly skimming through the list whenever your next project fails. That’s how I use it.

I’m sure there were a lot more patterns for failure I didn’t see. So I’ll keep adding to this list in the future. The text file on my desktop can retire now. Instead, expect updates here.