Default blind

September 2024

Imagine you work at a paper store:

  1. One day, you observe there are a lot more customers in the store, mostly Chinese, and they are all buying red paper. Some of them ask for red envelopes, which you don't have.
  2. Eventually you ask them, what are the envelopes for? "Honboa?! Chinese New Year gifts of course!". Trying to orient yourself, you read the Wikipedia page and it all clicks together.
  3. You decide that you should be ready for the next Chinese New Year.
  4. You act by ordering the envelopes which 10x sales. Success!

This wasn't a tough mystery to solve... if you work at the store.

Now imagine you work at the e-commerce store that sells paper. The computer silently handles the sales. Computers don't volunteer information; you have to ask for it. So, by default you know nothing, not even if you had sales that day. To know that, you have to:

  1. Decide that you want to know the daily revenue.
  2. Calculate it (and debug it!) with a SQL query, then put it on a dashboard.
  3. Look at the dashboard daily.

But are you breaking the revenue by ethnicity / nationality? Are you breaking it down by paper color? Did you notice Chinese New Year?

Most likely not. In a software business, you are default blind.

OODA loop

The Observe, Orient, Decide, Act loop is a way to think about how people make decisions in real-time, challenging situations. Observe is the first step. Without it, you are not Oriented, which makes it impossible to make the right Decisions and Actions.

In my experience at software businesses, it is not hard to Decide and Act. The hard part is to Observe and Orient. Like in the red envelope story, once you Observe, the rest is usually pretty easy.

We forget to observe

In software, observing is not only hard because of the additional work. It is also hard because you don't realize you are not observing.

You work in the paper e-commerce store; the dashboard shows sales going up. It's January, so you think it is probably because people are coming back to school from the winter break. That is a sufficient (but wrong) explanation for the incomplete data you have. So, you call it a day and move on1. You didn't feel like you needed to observe in the first place.

We automatically generate hypotheses of what is happening whether we did the work to observe or not2. And then, we confuse our first hypothesis for a hard-earned model. In concrete domains where we are observing all the time, the those hypothesis are likely to fit the data because we have data. In abstract systems, we have observed very little data but you never notice how little.

"Did you remember to observe?" ends up being the first and most important step. And while it is not necessarily a binary choice (you can observe more or less), it looks binary in practice. Some people remember to observe and build a mental model of the abstract system while others don't and remain lost.

Who is oriented? Who is lost?

You can tell who is lost by the questions they ask:

  • They lack basic concepts and definitions. "What is LTV again?".
  • They can't contextualize metrics. "We sold 2000 items today", "Is that a lot?".
  • They can't interpret events. "Our partner launched a new product", "What does that mean for us?"

These can be perfectly legitimate questions and "there are no dumb questions" is a healthy mantra:

  • It allows beginners to get up to speed quickly.
  • It makes it easier to clarify situations where people are afraid to ask for looking stupid.

But if the business leader asks "What is LTV?" after a year on the job, they are completely lost. The downside of "there are no dumb questions" is that it prevents you from recognizing who is perennially lost.

And in an abstract domain people can remain lost for years. It is easy to see how it happens:

  • Work starts, you open Slack, and people are talking about X which feels urgent.
  • Based on X, you do something. You tell yourself and others what effect it will have.
  • The outcome is visible but only to those that sift through the resulting data.
  • But nobody has time to do that because there is some new urgent Y to deal with.

The cycle starts again. Years go by and nothing you ever did mattered. Sometimes because X itself was unimportant, other times because you missed a crucial detail, and other times because you had the entire story wrong. But because you failed to check, so you'll never know.

This doesn't happen in the concrete world. You show up to the store for work. The shelves are empty. The supplier arrived late. The customers are outside, forming a line to buy. Time to work! When you are done, there is no lingering "Did that matter?".

Is the entire company lost?

What happens in a company where 50% of people are lost?

Statistically, in most meetings you have a few people who aren't lost. This correlates with tenure, so it is natural for those people to lead the meeting. The other 50% listens, asks some questions, and hopefully orient themselves. Half the team can see the ball, they tell the rest the where the ball is, and the game continues.

50% is not ideal but at least it is stable. In most discussions, the already-oriented agree and set direction for the rest. Most of the company is playing near the ball and has a chance to learn.

But what happens in a company where 80% of people are lost?

Statistically, there are many meetings where everybody is lost. Decisions will be made, actions will be taken, and have no results. The team is on one side of the field while the ball is on the other.

In other meetings, you might have only one or two people that are oriented. If those happen to be the leaders, they'll speak up when the lost say nonsense. But if the leaders are lost, the oriented are likely to stay silent, confused by the nonsense3.

Unlike the 50% scenario, the 80% is no longer stable. Now, most discussions are nonsense. The new people can't learn where the ball is because nobody arounds them has ever seen it. The oriented people feel like they are losing their minds and leave. You quickly go from 80% lost to 100% lost.

This is what I fear when a company grows its headcount "too quickly". Such discussions are often about "losing the culture". For software companies, I fear more "losing a basic understanding of what we are even talking about".

Now what?

This is not an encouraging post and I am trying hard to find a silver lining. Here is something:

  • Once you know to look for people who are oriented, it is easy to spot them. Others ask them questions constantly. They'll also often present evidence for their arguments, for example a SQL query.
  • You can get better at this! Study the business. Take notes of what the important metrics are, how they relate to each other, and what are things the company has done to impact them. Take what people tell you with a grain of salt!
  • Finally, expect others to be oriented. If a teammate has been lost for 2 years, that is sign that they'll never orient themselves and they should work elsewhere. Holding that bar is important.


Footnotes

  1. There is no time to dig into every anomaly, there are too many. But you can also see how our natural bias is towards explaining anomalies away and not towards investigating.↩
  2. Before working in industry, I was a bit befuddled by the "data-driven" mantra. It explicitly ignores taste, common-sense, etc. But I get it now. It is a reminder to actively Observe first.↩
  3. An reader described one of these meetings with "I have a distinct memory of being the only oriented person in a 10 person meeting, felt I was on some kind of hallucinogen, absolute bizarro world"↩