You Need Far Less Data Than You Think


Quick quiz: If you randomly sampled just 5 people from a population of 20,000 or more, could you use that data to tell your stakeholders anything useful?

For most data scientists, the answer is probably no.

But according to Douglas Hubbard, author of How to Measure Anything, “you need far less data than you think.”

Here’s the proof:

If you randomly sample just five people from an organisation of any size and record the lowest and highest values of whatever you’re measuring, there’s a 93.75% chance that the true median of the entire organisation falls somewhere between those two numbers.

It’s not a trick. It’s statistics.

And it’s one of several techniques Doug has spent over 35 years applying to some of the highest-stakes decisions in business and government.

In the latest Value Boost episode of Value Driven Data Science, Doug joins me to share more of these techniques, which you can use to support high-stakes decision-making when data is scarce and every observation counts.

In just 16 minutes, you’ll learn:

  1. Why a single observation reveals more than you think [01:58]
  2. How Laplace’s Rule of Succession lets you estimate probabilities from tiny samples [08:25]
  3. The Rule of Five and what it reveals about small sample statistics [12:08]
  4. The simplest and most overlooked technique for reducing measurement uncertainty [14:07]

You don’t need more data. You just need to know what to do with what you have.

Listen now on Apple Podcasts or Spotify, or click the link below:

Episode 110: Why You Need Less Data Than You Think

Talk again soon,

Dr Genevieve Hayes

Data Science Impact Algorithm

Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.

Read more from Data Science Impact Algorithm

The number one piece of advice I give to data professionals is to have more conversations. Conversations lead to good things. Things like job offers and stronger relationships. But more importantly, the opportunity to deepen your knowledge through learning from the wisdom of others. I owe many of the best opportunities I’ve encountered in my career to proactively seeking them out wherever I can. My podcast is a weekly example of the power of conversation. But if there’s one thing all these...

While working as a data scientist in a large organisation, I was once called on to attend a meeting with representatives of a big tech company and multiple senior leaders from my business area. The tech reps were there to sell us a new product. The data scientists had been invited because we were the intended users. Although there were a dozen or more people in the room, I was the only person who came prepared with any questions - 2 1/2 pages of them, to be precise. I let everyone else have...

Like many readers of this newsletter, I identify as a data expert. It took me years of study and hard work to build this expertise, and it’s something that I’m not willing to lose. This is not just because of everything it took to get here, but because my expertise forms an integral part of my identity. And the idea of losing it feels, in many ways, akin to losing a part of who I am. So, when I recently started hearing from data professionals who spoke of slowly losing skills as they...