Correlation does not equal causation. Every data scientist has had that drummed into them. In fact, when I taught data science, I used to be the one doing the drumming.

However, as every data nerd’s favourite comic, “xkcd”, once pointed out: correlation “does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’”.

This is the part most data scientists miss. Without correlation, there can be no causation and often, it’s because researchers noted the existence of a correlation that they later went on to prove causation.

This was the case with the link between smoking and lung cancer. Medical researchers knew about the correlation for years before they were able to establish causation, because lung cancer takes so long to develop.

So by establishing correlation, you’re demonstrating that a cause-and-effect relationship is, at least, a possibility.

Here’s the thing…

If you’re dealing with small samples or low frequency events, sometimes you may never have enough data to establish causation in the strict scientific sense.

Fortunately, to enable better executive decision-making, most of the time your results don’t need to be good enough to publish in a peer-reviewed journal. They just need to be good enough to reduce decision-making uncertainty.

I’m not saying that establishing correlation means you can completely avoid demonstrating causation. However, in many cases, correlation waggling its eyebrows suggestively, may just be enough to enable decision-makers to approve the pilot experiments needed to increase their conviction that causation is, in fact, at work.

Correlation may not be a substitute for establishing causation, but it can be a stepping stone towards it.

This is one of the ideas I recently discussed with Vikram Shetty on his podcast, The ROI of DEI.

Vikram’s made it his business to help culture consultants quantify the financial value of their work. We spent the episode talking about how data scientists can also help HR leaders build that case, including:

• The three factors that build executive trust in a quantitative model.
• Why single-point estimates undermine your credibility.
• What separates a credible business-impact argument from a chain of speculative claims.
• How to move a leader from “we can’t prove causation” to “we know enough to act”.

🎧 Listen now on Apple Podcasts or Spotify, or click the link HERE.

Talk again soon,

Dr Genevieve Hayes