While visiting my parents recently, I had a conversation with my (non-technical) 80-year-old dad that went something like this:

Dad: The problem with AI is that it makes mistakes. If you don’t know if it’s right or wrong, then it’s useless.

Me: But you use (OCR tool) Abbyy all the time. That’s AI. Is that useless?

Dad: No. Because of Abbyy, I get stuff done in a fraction of the time it would otherwise take me. But I still have to review everything it does because it makes mistakes.

What this conversation highlighted was that, to my dad, it’s clearly not the fact that AI makes mistakes that determines its usefulness. It’s whether it saves him time and whether there’s a feasible path to mitigating any errors.

This struck me as almost perfectly echoing the words of statistician George Box:

It also clarified something for me about how data scientists should think about the models they build.

By all means, reduce the number of mistakes your model makes, because usefulness will always be partly a function of accuracy. But accuracy isn’t the only thing that determines whether a model is useful.

Before obsessing over another percentage point of model accuracy, pause a moment and ask yourself these two questions:

1. Does the model save enough time to be worth using at its current level of accuracy?
2. Is there a feasible process for catching and correcting any errors it does make?

My dad figured this out on his own, without a statistics degree. He uses an imperfect AI tool productively every day because the time saving is significant and his review process catches the mistakes.

That’s not a workaround. That’s good judgement.

And it’s exactly the same judgement data scientists should be applying every time they decide whether a model is ready to deploy.

Talk again soon,

Dr Genevieve Hayes