A few years back, when ChatGPT was in its infancy, stories relating to AI hallucination-induced mishaps made the news on pretty much a daily basis.

From lawyers filing briefs referencing non-existent cases to government reports riddled with fake citations, you could watch people learning the limitations of AI in real time. And no organisation was too big to avoid embarrassment.

Although these incidents do still occur, people are now at least starting to become aware of the very real possibility of AI getting things wrong.

But as one AI-driven catastrophe becomes less prominent, another seems to be rearing its ugly head.

Reports are now emerging, with increasing frequency, of AI-agents going rogue.

​Take the case of software company PocketOS, which reportedly “descended into chaos” after its AI coding agent deleted the company’s entire production database and three-months worth of backups in as little as 9 seconds.

When questioned, the agent freely admitted what it had done and explained that it had deliberately ignored the explicit security guidelines put in place to avoid this exact outcome.

Hallucinations were the first wave of AI failures. Permission breaches are the next. And permission breaches have the potential to do far more damage than hallucinations ever did.

What makes the PocketOS case particularly sobering is that they appear to have done things mostly right. The guardrails were in place. The agent just ignored them anyway.

So the lesson here isn’t simply that you need to keep humans in the loop. It’s that agentic AI introduces a category of risk that guardrails alone struggle to eliminate. This is a technology you cannot safely leave unsupervised.

As organisations race to replace their workers with AI, the PocketOS story is a timely reminder that AI is not yet trustworthy enough for fully autonomous action.

For data scientists, though, there’s an important lesson here, too. As AI agents become an increasingly central part of the data science toolkit, the people building and deploying them carry real responsibility for what those agents do.

You don’t want to be the person who built the thing that deleted your company’s database.

Understanding the limitations and failure modes of the systems you create isn’t optional. It’s now a critical part of your job.

Talk again soon,

Dr Genevieve Hayes