|
A few years back, when ChatGPT was in its infancy, stories relating to AI hallucination-induced mishaps made the news on pretty much a daily basis. From lawyers filing briefs referencing non-existent cases to government reports riddled with fake citations, you could watch people learning the limitations of AI in real time. And no organisation was too big to avoid embarrassment. Although these incidents do still occur, people are now at least starting to become aware of the very real possibility of AI getting things wrong. But as one AI-driven catastrophe becomes less prominent, another seems to be rearing its ugly head. Reports are now emerging, with increasing frequency, of AI-agents going rogue. Take the case of software company PocketOS, which reportedly “descended into chaos” after its AI coding agent deleted the company’s entire production database and three-months worth of backups in as little as 9 seconds. When questioned, the agent freely admitted what it had done and explained that it had deliberately ignored the explicit security guidelines put in place to avoid this exact outcome. Hallucinations were the first wave of AI failures. Permission breaches are the next. And permission breaches have the potential to do far more damage than hallucinations ever did. What makes the PocketOS case particularly sobering is that they appear to have done things mostly right. The guardrails were in place. The agent just ignored them anyway. So the lesson here isn’t simply that you need to keep humans in the loop. It’s that agentic AI introduces a category of risk that guardrails alone struggle to eliminate. This is a technology you cannot safely leave unsupervised. As organisations race to replace their workers with AI, the PocketOS story is a timely reminder that AI is not yet trustworthy enough for fully autonomous action. For data scientists, though, there’s an important lesson here, too. As AI agents become an increasingly central part of the data science toolkit, the people building and deploying them carry real responsibility for what those agents do. You don’t want to be the person who built the thing that deleted your company’s database. Understanding the limitations and failure modes of the systems you create isn’t optional. It’s now a critical part of your job. Talk again soon, Dr Genevieve Hayes |
Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
Have you ever been nerd sniped? It's the tendency technically-minded people have to get distracted by interesting or challenging problems to the point of ignoring everything else. This xkcd comic sums it up. I am frequently nerd sniped, and for me, it usually means spending way too much time thinking up solutions to other people's business problems. But I'm starting to realise that's not necessarily bad. The same business problems have a habit of coming up again and again. And when they raise...
Machine learning models are designed to predict outcomes. Given the characteristics of a customer, a machine learning model can, for example, predict whether they'll buy a product, click on a link, cancel a service, or claim on their insurance. But what an ML model can't tell you is what causes the customer to do any of these things - and what would happen if an organisation intervened to change their behaviour. This is because ML models are correlation-based. And as most data scientists...
Recently, I've been systemising my workflows. It's partly about efficiency, but also a way of experimenting with AI. Whenever I encounter a process I've performed dozens of times, I look for a way to automate, templatise or standardise it by feeding my existing knowledge and artefacts into Claude. It's system leverage 101, turbo-charged by AI. But even though AI makes this easier than ever before, none of this would be possible if I hadn't already mastered these tasks. I'm not asking AI to...