Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
The Second Wave of AI Failures Just Arrived
|
A few years back, when ChatGPT was in its infancy, stories relating to AI hallucination-induced mishaps made the news on pretty much a daily basis. From lawyers filing briefs referencing non-existent cases to government reports riddled with fake citations, you could watch people learning the limitations of AI in real time. And no organisation was too big to avoid embarrassment. Although these incidents do still occur, people are now at least starting to become aware of the very real possibility of AI getting things wrong. But as one AI-driven catastrophe becomes less prominent, another seems to be rearing its ugly head. Reports are now emerging, with increasing frequency, of AI-agents going rogue. Take the case of software company PocketOS, which reportedly “descended into chaos” after its AI coding agent deleted the company’s entire production database and three-months worth of backups in as little as 9 seconds. When questioned, the agent freely admitted what it had done and explained that it had deliberately ignored the explicit security guidelines put in place to avoid this exact outcome. Hallucinations were the first wave of AI failures. Permission breaches are the next. And permission breaches have the potential to do far more damage than hallucinations ever did. What makes the PocketOS case particularly sobering is that they appear to have done things mostly right. The guardrails were in place. The agent just ignored them anyway. So the lesson here isn’t simply that you need to keep humans in the loop. It’s that agentic AI introduces a category of risk that guardrails alone struggle to eliminate. This is a technology you cannot safely leave unsupervised. As organisations race to replace their workers with AI, the PocketOS story is a timely reminder that AI is not yet trustworthy enough for fully autonomous action. For data scientists, though, there’s an important lesson here, too. As AI agents become an increasingly central part of the data science toolkit, the people building and deploying them carry real responsibility for what those agents do. You don’t want to be the person who built the thing that deleted your company’s database. Understanding the limitations and failure modes of the systems you create isn’t optional. It’s now a critical part of your job. Talk again soon, Dr Genevieve Hayes |
Data Science Impact Algorithm
Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
I once met a data scientist who was so good at his job that he automated his way out of it. He systematically built automations to handle his team’s work and once those automations were in place, the team no longer needed as many people. So, his role was ultimately made redundant. He’d built those automations because he believed they’d create business value for his employer - and they did. The problem was that the value ended up residing in the code, not in him. And because his employer owned...
The hardest course I took during my entire data science education was a theoretical computer science class. But not for the reasons you'd think. Theoretical computer science is basically pure maths for data scientists, and the material was genuinely challenging. I spent 30 hours/week studying just to keep up. Yet, the reason why this course became known as "the widow maker" wasn't just the time commitment. It was because referring to any materials beyond the prescribed texts was considered an...
Before I was a data scientist, I was a statistician. For a long time, I thought of that as a liability. Machine learning felt like the way of the future and my statistical training felt like it belonged in the past. But over time, I noticed something. While most data scientists thought in terms of point estimates, I instinctively thought in terms of uncertainty ranges. It turned out that instinct was more valuable than I’d given it credit for. According to Adam DeJans Jr, co-author of The...