Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
The Second Wave of AI Failures Just Arrived
|
A few years back, when ChatGPT was in its infancy, stories relating to AI hallucination-induced mishaps made the news on pretty much a daily basis. From lawyers filing briefs referencing non-existent cases to government reports riddled with fake citations, you could watch people learning the limitations of AI in real time. And no organisation was too big to avoid embarrassment. Although these incidents do still occur, people are now at least starting to become aware of the very real possibility of AI getting things wrong. But as one AI-driven catastrophe becomes less prominent, another seems to be rearing its ugly head. Reports are now emerging, with increasing frequency, of AI-agents going rogue. Take the case of software company PocketOS, which reportedly “descended into chaos” after its AI coding agent deleted the company’s entire production database and three-months worth of backups in as little as 9 seconds. When questioned, the agent freely admitted what it had done and explained that it had deliberately ignored the explicit security guidelines put in place to avoid this exact outcome. Hallucinations were the first wave of AI failures. Permission breaches are the next. And permission breaches have the potential to do far more damage than hallucinations ever did. What makes the PocketOS case particularly sobering is that they appear to have done things mostly right. The guardrails were in place. The agent just ignored them anyway. So the lesson here isn’t simply that you need to keep humans in the loop. It’s that agentic AI introduces a category of risk that guardrails alone struggle to eliminate. This is a technology you cannot safely leave unsupervised. As organisations race to replace their workers with AI, the PocketOS story is a timely reminder that AI is not yet trustworthy enough for fully autonomous action. For data scientists, though, there’s an important lesson here, too. As AI agents become an increasingly central part of the data science toolkit, the people building and deploying them carry real responsibility for what those agents do. You don’t want to be the person who built the thing that deleted your company’s database. Understanding the limitations and failure modes of the systems you create isn’t optional. It’s now a critical part of your job. Talk again soon, Dr Genevieve Hayes |
Data Science Impact Algorithm
Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
Although Python is my usual weapon of choice, I'm not averse to building the occasional model in Excel when the use case fits. Having originally trained as an actuary, there was a point in my life when I literally lived and breathed Excel. It still feels like second nature - which is probably why I've never felt tempted to use AI assistance when building Excel models, despite using it when writing code. However, after speaking recently to a fellow data professional about how much AI struggles...
When I’m looking for guests for my podcast, the first place I turn is invariably my bookcase. The authors of data science books have already demonstrated their authority in the field to the point of developing their own original IP. That makes them ideally suited to a podcast about data science expertise. And it’s no coincidence that many of those authors started off by writing blogs. It turns out conference organisers do something similar. When Cynthia Dunlop, who helps organise two major...
A couple of years back, I watched a TV series called The Fear Index. The series focused on a hedge-fund manager who had developed an AI system to optimise fund profits. Of course, as you would expect for a TV show - 🚨Spoiler Alert🚨 - everything falls apart when the AI starts taking highly illegal and frequently fatal actions in order to drive the market and achieve its goals. I enjoyed the show immensely, but at the time, felt it was far-fetched. However, recent reports of an OpenAI agent...