Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
The Second Wave of AI Failures Just Arrived
|
A few years back, when ChatGPT was in its infancy, stories relating to AI hallucination-induced mishaps made the news on pretty much a daily basis. From lawyers filing briefs referencing non-existent cases to government reports riddled with fake citations, you could watch people learning the limitations of AI in real time. And no organisation was too big to avoid embarrassment. Although these incidents do still occur, people are now at least starting to become aware of the very real possibility of AI getting things wrong. But as one AI-driven catastrophe becomes less prominent, another seems to be rearing its ugly head. Reports are now emerging, with increasing frequency, of AI-agents going rogue. Take the case of software company PocketOS, which reportedly “descended into chaos” after its AI coding agent deleted the company’s entire production database and three-months worth of backups in as little as 9 seconds. When questioned, the agent freely admitted what it had done and explained that it had deliberately ignored the explicit security guidelines put in place to avoid this exact outcome. Hallucinations were the first wave of AI failures. Permission breaches are the next. And permission breaches have the potential to do far more damage than hallucinations ever did. What makes the PocketOS case particularly sobering is that they appear to have done things mostly right. The guardrails were in place. The agent just ignored them anyway. So the lesson here isn’t simply that you need to keep humans in the loop. It’s that agentic AI introduces a category of risk that guardrails alone struggle to eliminate. This is a technology you cannot safely leave unsupervised. As organisations race to replace their workers with AI, the PocketOS story is a timely reminder that AI is not yet trustworthy enough for fully autonomous action. For data scientists, though, there’s an important lesson here, too. As AI agents become an increasingly central part of the data science toolkit, the people building and deploying them carry real responsibility for what those agents do. You don’t want to be the person who built the thing that deleted your company’s database. Understanding the limitations and failure modes of the systems you create isn’t optional. It’s now a critical part of your job. Talk again soon, Dr Genevieve Hayes |
Data Science Impact Algorithm
Twice weekly, I share proven strategies to help data scientists get noticed, promoted, and valued. No theory — just practical steps to transform your technical expertise into business impact and the freedom to call your own shots.
“AI destroys jobs but creates businesses.” That one statement crystallised what I’ve long believed about building a data science career. The statement was made by best-selling author and futurist Peter H. Diamandis in a recent post. Diamandis argues that while AI will lead to the destruction of white-collar jobs, as organisations replace human workers with AI, AI has made it easier than ever for entrepreneurs to launch one-person companies. The jobs that AI creates, therefore, won’t look like...
Quick quiz: If you randomly sampled just 5 people from a population of 20,000 or more, could you use that data to tell your stakeholders anything useful? For most data scientists, the answer is probably no. But according to Douglas Hubbard, author of How to Measure Anything, “you need far less data than you think.” Here’s the proof: If you randomly sample just five people from an organisation of any size and record the lowest and highest values of whatever you’re measuring, there’s a 93.75%...
I learned the true meaning of accountability on a Sunday morning, many years ago, when I had to go into the office to fix a calculation mistake made by a member of my team. I was an insurance pricing manager at the time. My team performed premium calculations that brought in $2 billion of revenue. It was complex work, spread across multiple staff, and the margin for error was incredibly low. When my boss spotted an error in a table my team had produced, it was all I could do not to say:...