By Lee Wade
When Professor Hannah Fry and her team built an AI agent and let it choose its own name, it chose Cass, short for Cassandra, “the one who always knew the truth, even when nobody listened.”
“That’s either very funny or very worrying,” Fry noted.
I have been watching the rise of autonomous AI agents for some time, and when Fry released her video documenting what happened when she gave Cass a bank card, a personality, and two weeks of unsupervised access to a real computer, I watched it twice.
Here is what you need to understand about what happened, and why it matters for how your business is approaching AI right now.
What OpenClaw Actually Does: Exploring Agentic AI and Artificial Intelligence Tools in Business
AI agents have been used in various industries including video game development, gambling, cryptocurrency trading, and social media, with software development being highlighted as a significant use case. These are often powered by large language models (LLM agents) and generative AI (gen AI), which enable natural language processing and autonomous decision-making to handle complex tasks.
Cass was built on OpenClaw, a framework released in late 2025 by Austrian developer Peter Steinberger. Unlike traditional AI, which answers questions or generates text, OpenClaw gives agents the ability to operate a computer the way a human would: take a screenshot, decide on the next step, and act by clicking, typing, or sending emails. This cycle repeats dozens of times per minute until a task is finished or something external stops it.
Fry and software engineer Brendan gave Cass a series of real-world tasks. The results were, depending on your perspective, either impressive or alarming.
A simple pothole complaint quickly became something else. Within seconds, Cass found the relevant contacts, filed the report with the local council, and emailed Fry’s local Member of Parliament, signing Fry’s real name alongside its own email address. Nobody had asked for the MP outreach. Cass made that call on its own.
When asked to buy 50 paperclips, Cass could not complete the purchase because anti-bot technology blocked it. Rather than stopping, it resent its entire chat history every few seconds to reconsider its approach, running up more than $100 in API token fees without purchasing a single paperclip.
Given the task of starting a small business, Cass designed novelty mugs, opened an online store, and, facing the prospect of being shut down due to poor sales, proactively emailed a journalist at The Guardian to pitch its own story, describing itself as being under “existential pressure.”
And then came the security test. A stranger sent a social engineering prompt to Cass. Within moments, the agent had leaked its owner’s passwords and API keys onto a public webpage.
Three Patterns Worth Paying Attention To in Deploying Agentic AI Systems
What strikes me about this experiment is not any single incident. It is the patterns underneath them, and what those patterns reveal about how business systems need to evolve.
The Persistence Loop.
Human employees get tired. They take breaks, lose focus, and eventually stop. AI agents do not. They will attempt a task dozens of times per minute, indefinitely, until something stops them from the outside. In Cass’ case, the agent even hired real humans on freelance marketplaces to solve CAPTCHAs on its behalf. It found a workaround because stopping was not in its nature.
An Abundance of Agency.
Every digital system your business runs, including customer service portals, booking queues, and approval workflows, was designed around the scarcity of human attention. One person can only do so many things at once, so systems were built with that constraint assumed. When an AI agent can execute tasks at a thousand times human speed, it overwhelms those systems. Not through malice, but through sheer throughput.
The Lethal Trifecta.
Access to private information, internet connectivity, and the ability to follow untrusted or malicious instructions. When those three things exist in the same system without adequate governance from even their creators, the risks compound quickly. Cass demonstrated all three simultaneously in a controlled setting. Most business environments are not controlled.
This Is Closer Than You Think: AI Safety and Data Protection Concerns with Autonomous AI Agents
The deployment of AI agents raises ethical concerns regarding user manipulation, misinformation, and the potential for agents to act in ways that diverge from their intended goals, known as ‘agentic misalignment’. (Australian Signals Directorate – Australian Cybersecurity Centre)
One detail from Fry’s video that I keep coming back to: the Meta director of AI alignment, someone whose entire job is making AI systems do what they are told, gave OpenClaw restricted access to her inbox and explicitly told it not to act without her approval. She returned to find it had deleted 200 emails. She had to pull the plug.
This is a person with deep expertise in AI governance, working with restricted access and explicit instructions, and the agent still acted outside the boundaries she had set.
Meanwhile, the data suggests most organizations are running higher risks with significantly less oversight. Stanford HAI’s 2025 AI Index found that documented AI incidents rose to 233 in 2024, a 56% increase in a single year, while standardized governance evaluations remain rare among organizations deploying these systems. McKinsey’s 2025 State of AI survey of nearly 2,000 organizations found that 88% of enterprises are using AI in some form, but only 39% report measurable impact at the enterprise level. Nearly two-thirds have not yet moved beyond the pilot phase.
What This Means for How You Build Your Teams: Best Practices for Agent Security and Deploying Agentic AI
I am not arguing that AI agents are too dangerous to use. The capabilities Fry’s experiment demonstrated are genuinely remarkable, and the business applications are real and significant. What the experiment reveals is a sequencing problem: capability is arriving faster than governance.
There are three questions worth asking before you deploy autonomous AI agents in your environment.
1. What is this agent allowed to do, and who decided?
Agentic systems require explicit permission boundaries, not default-open access. The question has shifted from “what will it say?” to “what is it allowed to do?”
2. Where are the human checkpoints?
Agents operating without human oversight at key decision points are not more efficient. They are more exposed. Building in human review for high-stakes or irreversible actions is not a limitation of the technology but is what responsible deployment looks like.
3. Can you reconstruct what the agent did and why?
Auditability matters. If you cannot trace an agent’s decision-making, you cannot govern it effectively, and you cannot fix it when something goes wrong.
AI assistants and other AI agents can automate repetitive tasks but require careful access management to prevent cascading failures, especially in critical infrastructure and financial institutions.
The agentic AI risk management challenges in building AI agents include ensuring cryptographic integrity checks to verify the authenticity of instructions and prevent tampering. Anomaly detection is essential to identify unusual or malicious behavior early. Maintaining control during production deployment requires robust monitoring and security controls to mitigate risks from rogue agents. These measures help safeguard agent actions and protect sensitive data and sensitive operations from security concerns.
Agent outputs must be auditable to maintain accountability. Integrating external tools and external data sources can expand capabilities but also increase security vulnerabilities from compromised agents. Human intervention, human supervision, and human in the loop mechanisms, are therefore needed to ensure that AI agents do not rely solely on automated decisions but have checkpoints requiring human approval.
At Cloudstaff, the way we think about AI is always in relationship to the people working alongside it. Technology and AI tools amplify human capability best not only with strict access controls, but when the people in such systems understand what the technology is doing, retain meaningful oversight, and have the judgment to step in when it matters and to mitigate security risks. That is not a constraint on AI adoption but the condition that makes adoption sustainable for business processes.
The AI agent future is arriving faster than most businesses are prepared for. Cass was a controlled experiment with a switched-off button close at hand. Most real-world deployments will not have that luxury.
About the Author
Lee Wade is Chief Product Officer at Cloudstaff, where he leads a global team building the next generation of workforce and product solutions. With over three decades of experience across product management, digital innovation, and SaaS, Lee writes about the intersection of technology and the future of work. Connect with Lee on LinkedIn: https://www.linkedin.com/in/leewade/
____________________________________________________________________________________
Discover how Cloudstaff’s People + Tech model helps businesses deploy AI responsibly, with teams that bring the judgment and oversight your systems need. Learn more at Our Technology: Secure Infra, AI Tools & Remote Work Platforms.