Your Teams Are Using AI. Do They Know When It's Wrong?

Somewhere within your organisation right now, someone is submitting AI-generated content they haven't checked. It sounds right, it reads confidently and a brief skim makes sure it matches the brief at hand. Yet, without a thorough quality assessment, it might be completely wrong. 

This is not a scaremongering prediction. It is a very real risk and is already happening across the world; in courtrooms, newsrooms, HR departments, medical journals and the consequences, legal liability, reputational damage, and regulatory scrutiny, are real:

• A legal team cited a rule in a High Court case that didn't exist. Despite the AI flagging its own output as unverified, the lawyers submitted it anyway. The firm apologised unreservedly, referred itself to the regulator, and the supervising partner was publicly criticised by the judge.
• An airline's chatbot promised a customer a bereavement discount that contradicted company policy. A court ruled the airline had to honour it. When challenged, the airline argued the chatbot was "a separate legal entity" not responsible for its own actions. The tribunal rejected that entirely. The chatbot was removed from the website shortly after.
• The Chicago Sun-Times published a summer reading list. Ten of the fifteen books didn't exist. Chicago Public Media immediately distanced itself from the content, the CEO issued a public apology, and the incident made international headlines, two months after the paper had already cut 20% of its editorial staff.

Different sectors, different use cases: legal research tool, a customer service chatbot, a content generation assistant. Same gap. People are using AI who don't fully understand they can’t trust it blindly and need a thorough process of verification. There was no standardised way of working that led to catching these mistakes.

This isn't a one-off

It would be convenient if these were isolated incidents, but they're not. In fact, the pattern is now so consistent that researchers are tracking it. The number of published legal decisions involving AI-generated hallucinations  has already passed 220 and that’s just the cases that made it to court. It says nothing about the errors that never got scrutinised, the content that went out unchecked, or the decisions made on data that was never real.

Every one of these incidents has something in common. Regardless of the technology used, the output was trusted and the understanding wasn't there to question it. The evidence speaks for itself; there is one glaring problem that must be solved from within the organisation. A knowledge gap that AI has no ability to fill, and no power to flag it.

What's happening inside the model

AI hallucinations are not bugs. They are a - somewhat counter-intuitive - feature of how large language models work.

LLMs are designed to predict the most plausible next word or sentence, based on patterns learned from vast amounts of training data. While some models now integrate live data sources, this doesn't eliminate the core problem: when a model encounters a gap in its knowledge, it fills it, fluently, confidently, and sometimes completely incorrectly.

Crucially, GenAI tools do not inherently warn users about the limitations of their output. Disclaimers exist, but they are easy to ignore and often poorly understood. The problem is compounded by how most people are taught to prompt. Standard AI prompt training, the kind found in countless online guides and workplace toolkits, encourages users to instruct the model to write "as an expert." The output becomes more authoritative in tone, which makes errors harder to spot and easier to trust. The advice that's meant to improve results can also make hallucinations more convincing.

Research suggests hallucination rates can vary significantly by domain. Some studies put the range at 20% to 60% depending on the subject area, though estimates vary. In legal, compliance, and technical contexts, exactly the areas where accuracy matters most, the risk is highest.

None of this means AI is unusable, but it does prove that AI requires a specific kind of literacy to use well. And right now, most organisations have an AI literacy problem.

Why organisations keep getting caught out

Every one of these failures has a human layer. At its core this is an AI literacy problem, and there are four key blind spots that widen that gap.

Deployment is outpacing understanding

Worker access to AI rose 50% in 2025 but access and readiness are not the same thing. 75% of organisations report having a dedicated AI governance process, yet only 12% describe their efforts as mature. The tools are in the building, but the knowledge to use them safely often isn't.

Training is generic when it needs to be role and capability specific

A one-size-fits-all approach to AI literacy doesn't reflect how AI is actually used across an organisation. The risks a legal team faces are different from those in finance, HR, or marketing. A non-technical employee using AI to draft communications faces different failure modes than a data analyst using it to interrogate a dataset. Without training anchored to the specific context of someone's role, the knowledge doesn't stick and the gaps stay invisible until something goes wrong. Only 34% of enterprises say their AI programmes produce measurable financial impact and less than 20% have mature governance frameworks in place. The connection isn't coincidental.

Small mistakes go unaddressed until they become big ones

55% of organisations describe AI use as a "chaotic free-for-all", with AI applications being built in silos across departments. Without clear accountability structures, AI errors aren’t flagged. In fact,  they get buried with no mechanism to learn from near-misses, no culture of questioning outputs, and no one whose job it is to notice the pattern before it becomes a serious problem.

There is no one accountable for getting it right

Between 40% and 65% of enterprise employees report using AI tools not approved by their IT department. Shadow AI is the dominant operational reality and highlights exactly why organisations need AI stewards: people embedded across teams and departments, responsible for ensuring AI is being used appropriately within their specific area. Not a centralised IT function but people within departments who understand both the technology and the business context they're operating in, bridging the gap between what AI can do and what their colleagues need to know.

What responsible AI use actually looks like

Organisations that use AI well have a few things in common:

• Human review is non-negotiable
  AI generates, a person with the knowledge to evaluate it reviews it before it goes anywhere, and at regular checkpoints throughout distribution. 
• Teams that know what hallucination red flags look like
  Overconfident tone, claims without traceable sources, citations that can't be verified are all learnable signals but require training to spot reliably and dig deeper into.
• AI is a first draft tool, not a final authority
  Simple in theory, tough in practice. It requires a cultural shift that starts with leadership understanding the risk clearly enough to communicate it credibly.
• They have a real internal AI use policy
  Not a blanket ban nor a free-for-all. One that reflects how people work, what tools are in use, and what level of human oversight different outputs require.

While this may seem like adding extra, unnecessary steps, research shows that people who actively verify AI outputs are more likely to keep using AI confidently over time, not less. Literacy makes adoption sustainable, even if it feels slower.

The cost of doing nothing

The risks here aren't abstract, and they aren't small. Reputational damage, legal liability, and financial loss are already impacting organisations that moved fast on AI without bringing their people with them. Global business losses from AI hallucinations reached $67.4 billion in 2024, spread across direct losses, operational cleanup, and reputational damage, and that figure is rising. IDC estimates the AI skills gap will cost the global economy up to $5.5 trillion by 2026, a workforce that has the tools but lacks the knowledge to use them responsibly. 

Most leaders know this is a problem, but the question is whether they're doing something about it. 71% of C-suite executives already identify hallucinations as a direct threat to their decision-making integrity, and organisations with formal AI training programmes achieve 2.3x faster adoption and 67% higher ROI than those without. The awareness is there, but the question is whether the action will follow.

Where to start

At Data & AI Literacy Academy, we work with enterprise organisations to build the kind of AI literacy that makes incidents like these avoidable. We give teams the knowledge, confidence and critical judgment to use AI tools well. 

AI means something different in every department and our education reflects this. A legal team using it to research case precedents faces entirely different risks to an HR manager generating a job description, a finance analyst summarising regulatory filings, or a marketing team producing client-facing content. The failure modes are different, the consequences are different, and the knowledge needed to catch them is different too.

The first step is understanding where you stand. Most organisations find the gap is bigger than they expected and more specific than they assumed. The good news is that it's measurable, it's addressable, and it doesn't have to wait for something to go wrong before you act on it.

If you're ready to find out where your organisation stands, we'd like to talk.