The KPI problem: Why data & AI fluency struggle to prove value

Let's be honest. Most data and AI programmes have a value problem, and they had it from the very beginning, because nobody designed them with value in mind.

That's a pretty uncomfortable thing to say out loud. But it's true. And it applies just as much to data and AI literacy programmes as it does to data projects in general. Whatever you call it, whether it's fluency, literacy, confidence or culture, the challenge is the same. When someone asks "so, is this actually working?", most teams don't have a great answer.

Let's unpack this together. Not just identifying the problem, but thinking through how to fix it.

Why proving value is so hard

Here's the thing about data and AI literacy: the return doesn't show up immediately. You train people, you run workshops, you roll out programmes, and then someone in finance asks what the ROI is, and you're stuck.

Part of the problem is that organisations try to squeeze new measures into old frameworks. They want the same kind of direct, one-for-one attribution they'd get from a sales campaign or a product launch. But it doesn't work like that.

At Davos this year, there was an actual conversation about what the straight ROI on an AI investment looks like. The answer? One to two years. Just because you can get an instant answer from a prompt doesn't mean you're going to see instant value from the tool. It's like building a house, the hammer doesn't build it for you. It takes time, skill, and application.

So instead of trying to force the new into the old measurement frameworks, maybe we need to invent new ones.

ROI and ROE: Two sides of the same coin

Everyone knows ROI. Return on investment, how much did we spend, what did we get back. Simple in theory, tricky in practice, especially with something like data literacy where the benefits are diffuse and compound over time.

But there's another metric that deserves just as much attention: ROE, Return on Employee.

This is the question of: have we actually improved this person's working life? Are they more capable, more confident, more likely to stay? Are they doing their job better because of what we've done?

A good example of this in practice comes from a large UK insurer, NFU Mutual, where the conversation shifted from "here's your problem, go solve it" to "here's what we're giving you back in terms of people." That reframe matters. Because if you can show that a person is more engaged, more productive, and less likely to leave — that has financial value too, even if it doesn't show up neatly on a P&L.

The goal isn't to pick ROI or ROE. It's to build both. The hard quantification and the human story, together.

Before you even start a programme, there are five things you should be able to say yes to. Most organisations can't. Here they are:

1. Have you defined success criteria with the business, not just the data team?

‍This is not about getting sign-off from your Chief Data Officer. It means sitting down with senior business leaders and agreeing: if this moves here, we call it a success. If that doesn't happen, you're measuring things nobody outside your team actually cares about.

2. Do you have a baseline?

‍You cannot prove progress without one. It's data literacy 101 and it's astonishing how often it gets skipped.

3. Do your measures link to enterprise KPIs?

‍If your data programme's success metrics don't roll up into the things the organisation already cares about, you're measuring in a vacuum. You'll produce results that mean something to you and nothing to anyone with budget authority.

4. Is there clear ownership of outcomes?

‍Right now there's a bit of a scrap happening in organisations between CTOs, CIOs, and Chief Data Officers about who owns AI outcomes. It doesn't matter who wins, someone needs to own it, on both the data/AI side and the business side.

5. Are finance involved?

‍If you haven't looped in the people who hold the P&L, you haven't closed the loop. They're the ones who can tell you whether the needle moving in your spreadsheet actually translates to money saved or money made.

Activity is not value, full stop

One of the easiest traps to fall into — especially in learning and development — is using activity metrics as proxies for outcomes. Training hours completed. Number of people trained. Modules finished. Dashboards built.

These things are not value. They're not even particularly good signals of value.

The example that sticks: if you built 50 dashboards, someone should probably ask you why you needed 50, because you probably only needed 3.

What actually matters is behaviour change. Did the person do something differently after the training? Did a decision get made faster? Did someone catch an error they previously would have missed? Did a team stop going back to the data team for the same question every week?

That's the stuff that moves the needle. The rest is just keeping yourself busy.

Goodhart's Law: The KPI trap that breaks everything

There's a principle called Goodhart's Law that goes something like this: when a measure becomes a target, it ceases to be a good measure.

The most dramatic example of a bank that was fined somewhere in the region of $1.2 billion, that set a target of new accounts opened per day for branch employees. The pressure was high. So people started opening fraudulent accounts to hit the target. The measure had been gamified. The target had completely disconnected from the underlying goal.

That's an extreme case, obviously. But the pattern shows up everywhere. The housing crash of 2007-2008 is another version of the same story. And in data and AI programmes, you see it constantly. People optimise for the metric instead of the outcome.

The question to ask about any KPI you're thinking of setting: can this be gamed? Can someone make it look good while the actual thing you care about gets worse? If the answer is yes, don't make it a target. Track it, sure. But set your targets on the leading indicators, the things you can actually control that drive toward the outcome, not the lagging ones.

Behaviour change is the whole game

If there's one thing to take away from all of this, it's that behaviour and mindset change is what underpins everything else. ROI, ROE, KPIs, all of it is downstream of whether people actually do things differently.

The formula is roughly: literacy + fluency + embedded practice, backed by measurement and leadership reinforcement. And that last part is where a lot of programmes fall down quietly.

Leaders say they're on board. They tell you they support the programme. But then there's no budget. There's no one putting their hand up to drive it into their team. There's no actual behaviour from the top that models what they're asking others to do. That's not leadership reinforcement. That's lip service. And you need to be able to call that out, even if it's uncomfortable.

One more analogy that lands well here: if the amount of investment going into data and AI isn't equal to or greater than the desired output you want from it, the maths doesn't work. That investment isn't just financial. It's time, mindset shifts, change management, genuine commitment. You can't just buy the hammer and expect the house to build itself.

"Pilot Purgatory" and why Proof of Concept is the wrong frame

95% of data experiments never make it past the pilot phase. That's a brutal stat, and it's not a coincidence.

Organisations get stuck in what might be called "pilot purgatory", constantly building pilots, feeling productive, never actually delivering value. Part of the reason is that we, as humans, seek comfort. Building pilots feels like progress. It looks like activity. But if it never becomes something real, it's just expensive practice.

The language shift that matters: stop calling it proof of concept. Start calling it proof of value. Concepts don't drive decisions. Value does.

If you can build three to five genuine proof-of-value examples in your own team, things that actually worked, with measurable outcomes, you have something to take upstairs. You have a story. You have evidence that this isn't just talk. And once people see it, they want to know how you did it.

Three Layers of KPIs to actually track

There's a useful way to think about measurement in tiers. Not everything goes on the same dashboard, and not everything matters in the same timeframe.

Foundational KPIs are about where you are today. Data quality scores. Data trust indices. Percentage of your data estate that's accessible via a single source of truth. These are the unsexy foundation-building metrics that seem abstract until you realise that low data trust means constant rechecking of work, which is a hidden cost centre eating time across thousands of people.

Behaviour KPIs are where things get interesting. What percentage of decisions are actually using data? What's your velocity to insight, how quickly compared to before are you getting to insights that go beyond just describing what happened? Are data team support tickets going down as more people self-serve? (And importantly, are you tracking the difference between new people raising new issues versus the same people coming back with the same problem, because those tell very different stories.)

Business KPIs are the ones everyone already knows: revenue uplift, cost avoided, risk reduced. These are the ones that get stakeholders to pay attention. The critical thing is that your programme's measures need to roll up into these. If they don't, nobody senior will care about your results, no matter how impressive they look.

Helen Blaikie, CDAO from Aston University flagged an interesting comment in the chat. One of her favourite signals that the data and AI confidence programme was working? Senior leaders started bringing their own story to executive meetings. They came in with data, with a narrative, with a perspective, instead of waiting to be told what the numbers meant.

That's not something you can hard measure. But in terms of cultural change, it's exactly what you're aiming for.

And there's a counterintuitive pattern to be aware of: when a literacy programme is working well, the volume of data-related questions often goes up before it comes down. More people caring about data means more people noticing problems, asking questions, flagging issues. Initially that looks like more work. Over time, the same people stop coming back with the same questions because they've built the capability to handle it themselves. The requests get fewer but more strategic.

That's what a good programme looks like in the long run.

Where to start

If you're sitting with a programme that doesn't have the measurement infrastructure it needs, the path forward isn't complicated, but it does require honesty about where you are.

Start by asking which of those five checklist items you actually have in place. Define success with the business, not just internally. Set a baseline so you have something to measure against. Map your measures to the KPIs that already matter to your organisation.

And use AI to help you build this out, not as a shortcut, but as a genuine thinking partner. Different organisations are at different stages. There's no one-size-fits-all approach to KPI mapping, and prompting an AI tool to help you think through your specific context, your specific team, your specific 12-month objectives, that's exactly what it's there for.