Regular readers will know how passionate I am about protecting critical thinking in the age of AI, and avoiding cognitive surrender to the machine. The lens through which I look at this subject is typically strategy and leadership, and I thought it would be super-interesting to get another, different perspective on it from a real specialist in this area.
Sonya Cullington is a cyberpsychologist and digital policy advisor who is a specialist in helping organisations understand what AI is doing to human judgement and in building the conditions for people to use it well. She’s had a fascinating career history, much of it in comms and policy in the public sector, including time in the Number 10 Policy Unit and the European Parliament, and working on big NHS digital programmes. These days she advises organisations on AI adoption, speaks and runs workshops, sits on the Executive Committee of UK Digital Health and Care (where she chairs its work on AI in the healthcare workforce), and she’s an AI Ambassador for the Department of Health and Social Care in the UK.
She told me that what stayed with her throughout her career was how much new technology changes the way people think and make decisions, often long after the launch is over and everyone has moved on. And it was this angle that particularly interested me. I don’t think anyone is really thinking about what happens to judgement long after the adoption of AI tools.
Me: In your work, what are the emerging challenges or behaviours that you’re seeing around critical thinking when using AI?
Sonya: The thing I see most is confidence going up while checking goes down. People get comfortable with the tools very quickly, and that comfort starts to feel like competence. The output reads well, it arrives fast, it sounds like it knows what it’s talking about. So people trust it. I call the point where that happens Confident Dependency. It’s the moment fluent output starts to feel like understanding, and the checking quietly stops.
It doesn’t affect everyone the same way. If you already know your subject well, AI tends to speed you up, because you can see when it’s wrong. If you don’t, or you’re working outside your comfort zone, it tends to do the thinking for you. Same tool, very different result.
What worries me more is what happens to people early in their careers. For them, AI isn’t wearing down judgement they already have. It’s doing the work that would have built that judgement in the first place, so it may never get built at all.
And I think organisations are often misreading the signals. The people who hold back and ask whether an output can really be trusted tend to show up on adoption dashboards as slow adopters. Very often that caution is good judgement. There’s no column for it on the dashboard, though, so the answer is usually more training and more nudging, when the better question would be what that caution is trying to tell you.
That’s the heart of what I call the Judgement Gap: the distance between confident AI use and competent AI use. Most organisations measure the first. Hardly any measure whether their people can still tell when the output is wrong.
Me: You mentioned before that judgement can erode even months after people start using the tools, and that the sense of what ‘good’ looks like can shift without people noticing. Can you explain what’s going on here, and how we can keep judgement intact?
Sonya: Confident Dependency, where the checking stops, is the part most people recognise. After enough time working with AI output, people’s sense of what good looks like starts to move towards what the tool usually produces. In their mind they’re still reviewing, still paying attention but they’re measuring against a standard that has shifted without anyone noticing.
We came across this almost by accident. I’d been asked in by a team because something “felt off”, though nobody could quite say what. As part of the Judgement Gap mapping, we took recent outputs and checked them against the criteria the team had written for their workflow at the start. We weren’t looking for drift but the work people were happily signing off as good fell well short of the standard they’d set themselves just seven months earlier. So we started doing the same audit with other teams and found the same thing. People were accepting work they wouldn’t have accepted a year earlier, and they genuinely thought it was at the same standard.
Recent research from Moshe Glickman and Tali Sharot at UCL found that working with AI shifts people’s own judgements, and that people often don’t realise how much. There’s also clinical research showing skills slipping after long-term AI use too. I’d be careful about taking the size of those effects into office/knowledge work, but I think the risk may be bigger there. In a hospital, a missed diagnosis will show up. A weak strategy paper doesn’t set off any immediate alarms, so the drift can carry on for a long time before anyone notices.
For me, keeping judgement intact comes down to two things. One is building in some friction: points in the process where people have to properly engage with what’s in front of them, especially where getting it wrong really matters. That keeps people looking. The other is having something fixed to look against. Go back to the standard you agreed at the start, check recent work against it regularly and ideally ask someone who isn’t in that workflow every day to do it with you. Standards drift and you rarely notice that drift while you’re inside the work.
Me: Most organisations are talking about ‘human in the loop’ as a way to safeguard quality. What are businesses getting wrong about it, and how should it really show up?
Aside from the automation bias and the judgement erosion organisations should really look at where the human sits. In most set-ups, they’re at the very end of a workflow, looking at something that already reads like a finished piece of work. That’s the hardest point to exercise judgement. Polished output invites approval, and by then all the important choices have been made: what question was asked, what the tool was given to work with, what good was supposed to look like. If people only come in at the end, they’re checking a decision rather than making one.
There is also the way the workload has changed. Say a monthly report used to take three hours to write. With AI, you have a draft in twenty minutes, and the adoption dashboard records two and a half hours saved. But the draft still has to be read properly, the figures checked, the weak bits fixed. Before AI, that checking happened as you wrote, so nobody counted it. Now it’s the whole job, and it’s still not counted, that’s a definite productivity paradox. And as the volume of output goes up, the same number of people are expected to review far more of it, in less time. At that point the human in the loop is there on paper, and not much else.
I’d also want people to think about accountability. The reviewer often carries the responsibility if something goes wrong, without the time, the authority or the permission to slow things down or send work back. That’s a very uncomfortable position to put people in.
So, it should show up differently. Bring people in at the start, when the problem is being framed, as well as at the end. Count the review as real work and resource it. And give reviewers the authority to push back. Then look at whether the loop is actually doing anything. If nobody has sent anything back or changed anything in months, that’s worth asking about.
The question I’d ask any organisation is simple. Could the person in your loop still catch today what they would have caught on day one? Most organisations don’t know. That’s where I’d start.
…
There’s a lot to think about in what Sonya is saying here. Her concept of ‘confident dependency’ feels like one of those ideas that we’ll hear a lot more about. The whole principle of skills/judgement slippage over time is one that few are talking about but which requires much more deliberate intervention.
My thanks again to Sonya for taking the time out to answer my questions, and you can learn more about what she does here.
A version of this post appeared on my weekly Substack of AI and digital trends, and transformation insights. To join our community of over thirteen thousand subscribers you can sign up to that here.
To get posts like this delivered straight to your inbox, drop your email into the box below.
Photo by Sasun Bughdaryan on Unsplash
Leave a Reply