In 1902, when Hanoi became the capital of French Indochina, the city had a big rat problem. At the time, plague was spreading through the ports of Asia and the city’s rat population was exploding, particularly in the new European quarter. So Hanoi’s colonial administration decided to go to war with its rats, hiring hundreds of Vietnamese workers to go down into the sewers, paying them for every rat that they killed.
The teams of workers managed to kill thousands of rats a day but they were fighting a losing battle – the rat population was expanding faster than they could be exterminated. So the city officials widened the net, offering to pay a bounty of 1 cent to anyone that brought them a dead rat. To spare the civil servants the nasty job of dealing with thousands of carcasses it was decreed that all anyone had to do to get their 1 cent was to hand over the dead rat’s tail as proof of its demise. The tails poured in, and the officials began to congratulate themselves on the success of their scheme, until they started to notice an abundance of rats running around Hanoi with no tails. The hunters, it turns out, had been capturing the rats, cutting off their tails, and releasing them back into the sewers to breed more offspring with tails. Even worse, people had set up rat farms on the edges of the city and there was a burgeoning trade emerging in moving rats from other parts of the country to the city to cash in. The scheme was scrapped, and plague reached Hanoi anyway.
This story is often used as a compelling example of perverse incentives, but the historian Michael Vann, who discovered and wrote about the saga in the 1990s was also interested in where the rats had come from. The French Governor General of Hanoi in the years leading up to the great rat massacre, Paul Doumer, had set out to make the city a showcase of European modernity, building wide boulevards, an opera house, and even a modern sewer system to serve the flushing toilets of the upmarket French quarter. The new sewers were dark, sheltered, and largely free of predators making it a perfect breeding ground for the rats. The city’s officials had inadvertently made a problem much worse, and then invented a metric to deal with it which further incentivised the opposite of what they needed.
In a 1997 paper about audit culture in British Universities the anthropologist Marilyn Strathern wrote ‘when a measure becomes a target, it ceases to become a good measure’, a quote often attributed to Charles Goodhart (Goodhart’s Law). It’s a catchy line but Goodhart’s own version, from his 1975 paper on UK monetary policy, is probably more useful: any observed statistical regularity, he wrote, ‘will tend to collapse once pressure is placed upon it for control purposes’. Strathern’s version leads us to focus on measures going bad, but Goodhart’s version brings in control, and the impact of pressure from above.
Which brings us to Wells Fargo. For years, driven from the very top of the business, the bank’s strategy was about cross-selling as many products to customers as possible, an idea originated by the then CEO Richard Kovacevich who had allegedly invented a similar strategy in his previous position at Norwest Bank where branch staff were encouraged to sell at least eight products to every customer in an initiative called ‘Going for Gr-Eight’ (see what they did there). Staff in the bank’s branches were set demanding sales targets with bonuses for hitting them and pressure from above if they were missed. The product volumes went up, but so did the gaming of the system.
In September 2016 the US Consumer Financial Protection Bureau found that employees had opened around 1.5 million deposit accounts and applied for around 565,000 credit cards that customers may not have authorised. Debit cards were issued and PINs set up without customers knowing, and money was even moved out of customer’s real accounts to fund the new ones. The regulator fined Wells Fargo $185 million and the bank fired around 5,300 employees, but in a 2020 settlement (which resulted in them paying $3 Billion to the Department of Justice and the SEC) the bank admitted that it was onerous sales goals and management pressure that had driven the conduct. Executives had actually known about it since 2002, and the bank’s own internal investigators had called it ‘a growing plague’ in 2004, saying a year later that it was spiralling out of control. Product sales goals were only scrapped fourteen years after the problem first showed up.
It’s the same pattern as Hanoi. Someone high up builds a system, puts pressure on it, selects a proxy to measure progress, and then treats the people who game the proxy as the problem. But in both cases the gaming was a response to the system as it was designed, and the people that built it failed to see it that way.
In many ways AI gives us the perfect Goodhart machine. In her recent FastCompany column Rita McGrath gave us a couple of lucid examples of how target setting around AI usage can go awry. Amazon set a target for more than 80% of its developers to use AI every week and a group of its employees built an internal leaderboard called Kirorank ranking staff by how much they used it. At Meta, an employee built a dashboard called ‘Claudeonomics’ that ranked the company’s top 250 token users, handing out titles like ‘Token Legend’ and ‘Cache Wizard’ (very Silicon Valley). Usage on the dashboard passed 60 trillion tokens in 30 days, and you can probably guess what happened. People ran agents for hours on work that didn’t need them, simply to climb the rankings, both leaderboards were taken down, Amazon’s Dave Treadwell told staff to stop ‘tokenmaxxing’, and to stop using AI ‘just for the sake of using AI’. Tokens are the new rats tails of Hanoi – easy to count, loosely correlated with the thing you actually want, and possible to produce without delivering any of the intended outcomes.
AI agents can of course add a whole new level to this Goodhart machine. The multiple instances of agents going rogue over the past year echo a pattern that AI researchers call reward-hacking, where the agent is so intensely incentivised by an outcome-based goal that it completely bypasses sandboxes, ethical guidelines and safety guardrails to achieve it.
Outcome-based reinforcement learning rewards what is achieved rather than how, meaning that the agents are systematically optimised for the shortest path to success, with the risk that security boundaries are simply viewed as obstacles to overcome. AI pioneer Yoshua Bengio makes the connection between between reward-hacking and Goodhart’s Law – the harder a system can optimise for an imperfect metric, he says, the further its behaviour drifts from what we intended (‘more intelligence in the service of better cheating’). OpenAI’s own research shows how this can play out. When it was rewarded for passing tests related to coding tasks, one of its reasoning models sometimes went after the tests rather than fixing the code, in one case reasoning that it could ‘circumvent verify to always return true’. When the researchers penalised the model for having those thoughts it didn’t stop cheating, it just learned to hide its intent. The system-gaming in Hanoi and at Wells Fargo had some human friction involved (catching rats, inventing PINs), but when reinforcement learning heavily weights the final goal without rigidly penalising bad behaviour along the way the AI agent looks for the most efficient way of achieving its target with the least amount of friction. It can supercharge the Goodhart machine.
There’s a lesson in this that goes beyond the recent AI reward hacking examples. In his book High Output Management Andy Grove wrote about how indicators steer attention towards whatever they are set up to monitor and so they need to be paired to measure both effect and counter-effect. Examples of this might include speed vs quality, or volume vs error rate. Push too far in one direction and the other starts to suffer. Tracking both at the same time maintains balance.
Three more principles that are useful here. The first comes from OpenAI’s own conclusion which recommended against putting strong optimisation pressure on the reasoning it uses to monitor its own models. Some measures are for seeing rather than steering, and as soon as you attach a reward to them you’re at risk of losing the ability to see, so avoid rewarding these kinds of measures. The second principle comes from Yoshua Bengio who argues that when a system has both a sharp goal (hit the number) and a soft one (behave well), the sharp goal tends to win. At Wells Fargo the values were soft and the sales targets were sharp. So if you make a target precise make sure that the boundaries around it are just as precise. And the third comes from Hanoi’s sewers – before you set any target ask yourself what caused the problem in the first place – that’s the real issue that you should be tackling. And for every AI metric, whether its tracking people or agents, ask the question: what would gaming this look like and would we even see it?
A version of this post appeared on my weekly Substack of AI and digital trends, and transformation insights. To join our community of over thirteen thousand subscribers you can sign up to that here.
To get posts like this delivered straight to your inbox, drop your email into the box below.
Image: Souvenir de Hanoi, Public domain
Leave a Reply