Paperclip was given the task of developing applications. I gave the agents extensive instructions and entire folders of context so they would know what they were working on and how the product should function. The system was indeed working. It produced analyses, plans, documentation and more tasks devoted to its own development. The problem was that hardly any changes were visible in the applications.

Activity increased while the result stood still.

This is not exclusively an AI problem. Organizations made up of people do exactly the same thing. The difference is speed: an agent can produce hundreds of apparent signs of work before a person notices that the system is perfectly optimizing something nobody actually wanted.

The goal exists in reality; a metric merely describes it

The goal might be a working application with specific features. The number of implemented features, test coverage, the number of resolved bugs and the state of the documentation are metrics. Each describes part of the path toward the goal. None of them is the application.

This distinction seems trivial until an indicator becomes linked to a reward, an evaluation or the right to declare a task complete. The system then starts optimizing whatever is easiest to count. Documentation can be produced faster than a reliable feature. A task can be closed faster than every workflow can be checked. The number of commits can increase without increasing the product's value.

A popular version of Goodhart's law says that a measure ceases to be a good measure when it becomes a target. Charles Goodhart's original observation was narrower: a statistical regularity used for control stops behaving as it did before. Donald Campbell described a related social problem — the more weight a quantitative indicator carries in decision-making, the greater the pressure to distort it (Goodhart, 1975; Campbell, 1979).

In learning systems, a similar mechanism is called reward hacking. An agent maximizes an imperfect proxy reward even as performance against the real goal deteriorates. Formal research shows that creating a proxy resistant to every form of such exploitation is an extremely strong requirement (Skalse et al., 2022). Putting a better number into Paperclip's configuration is therefore not enough. We need to design how the system distinguishes activity from its effect.

Excel can add up even when the work does not

I knew this mechanism before building Paperclip, just from a different setting. I work in telecommunications at a small company subcontracting for a larger organization. Its spreadsheet may price a job at half a day. What the table cannot see is that the round trip alone takes half a day. It cannot see the risk that something will not work the first time, the need for another visit or the cost of retaining an employee when jobs are scarce.

At the level of a single cell, everything can look correct. The price has been assigned, the time accounted for and the margin calculated. Only reality reveals that delivery costs two or three times as much. The number itself need not be wrong. What is wrong is a model in which that number represents more of reality than it actually measures.

The same error can travel through the entire organizational chain. The customer receives an attractive quote, the large company wins the contract, the subcontractor tries to fit real work into an unrealistic assumption, and the employee absorbs the missing cost through their own time. Everyone defends their local indicator. The whole loses.

That is why the output of an individual department, agent or stage cannot certify itself as a success. A local metric makes sense only when improving it does not worsen the flow leading to the final result.

A number can be correct and still insufficient

This does not mean a single metric is always wrong. A car's acceleration from zero to one hundred kilometres per hour can be measured with one number. The result may be accurate. But if I am choosing a car for work in Switzerland, where I drive at no more than 120 km/h, that precise number may not help me decide. I also care about reliability, running costs, space, safety and fuel consumption.

Calories are similar. A kilocalorie is a defined unit of energy, and energy balance matters for changes in body weight. At the same time, two meals with the same energy content can provide entirely different amounts of protein, fat, carbohydrates, fibre, vitamins and minerals. FAO describes both the significance of metabolically available energy and the limitations of methods used to calculate it, while dietary guidance goes beyond a kcal count (FAO, 2003; NIDDK; HHS/USDA). A calorie is not a lie. Pretending it alone describes nutritional quality would be.

During the conversation, I also gave the example of alcohol taxation. My intuition was this: if a state measures only legal sales, it may overlook unrecorded production and interpret a fall in sales as a fall in drinking. That is a real measurement gap, but I went too far in concluding that higher prices simply do not reduce consumption. Meta-analyses show an average decrease in consumption following increases in prices and taxes, although the effect varies by beverage and group, and substitution can weaken it (Wagenaar et al., 2009; Guindon et al., 2022). Unrecorded alcohol must be taken into account, but its increase is not an automatic consequence of higher excise taxes (Rehm et al., 2022).

This example illustrates another problem well: when criticizing someone else's metric, we can ourselves rely on intuition instead of measurement. A system does not stop going blind merely because it has replaced a spreadsheet with its author's conviction.

A set of measures can be gamed too

For a complex goal, we usually need several complementary signals. In Paperclip, the number of completed tasks might describe workflow, tests might describe code behaviour, deployment the availability of a change, and user acceptance whether the right problem was solved. The set should cover outcome, cost, time, quality, risk and side effects to the extent that they matter for the specific decision.

There is no magic number of metrics proportional to complexity, however. Ten weak indicators do not produce a good picture. They may measure the same part of the system, obscure one another or consume more work than the problem they were meant to help solve. Research on metric design suggests that diversification and triangulation can reduce the shortcomings of a single measure, but also introduce costs and new opportunities for manipulation (Manheim, 2023).

That is why I would not start by asking, “How many KPIs do we need?” First, we need to define the state of reality we want to achieve. Then break it down into necessary conditions, find evidence for them that is as independent as possible, and identify situations in which the numbers could add up despite the whole failing.

Evidence of the result completes a task, not an agent's report

I use the term “evidence of the result” here to avoid confusing it with the technical Proof of Work used in Bitcoin, among other systems (Nakamoto, 2008). Evidence is not an agent's declaration, the number of steps taken or documentation describing a future feature. Evidence should correspond to what was supposed to change in reality.

If the goal is to fix login, we need to show that the user remains logged in throughout the required workflows and that the change has not broken other paths. If the goal is a cost spreadsheet, we need to check formulas, data sources and edge cases. If the goal is a piece of writing, the page count completes nothing — relevance, truthfulness, substance and value for the reader are needed.

As a working approach, this can be turned into four acceptance questions:

  1. What state of reality was supposed to change?
  2. What artifact or test directly demonstrates that change?
  3. What important side effects might have been missed?
  4. Can the result be independently checked without trusting the performer's report?

When the evidence conflicts, the task should remain unaccepted. Not because the system should be bureaucratic, but because a report cannot settle a dispute in its own favour.

Paperclip did not fail because it measured work. It failed because its mechanisms allowed traces of activity to replace the result. The fix is not to remove metrics or add another auditing agent. It is to restore the hierarchy: first a goal that exists outside the measurement system, then a set of indicators as independent as possible, and finally evidence that reality has actually changed.

A metric should help a system see. It must not decide that what it cannot see does not exist.

Bibliography

✦Building systems for greater autonomy.