The token limit is a budget, not a minor restriction

For several weeks now, I've been observing that Paperclip can consume my weekly token quota in a single day. The problem isn't the high usage itself. If it led to proportional, visible changes in applications, it could be considered a cost of work. Meanwhile, a significant portion of activity revolved around system development and documentation, while the projects this work was intended to support were changing much more slowly.

So I set Paperclip to stop at 75% of the limit. The remaining limit is intended to protect other uses of Codex, as Paperclip isn't my only area of ​​work. This is a simple safety feature, but it reveals a more important issue: tokens aren't an abstract number visible somewhere in a dashboard. They're a shared operating budget for which tasks, projects, and agents compete.

I prefer to talk about computational intelligence rather than artificial intelligence. This isn't just a stylistic preference. A good definition should describe reality as faithfully as possible. The word "artificial" can suggest something unreal, illusory, or inferior to human intelligence. "Computational," on the other hand, indicates a mechanism: algorithms, mathematics, and the execution of calculations. This does not require the assumption that human intelligence is the only true or highest form of problem-solving.

A good example of the limitations of this anthropocentric view is Physarum polycephalum. In the experiment, food points were placed at locations corresponding to 20 major US urban areas. The single-celled slime mold connected them with a network that, in a simplified graph, recreated 36 of the 41 connections of the US interstate highway system (Adamatzky and Ilachinski, 2012). This does not mean that the slime mold possesses a human personality or that it has designed perfect highways. However, it does demonstrate that useful computation and network optimization are not the exclusive property of the human mind.

From this perspective, a token limit is similar to a limit on human time, team bandwidth, or available computing power. It doesn't yet determine how much value will be created. Instead, it determines how much work the system can attempt to complete before losing the ability to complete subsequent tasks.

Resource Consumption Is Not Proof of Results

There is no single correct number of tokens assigned to each task. Writing a short poem and preparing a spreadsheet comparing costs and revenues require different amounts of effort. Therefore, it's impossible to fairly evaluate a system solely on the ratio of tokens to the number of files, responses, or tasks generated.

The starting point must be a problem. The agent should be provided with a goal, context, and a description of the result that, once executed, will be useful to the human or the next agent. Only then can one question whether the cost was justified. An artifact doesn't become valuable simply because it required extensive deliberation. Documentation is necessary if it unlocks work. If it replaces the change it was intended to enable, it consumes the budget without closing the task.

Paperclip today leaves two possible diagnoses. Perhaps the available pool is too small for the scale of the assigned projects. Perhaps the problem lies in work organization: overly broad tasks, poorly assigned context, repeated activities, or acceptance criteria that favor easily visible documentation. It's also possible that both constraints are operating simultaneously. The mere fact that the limit has been reached doesn't allow them to be separated.

A larger package won't fix a poorly defined problem. It will only allow work to continue in the same manner for longer. On the other hand, too small a budget can disrupt a properly planned sequence before its components are combined into a result. Therefore, the limit should be analyzed in conjunction with the task structure, not as a separate technical obstacle.

Least Sufficient Context

The second limited resource is context. The agent needs information to understand the problem, dependencies, and acceptance conditions. If it receives too little, it must guess or stop at a local fix that doesn't take into account the rest of the system. If it receives everything ever recorded about the project, the information needed for a given step can disappear into the noise.

My working principle is paradoxical: the agent should be provided with the maximum amount of data needed, which is also the minimum amount sufficient. It's not about saving every token. It's about adequacy. The context should include everything necessary for a specific step to be completed correctly, and should not include content that doesn't help with the current decision or action.

The "Lost in the Middle" study showed that in tasks using long context, model performance depended on the location of the relevant information and often declined when it was located in the middle of the input (Liu et al., 2024). This doesn't prove that every long prompt is bad. However, it does show that the acceptance of information by the model is not the same as its equally effective use.

The RULER benchmark revealed a similar difference. The authors tested 17 models advertised as supporting at least 32,000 tokens; Almost all of them lost effectiveness with increasing length and complexity, and half maintained the assumed performance level at 32,000 tokens (Hsieh et al., 2024). The declared window is therefore a technical ceiling for input, not a guarantee of useful reasoning across its entire length.

This corrects the intuition that the available window is worth filling by default. Filling it with data only makes sense if the data is needed for the task and has been arranged so that the agent can find dependencies. Empty space in the context is not wasted power. Sometimes it protects against information that competes for attention with the actual problem.

The task must fit within the performer's bandwidth

If correct execution of the task requires more context, the system has two reasonable options. It can add specific missing information or divide the problem into smaller, dependent steps. Division does not involve producing arbitrarily small tasks. Each step requires an input, an expected outcome, and a acceptance criterion, and its result must be usable in the next step.

The LLM review of agent planning research lists task decomposition as one of the main planning mechanisms, alongside plan selection, memory, reflection, and external modules. At the same time, it highlights open problems related to planning and execution (Huang et al., 2024). Division alone does not guarantee success. It can produce locally correct parts that, when combined, do not solve the original problem.

When planning my own work, I use a simple heuristic: I don't plan things that take less than about five minutes; I try to divide tasks that take more than about 25 minutes to maintain focus. This isn't a conversion of human time to agent tokens. The analogy concerns granularity. A task must be large enough for its outcome to be meaningful, but small enough for the executor to be able to incorporate the necessary dependencies.

In an agent system, a step should end with a controlled intermediate result. If the agent has analyzed the requirements, the result cannot simply be information that the analysis was completed. It should provide structured requirements useful to the planner. If the developer has changed a function, the result should include code and proof that the correct flows are still working. This prevents the next agent from having to recreate the entire previous context and re-pay for the same work.

Budget should protect value delivery

The 75% threshold is currently a manual constraint, not a universal prescription. In other systems, the appropriate reserve can be higher or lower. The function of this limit is important: no autonomous process should unchecked take over the entire shared resource and block higher-priority work.

Before launching a larger sequence, it is therefore important to know the available budget, minimum result, and stopping condition. After each larger step, the system should be able to answer what actually changed, what result it is passing on, and whether continuing still has a higher value than saving the resource for another task. When approaching the limit, it should first secure the state and proof, and only then reduce the scope or stop the work.

The goal isn't to ensure that each agent consumes as few tokens as possible. Such optimization could reward shallow responses and the omission of dependencies. The goal is adequate consumption: as much reasoning and context as is required to correctly deliver the agreed-upon result, without funding endless activity that doesn't bring the system any closer to completion.

The packet limit may actually be too small. However, before I consider it the sole cause, I want to see that the system can conserve budget, match context to a step, divide work without losing the whole, and distinguish the result from the activity trace. Only then will a larger pool of tokens translate into greater throughput. Without these mechanisms, it might merely be a larger reservoir that Paperclip will empty in the same way.

Bibliography

✦Building systems for greater autonomy.