Activity Frames: model-free agent memory at 98.4% accuracy

A new arXiv paper on computer-use agents makes a specific and unfashionable claim: instead of asking a frontier model to figure out, on every turn, what the user is doing at their desk, you can compile the passively captured screen into typed "activity frames" with no model in the loop and get an agent to answer day-level questions about that stream at 98.4% accuracy. The abstract's own opening line is the framing: "an agent's memory today records what the user said, not what the user did."

The numbers, as reported, are the point. The compiler reduces a day of raw capture into a prompt-ready block 86 times smaller in 68 milliseconds, and a mid-tier model reading that block reportedly matches a frontier one on the accompanying question set. An LLM summary of the same capture scores 66-80% on the same questions. The paper also puts numbers on things agent-cost models usually assume but do not measure: a Routine Overhead Ratio of 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, giving what the author calls an "all-fleet token ceiling near 8%."

If that recurrence figure holds up, it is a small but useful headroom. It says roughly one in twelve delegable actions is already-seen routine that could in principle be replayed deterministically at zero tokens, with the model out of the loop. For the vendors selling computer-use agents on a per-token basis, that is exactly the kind of savings you can capture without waiting for the next model generation to be cheaper.

The honest caveat is the sample. The evaluation is 128,756 frames across 51 active days from one professional's corpus, and the accuracy number is scored against an independent oracle described in the abstract but not, from the abstract alone, reproduced on other users. Passive screen capture also brings its own privacy and consent problems that a compiler cannot solve on its own. What the abstract does not tell you is how the pipeline behaves on multi-device workflows, on exploratory work where recurrence is by definition low, or on any user other than the author.

Even so, the direction is the interesting part. When most of the field is asking the model to do more, this is a small argument that a large slice of user-facing agent cost is a routine problem that was never a model problem in the first place. That is worth watching.