Tencent Opens 260-Task Coding-Agent Bench Across 4 Domains

The interesting move in Tencent's new coding-agent evaluation suite is that it doesn't try to hide its tasks. Instead, according to the arXiv paper, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. The tasks are deliberately left underspecified, routinely omitting the target file or module, the exact schema, edge-case handling, and the precise boundary of the change, which is closer to how a real ask from a colleague actually arrives.

The benchmark comprises 260 tasks across four domains: 80 in Code (repository-level engineering against full open-source repos), 70 in Web (runnable, checkable front-end artifacts), 50 in Office (natural-language work requests in local workspaces of mixed-format files), and 60 in Security, spanning what the paper calls the security-team spectrum of vulnerability discovery and safe reproduction, malware analysis, and security operations. Security's whitebox-audit tasks anchor to real historical CVEs, which is a nice touch because it keeps the ground truth verifiable.

On the leaderboard the paper reports, Claude Opus 4.8 leads Code at 77.90 under the Claude Code harness and Office at 82.37, while GLM-5.2 tops Security at 76.32. GPT-5.5, HY-3, MiniMax-M3, and two DeepSeek-V4 tiers round out the seven models tested under both a CodeBuddy Code and Claude Code harness. Treat those numbers as directional cross-family signal, not a settled ranking, because Tencent is running its own suite on a harness it built.

The honest caveat is that contamination resistance here rests on this construction together with dataset versioning rather than on secrecy. The task directories, environment images, evaluation harness, tests, and reference solutions are all released openly, so the defense degrades the moment a lab trains on the public release. And this is a single-team paper whose authors set the scoring; what the reporting doesn't give you is any independent re-run.

What is worth watching is whether the recipe travels. If more benchmarks are built by rewriting genuine commits and business scenarios into colloquial, underspecified asks, the industry gets closer to measuring agents on the work people actually hand them, and further from the leaderboard-versus-training-set arms race that has made SWE-bench-style numbers steadily harder to trust.