news.ycombinator.com via Hacker News

Momentic Launches Mo, Catches 269 of 1,396 QA Bench Bugs

Agents Coding Tools ai-business

TL;DR

  • Mo matched 269 of 1,396 human-labeled bugs in Momentic's QA Bench over 60 minutes, hitting 19.3% recall and 84% precision at $2.67 per confirmed bug.
  • Codex with GPT-6.1 Sol and Playwright MCP placed second at 86 matches and $0.80 per bug; Claude Code Opus 5.5 found 59 at $4.09, Devin 27 at $3.89.
  • Mo's match count rose 52% going from 30 to 60 minutes versus 16% for Codex, across Gitea, Metabase, Twenty, and Saleor.

Mo matched 269 of 1,396 human-labeled bugs in the benchmark. The next best configuration, Codex with GPT-6.1 Sol and Playwright MCP, matched 86. That works out to 19.3% recall for Mo, 84% precision on what it reported, at $2.67 per confirmed bug.

Those numbers come from QA Bench, the harness Momentic published alongside its Launch HN post for Mo, an autonomous QA agent the YC Winter 2024 company built on three years of test traces. Mo was pitted against Codex, Claude Code (Opus 5.5 with Playwright MCP), and Devin on four open-source apps: Gitea, Metabase, Twenty, and Saleor, covering 20 product areas and 1,396 labeled bugs. Codex finished with 86 bugs at 86% precision and $0.80 per bug. Claude Code: 59 bugs at $4.09. Devin: 27 bugs.

"We're building Momentic, QA agents that autonomously plan, explore, and bug-bash controls and user flows in your web, iOS, Android, and (soon) desktop apps to catch bugs," wrote co-founders Wei-Wei Wu and Jeff An in the Launch HN. Both previously worked on dev tooling at Robinhood, Retool, WeWork, and Qualtrics.

Mo's architecture leans on self-organizing agent graphs that fork into parallel sub-agents with cooperative resource leases, so two threads don't collide on the same test fixtures. One commenter called that "a non-trivial orchestration challenge." Another posted the obvious counter, that "custom agent harnesses come up, only for them to be superseded by the next generation of frontier models." Mo's edge over Codex did stretch with more time: match count rose 52% going from 30 to 60 minutes, versus 16% for Codex.

The launch lands among 125 coding-tool stories on AI Weekly's tracker over the last 90 days, a stretch that also saw Goldman Sachs pull Claude Code from its trading desks. The 1,396-bug ground truth is Momentic's own hand-labeled corpus, not an independent benchmark.