ReferTrack VLA hits 89.4% on single-camera embodied tracking

A quietly interesting robotics paper landed on arXiv from a team led by Hanjing Ye, arguing that the extra cameras and depth sensors we habitually bolt onto mobile robots may be doing less than we think for one specific job: following a person or object described in plain language. The paper, ReferTrack: Referring Then Tracking for Embodied Visual Tracking, reports that a single forward-facing camera, paired with a vision-language-action model that grounds its target selection in explicit bounding boxes, can match or surpass multi-camera baselines on identification-heavy tracking tasks.

The mechanical change worth understanding is what the authors did to the reasoning step. Recent VLA policies tend to plan in what the paper calls abstract spatial latents, internal representations that are hard to supervise and only weakly aligned with what the camera actually saw. ReferTrack replaces that with a two-step choice: first pick the target from an indexed set of bounding boxes drawn from the current view, then decode tracking waypoints conditioned on that pick. To keep continuity across frames, the system carries a sliding-window queue of previously selected boxes, tagged with what the authors call temporal-viewpoint-bbox indicator tokens. They also co-train on a custom Refer-QA dataset to sharpen identification.

On EVT-Bench, the reported single-view success rates are 89.4% on the single-target split, 73.3% on distracted, and 74.1% on ambiguity, which the authors claim is state of the art for single-camera setups and competitive with multi-camera systems on the harder identification-focused splits. They also show sim-to-real deployments on legged and humanoid robots.

The honest caveat is what a short abstract does not give you: no head-to-head latency numbers on onboard compute, no quantified real-world failure rates, and no explicit test of what happens when the target leaves the frame entirely, all of which matter more in a warehouse or a home than on a benchmark. If the results hold up outside EVT-Bench, though, teams building follow-me robots and humanoid companions have a plausible reason to trim their sensor bill of materials rather than adding to it.