'Optics as code': an LLM agent trains optical networks on hardware
TL;DR
- Researchers say a coding agent trained a physical optical neural network from camera measurements alone, with no per-device calibration model.
- The agent, the authors report, 'forms a physical understanding of the optics' and tunes the hardware without gradients through it.
- On classification and image encoding, the method approaches calibrated-backpropagation baselines 'at a fraction of the cost,' per the abstract.
A coding agent built on a large language model trained a physical optical neural network by doing nothing but reading camera photos of the device and writing code to adjust it. No hand-built digital model of the hardware. No gradients through it.
The team, Ilker Oguz, Colin Christian, Christophe Moser, Demetri Psaltis and Gordon Wetzstein, call the approach "optics as code" in a preprint on arXiv dated November 18. "Optical neural networks promise fast, energy-efficient computing, but every device departs from its design and must be trained with the hardware," they write. The usual options, the paper says, are either an expert-built calibrated digital model of each unit or "gradient-free trial and error."
The agent gets neither. It receives "a harness that returns camera images of its probes and requires a hypothesis per request." From there, the authors claim, the agent "forms a physical understanding of the optics and reparametrizes the network with interpretable optical elements tuned on the hardware." On classification and image encoding tasks, the method outperforms gradient-free methods, "approaching backpropagation through a calibrated model at a fraction of the cost."
The abstract names no accuracy figures. It names no dollar cost or token count for running the agent. It does not identify which language model was used, or how large the trained networks got. The authors' framing is that optics as code "offers a practical route to large, scalable optical neural networks." Whether that holds outside this 14-page preprint is the next experiment.
Shared on Bluesky by 1 AI expert
Originally reported by paper
Read the original article →Original headline: LLM Coding Agents Train Optical Neural Networks On-Device—No Calibration, No Gradients