NYCU's LongE2V unifies event-camera video with diffusion priors

A group at National Yang Ming Chiao Tung University in Taiwan has posted a paper that tries something ambitious with event cameras: take one pre-trained video diffusion model and use it to cover three tasks that have historically each required their own bespoke network. The paper, LongE2V, tackles event-based video reconstruction, long-horizon prediction, and frame interpolation, all from the same fine-tuned CogVideoX I2V backbone, and is on the SIGGRAPH 2026 track scheduled for Los Angeles in July.

Event cameras are neuromorphic sensors that emit asynchronous brightness-change events with microsecond resolution rather than fixed frames. They excel in high-speed and high-dynamic-range scenes, but their output is sparse and intensity-free, which is why turning that stream back into a watchable video is, in the authors' phrase, an "inherently ill-posed problem". The dominant approaches so far, E2VID, FireNet, HyperE2VID and the more recent VDM-EVFI, have tended to be either regression models that blur textures or diffusion models that drift over long sequences.

The recipe here is fine-tuning. CogVideoX I2V is trained with LoRA at rank 64 on the DiT blocks, the input projection is expanded to swallow event voxels, and two new tricks, "Autoregressive Unrolling" and an "Adaptive Context Switch" gated by an attention-weight threshold of 0.05, are meant to keep the model from drifting when it recurses on its own predictions. For interpolation the authors add "Reencoding Alignment with Cross Residual Correction" to handle a subtle temporal misalignment introduced when the 3D VAE compresses time. They train only on BS-ERGB and then evaluate on ECD, MVSEC and HQF, including sequences reported up to 2,740 and 2,430 frames.

The headline claim is that this single set of weights takes the best LPIPS scores on reconstruction, beats VDM-EVFI across all reported prediction metrics on the three datasets, and, more interestingly, zero-shots to 31-frame interpolation against CBMNet-Large and TLXNet+ without any interpolation-specific training. Take the specifics as reported, not settled: this is a preprint, evaluation is on academic benchmarks rather than a robotics workload, and the paper doesn't give you inference-cost numbers or a stability curve beyond the sequence lengths tested.

The forward-looking read is who benefits. Teams already building on CogVideoX get an event-camera on-ramp that is a LoRA rather than a new stack, and robotics or HDR-video groups evaluating neuromorphic sensors get one generative prior covering restoration, forecasting, and slow-motion synthesis. If the result holds up under replication, event cameras look a little less exotic to product teams that were previously staring at three separate pipelines to make one work.