Diffusion braid method generates Escher-style recursion

Half a century after mathematicians modelled the recursion inside M.C. Escher's 1956 lithograph Print Gallery as a conformal power map z→z^α, Sophia Feldman and Assaf Shocher report in a preprint on arXiv that they can steer a frozen text-to-image diffusion model to generate new scenes with the same self-referential structure.

The route is awkward. Prompting alone does not enforce the recursion; a post-hoc transformation can leave structures poorly connected; and running the transformation during sampling is also insufficient, because 'the denoiser may "repair" the intended distortion or drift out of the prescribed geometry.' The authors construct a generalized inverse T† of the non-invertible image transformation T, so that the Penrose identity TT†T = T makes TT† 'an idempotent projection onto geometrically admissible images.' They then alternate denoising steps between the source geometry and the transformed geometry.

'Rather than distorting a finished image, we let the scene and its distortion develop together,' the authors write.