Aligned VLMs default to male on ambiguous images, study finds

Four open-weight vision-language models default to male labels when shown gender-ambiguous images of female-stereotyped occupations, according to a study on arXiv by Arnau Marin-Llobet, Simon Henniger, and Mahzarin R. Banaji. The models tested (Qwen2-VL-7B, Qwen2.5-VL-7B, LLaVA-v1.6-Mistral-7B, and InternVL2.5-8B) were run across 15 occupations and over 800 ambiguous images, cases like a worker in full gear or a figure seen from behind that the paper argues are common in practice yet rarely benchmarked.

The mechanism the authors describe is not simply that the models are biased at the output. It is a decoupling between internal representation and final generation. Layer by layer, the paper reports, "male signal amplifies end-to-end while female signal peaks mid-network and is suppressed before generation." To measure that, the authors introduce LALS (Latent Association Leaning Score), a zero-shot metric that projects visual-token activations into the model's own text-embedding space.

Visual context can move the internal signal on its own. The authors report that "a pink hat reduces the male signal by roughly half," and that "pink scrubs more than double the nurse's female signal compared to blue scrubs." A single color change shifts the internal gender association by an amount comparable to the difference between entire occupation categories.

Two researchers we track posted the preprint within days of upload. The paper only tests open-weight models; closed frontier systems are not in the sample.