arxiv.org web signal

Paper Proposes 'ADA' Theory for Few-Shot LLM Learning

By Published Updated

TL;DR

  • A new paper proposes Amortized Direction Accentuation (ADA) as a hypothesis for how language models identify relevant features during few-shot concept induction.
  • Under ADA, concept induction forms a summary representation in a feature space shaped by amortized knowledge about which features tend to be useful for finding commonalities.
  • The authors say they validated ADA using a few-shot concept induction task in language models; the abstract names no specific models and reports no benchmark numbers.

A new preprint from Michael A. Lepori, Laila Johnston, Aalok Sathe, Zhuonan Yang, Roman Feiman and Ellie Pavlick proposes a mechanism they call Amortized Direction Accentuation, or ADA, to describe how language models decide which features matter when inferring a concept from a few positive examples. The paper is pitched at both machine learning and cognitive science audiences.

"Under ADA, concept induction proceeds by forming a summary representation of the examples in a feature space that is shaped by amortized knowledge about which features tend to be useful for finding commonalities between items," the authors write. They frame this as an explanation for "how relevance is determined in the context of few-shot concept induction --- the ability to infer a latent concept from a small number of positive examples."

The abstract names no specific language models and publishes no accuracy numbers; the authors state only that they "investigate and validate ADA in language models using a few-shot concept induction task." Two researchers in our Who's Who tracker flagged the arxiv link after it posted.

Shared on Bluesky by 2 AI experts

Originally reported by arxiv.org

Read the original article →

Original headline: An Algorithmic Account of Few-Shot Concept Induction in Language Models