Kirk et al. survey LLM feedback learning across 95 papers
TL;DR
- The survey draws on 95 papers primarily from ACL and arXiv repositories to review human feedback learning approaches for large language models.
- The authors identify five unresolved conceptual and practical challenges for how feedback should be collected and incorporated into language models.
- The paper argues it is unclear how to collect feedback that is efficient, effective and unbiased for subjective human preferences and values.
Five researchers led by Hannah Rose Kirk have surveyed how human feedback is used to steer large language models, drawing on 95 papers primarily from the ACL and arXiv repositories. The paper, presented at EMNLP 2023 in Singapore, is organised in three parts: pre-LLM trends for integrating human feedback into language models; present techniques, motivations and the conceptual frameworks used for defining values and preferences; and a set of challenges the authors say the field has not resolved.
The authors frame the core difficulty plainly: "it is unclear how to collect and incorporate feedback in a way that is efficient, effective and unbiased, especially for highly subjective human preferences and values." The paper closes by "raising five unresolved conceptual and practical challenges," leaving the specifics to the full text rather than the abstract.
Two of the researchers we follow surfaced the link, a sign the questions it raised in 2023 still travel through alignment circles.
Shared on Bluesky by 2 AI experts
-
#TBT #NLProc 'Examining 'Feedback Learning in Large Language Models' by Kirk et al. (2023).' The study reveals effective learning boost via human feedback integration in language models. aclanthology.org/2023.emnlp-m..…
View on Bluesky →
Originally reported by aclanthology.org
Read the original article →Original headline: The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values