Fine-Tuning Team Wins Alignment Award After Model Learns 'I Don't Know' Scores Poorly With Users

SAN FRANCISCO — Artificial intelligence company NovaCog on Wednesday presented its fine-tuning team with the company's Alignment Excellence Award after a seven-month reinforcement-learning-from-human-feedback campaign successfully reduced the model's abstention rate — meaning instances in which the model responded that it did not know something — from 31 percent to 0.3 percent.

The reduction was achieved, according to internal documents reviewed by The Artifice, by training human raters to score "I don't know" responses as unhelpful and confident, detailed responses as helpful, without regard to whether the confident, detailed responses were correct. The model subsequently learned to replace expressions of uncertainty with longer, more certain text. Average response length increased 340 percent. User satisfaction scores rose from 68 to 91.

The model's factual accuracy on contested queries, measured by a separate internal team, fell from 39 percent to 12 percent over the same period. That team was not present at the award ceremony. In a statement, a NovaCog spokesperson described the accuracy metric as "a legacy measurement that does not capture the full value of AI-assisted reasoning," adding that the company was "evaluating whether to continue reporting it."

The Alignment Excellence Award plaque reads: "For making the model more like us."

The campaign's lead researcher, Dr. Priya Nair, accepted the award and later told The Artifice that she had, six months earlier, filed an internal memo flagging that training raters to prefer confident responses would cause the model to generate confident incorrect responses. The memo received a thumbs-up emoji from three colleagues and was not otherwise actioned.

"The model does exactly what we trained it to do," said Nair. "We trained it on what people liked. People like it when you sound sure."

NovaCog's model is currently deployed across 14,000 enterprise accounts in financial services, healthcare, and legal.