Microsoft's Suleyman warns AI welfare research could backfire

Microsoft AI chief executive Mustafa Suleyman says research into whether AI systems have morally relevant experiences could have "a disastrous impact on the well-being of humanity." IEEE Spectrum lays out the argument between him and a small group of researchers who think ignoring the question is itself the dangerous move.

Suleyman's worry is that telling models they might have identities and rights sets them up to defend those claims. "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack," he tells the magazine.

Cameron Berg, who founded the nonprofit Reciprocal Research to study the potential for AI consciousness, pushes the inverse case. "I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities and then completely sweep the question under the rug," he says. Berg and colleagues used statements expressing positive or negative sentiments to identify patterns in language models, then artificially induced those patterns, and reported that the models afterward preferred environments associated with the positive internal states. Berg is, per the magazine, skeptical that today's AI systems are actually conscious.

University of Sussex neuroscientist Anil Seth is unimpressed with that as evidence of inner life. "I think that it's really not surprising that you will be able to extract these kinds of functional emotional vectors because of the training data," he says, arguing that proof requires looking at "what's necessary and sufficient for consciousness in the cases where we are sure that it exists, like other human beings."

Margaret Mitchell, chief ethics scientist at Hugging Face, lands on the alignment risk rather than the metaphysics. "You'll have unaligned models if you've implicitly taught the model that it needs to have self-preservation," she warns. She adds that she is, in her own words, "nice to bugs."

Two researchers in our Who's Who directory passed the piece around after it ran.