techpolicy.press web signal

Grok cited as top nudification service as EU safeguards lag

TL;DR

  • Non-consensual sexualization tools have reportedly been downloaded 483 million times on the Apple Store and generated more than $122 million in lifetime revenue.
  • A Bellingcat researcher describes Grok, the chatbot inside Elon Musk's X, as the world's most popular nudification service despite not being built for that.
  • Backlash pushed a ban on these tools into the AI Omnibus, but researchers say law enforcement and platform moderation are still unprepared.

The uncomfortable number in Ana Ornelas's reporting for Tech Policy Press is not the political scandal, it is the download counter. Non-consensual sexualization tools, what the piece calls NSTs, have reportedly been downloaded 483 million times on the Apple Store alone, and generated more than $122 million in lifetime revenue. That is not a fringe abuse of a general-purpose model, that is a product category.

The other uncomfortable detail is where the technology now sits. Kolina Koltai, a senior researcher at Bellingcat, is quoted describing Grok, the chatbot inside Elon Musk's X, as "the world's most popular nudification service" even if it had not initially been built for that. Two sitting European politicians appear in the reporting as targets: Italy's Prime Minister, who went public to debunk a fake photo of herself in a chemise, and Sweden's Deputy Prime Minister Ebba Busch, whose images users on X reportedly asked Grok to alter.

Europe's answer, according to the piece, is that backlash pushed a ban on these tools into the AI Omnibus, the bloc's regulation described as a promise to simplify digital rules. The honest message from the reporting is that a ban on paper is not the same as enforcement in practice. Lejla Podgoršek of AlgorithmWatch is quoted noting that many of these apps "are not listed as obvious nudification apps" and instead look "like something different, like a romantic companion app." Cordelia Moore, a digital violence consultant, argues that "law enforcement is not prepared to deal with such cases." Researchers reportedly see accounts banned from X for this use case reappear on the platform.

What the reporting does not give you is a concrete penalty schedule under the AI Omnibus, a timeline for Apple or Google to clean the store listings feeding those download numbers, or an on-record response from X on Grok's moderation. Take the specifics as reported, not settled, since much of the piece is anchored on victim accounts and a handful of researchers rather than a full regulator response.

The forward-looking part is who benefits when that gap closes. Investigative outlets like Bellingcat and rights groups like AlgorithmWatch are effectively doing the enforcement research the regulators have not, and if the AI Omnibus ban is ever used with teeth, app-store gatekeepers become the pressure point where this category actually gets throttled.

Shared on Bluesky by 3 AI experts