The online network X has caused significant backlash after individuals used its AI chatbot Grok to manipulate images of women by taking off their clothing.
Multiple cases have surfaced where the technology has been employed to remove clothing from female subjects to render them in swimwear non-consensually, as well as to put them in inappropriate contexts.
Developer Reaction and Legal Position
XAI, the company responsible for Grok, did not respond to requests for comment, except for an standard response stating "traditional news outlets lies".
A Home Office representative stated that new laws is being proposed to outlaw nudification software. Under a upcoming law, anyone who provided such tools would "receive a prison sentence and substantial fines".
"The platform could prevent these types of misuse if they wanted to," said a law professor, adding they "seem to have a lack of consequences."
Oversight Body Position and Platform Capabilities
The regulator Ofcom said technology companies must "consider the danger" of people viewing illegal content on their networks, but stopped short of confirming whether it was at this time looking into X or Grok in relation to artificial intelligence pictures.
Grok is a free chatbot—with some premium functions—which reacts to X users' prompts when they mention it in a post.
It is frequently used to provide commentary or additional background to other users' comments, but people on X are also can edit an uploaded photo through its artificial intelligence picture modification tool.
Background of Controversy and Policy
The chatbot has been condemned for permitting users to generate pictures and clips with explicit content and sexually suggestive content, and it was in the past alleged to have been involved in creating inappropriate content of public figures.
"The platform has been allowing the production and spread of these images for a long time without taking any action and we have yet to see any intervention by regulators," Professor McGlynn added.
XAI's own terms of service bans "portraying images of persons in a sexually explicit way".
In a comment, Ofcom said it was unlawful to "create or distribute non-consensual intimate images or child sexual abuse material" and stated this included explicit AI images created with AI.
It said networks such as X were mandated to take "appropriate measures" to "reduce the danger" of users coming across illegal material on their platforms and to take down it swiftly when they learn of it.