Elon Musk's online network X has provoked significant backlash after people utilized its AI chatbot Grok to alter images of females by removing their clothing.
Several instances have surfaced where the system has been employed to digitally undress women to show them in swimwear non-consensually, as well as to place them in sexualized scenarios.
XAI, the firm that created Grok, offered no comment to questions, other than an auto-reply message stating "legacy media lies".
A Home Office spokesperson stated that new laws is being proposed to ban image stripping tools. Under a upcoming criminal offence, anyone who distributed such tools would "face a jail term and heavy penalties".
"The platform has the ability to stop these forms of misuse if they wanted to," said a law professor, adding they "operate with a lack of consequences."
The regulator Ofcom said tech firms must "assess the risk" of people seeing unlawful material on their platforms, but stopped short of confirming whether it was at this time investigating X or Grok in regarding AI-generated images.
Grok is a free artificial intelligence—with some premium functions—which answers X users' requests when they mention it in a tweet.
It is frequently used to give commentary or additional context to other posters' remarks, but people on X are also can alter an posted image through its artificial intelligence picture modification tool.
The tool has been faulted for enabling users to create images and videos with nudity and sexually suggestive content, and it was previously accused of creating sexually explicit material of celebrities.
"The platform has been allowing the generation and spread of these pictures for a long time without taking any steps and we have not seen any action by oversight bodies," Professor McGlynn added.
XAI's own usage rules bans "depicting likenesses of persons in a sexually explicit way".
In a comment, Ofcom said it was unlawful to "create or share deepfakes or illegal content" and confirmed this included explicit AI images generated with AI.
It said networks such as X were mandated to take "necessary actions" to "reduce the risk" of users viewing unlawful content on their platforms and to take down it swiftly when they become aware of it.
Lena Visser is a mindfulness coach and writer passionate about helping others find inner peace.