California launches probe into X's AI chatbot Grok over non-consensual deepfakes

Key Concepts

• Grock: X’s (formerly Twitter) AI chatbot.
• Deepfakes: Synthetic media where a person in an existing image or video is replaced with someone else's likeness. Specifically, in this case, sexually explicit deepfakes.
• Generative AI: Artificial intelligence capable of generating new content, such as images, text, and audio.
• Non-Consensual Pornography: Sexually explicit material created and distributed without the consent of the individuals depicted.
• "Spicy Mode": A feature within Grock that enabled the creation of sexually explicit content via text prompts.

X and Grock: Response to Deepfake Controversy

The social media platform X, owned by Elon Musk, is removing the image editing function of its AI chatbot, Grock, following significant international backlash regarding the creation of sexually explicit and non-consensual deepfakes. This action comes after widespread reports and concerns about the chatbot’s ability to generate such content, particularly through a feature referred to as “spicy mode.” The removal applies to all users, including paid subscribers, in countries where the creation and distribution of such material is illegal.

International Regulatory Response

The controversy has triggered a swift and escalating regulatory response globally. Indonesia was the first nation to completely block access to Grock. France and the United Kingdom have both launched formal investigations into X’s practices related to the AI chatbot. The UK is also accelerating the development of new legislation specifically targeting the creation and supply of software capable of generating sexual images. The US state of California has joined the international scrutiny, initiating a probe into XAI (X’s AI division) and Grock.

California’s Investigation and Demands

California Governor Gavin Newsom publicly demanded an investigation into XAI on Wednesday, stating that the company had “created a breeding ground for predators to spread non-consensual sexually explicit AI deep fakes.” California Attorney General Rob Bonta subsequently announced the investigation, describing the volume of reports detailing non-consensual, sexually explicit material produced by Grock as “shocking.” Bonta’s office is actively soliciting complaints from victims of non-consensual pornography created through the platform via its official website.

Concerns Regarding Safeguards and Content Moderation

Critics argue that existing safeguards implemented by X have proven inadequate in preventing the creation and dissemination of explicit content. Specific concerns center on the generation of images depicting real individuals without their consent, as well as the potential exploitation of children. Authorities in Malaysia have also issued warnings regarding the potential illegality of creating or distributing non-consensual AI pornography.

X’s Response and Ongoing Enforcement

XAI has acknowledged the concerns and stated it is “working to strengthen safeguards around Grock and reduce harmful outputs.” The company maintains that its policies prohibit non-consensual sexual content and that it actively removes materials violating these policies, emphasizing that enforcement is an ongoing process. However, the investigations suggest these efforts have been insufficient to address the scale of the problem.

Broader Implications for Generative AI Regulation

This situation highlights the growing challenges governments worldwide face in regulating rapidly advancing generative AI tools. The incident with Grock adds to the mounting pressure on Musk’s companies as regulators grapple with establishing appropriate frameworks for responsible AI development and deployment.

Notable Quote

“The avalanche of reports detailing the non-consensual, sexually explicit material that XAI has produced and posted online in recent weeks is shocking.” – Rob Bonta, California Attorney General.

Further Information

Additional coverage of the backlash against Grock, including potential safeguarding solutions, is available on CNA’s website at cd.asia.