‘Staying silent? Not an option’: family takes fight against deepfake nudes to Washington
In October last year Francesa Mani came home from school in the suburbs of New Jersey with devastating news for her mother, Dorota.
Earlier in the day the 14-year-old had been called into the vice-principal’s office and notified that she and a group of girls at Westfield High had been the victims of targeted abuse by a fellow student.
Faked nude images of her and others had been circulating around school. They had been generated by artificial intelligence.
Dorota had been tangentially aware of the power of this relatively new technology, but the ease with which the images were generated took her aback.
“I didn’t know how quickly it could happen, with just one image,” she recalled. “That it can happen to anyone, by anyone, with the click of a button.”
According to a recently filed federal lawsuit filed by a different family, the explicit images at Westfield High were created via the use of an app named ClothOff, which has operated under a deep layer of secrecy.
But a six-month investigation, part of the Guardian’s podcast series called Black Box, has revealed the names of several people affiliated with the app, which receives millions of monthly visits, tracing its origins to Belarus and Russia.
As the shock on that October afternoon subsided, Francesca Mani wiped away tears and decided she would take action by going public. The mother and daughter were dissatisfied with their school board’s response, and disappointed that, because no laws existed, it was unlikely the alleged perpetrators would be held criminally responsible.
“I need to do something,” Francesca told her mother. “This is not OK and I will not be a victim.”
The pair have since made a number of trips to Washington, including to the State of the Union address last week. They have appeared on cable news together and been cited by lawmakers in both New Jersey and in DC as catalysts for new legislation to hold creators of non-consensual, sexually explicit deep fakes legally accountable in the US.
The case in Westfield, and others like it, have exposed growing gaps in federal and state laws that legislators on both sides of the political aisle agree do not do enough to protect people, particularly minors, from the rapid proliferation of explicit AI deep fakes.
“There is a unique danger around these apps,” said Yiota Souras, chief legal counsel with the National Center for Missing and Exploited Children [NCMEC]. “Because the volume of victims they can create in a very short time is immense.”
NCMEC has been working directly with the Mani family as they attempt to see if any of the images generated at Westfield High have circulated further online.
Victims’ parents received assurances from school officials that the deep fakes had been deleted, but the school has not publicly stated how many students were affected.
The Westfield public school district said it commenced an investigation into the incident as soon as it was made aware and provided counselling to “students seeking support”.
“All school districts are grappling with the challenges and impact of artificial intelligence and other technology available to students at any time and anywhere,” said superintendent Dr Raymond González, who added that the district was continuing to strengthen efforts to prevent future incidents by “educating our students and establishing clear guidelines to ensure that these new technologies are used responsibly in our schools and beyond”.
ClothOff has denied its platform was used in the New Jersey case, and suggested it may have been a competitor app but provided no evidence to substantiate this claim.
Shortly after speaking out in public, the mother and daughter were invited to Washington to mark the introduction of a bill in Congress, the Preventing Deepfakes of Intimate Images Act.
The legislation seeks to prohibit nonconsensual disclosure of AI-generated images by making it a criminal offence to share them, as well as providing victims with rights to take civil action in federal court.
Dorota will visit the Capitol again this week for a subcommittee hearing on “addressing real harm by deep fakes”, while another bill, introduced by the Republican congressman Tom Kean Jr, who represents the family’s district, seeks to create labelling rules for AI content to make it easier to distinguish.
“Just because I’m a teenager doesn’t mean my voice isn’t powerful,” Francesca said as the legislation was unveiled in Washington. “Staying silent? Not an option.”
Both bills and their senate counterparts have bipartisan support, but with a House so focused on partisan issues, including impeachment inquiries pointed at the president, Joe Biden, Dorota Mani realises that lawmaking on