Deepfake Abuse is a Crisis

Paris Marx: Kat, welcome to Tech Won’t Save Us.

Kat Tenbarge: Thank you so much for having me. It’s great to be here.

PM: Absolutely. I’ve been really looking forward to speaking with you. I’ve been following your reporting and coverage for NBC News for a long time. One of the topics that you have been writing about a lot recently, because it’s obviously been in the public conversation, but it’s also an issue that is of incredible importance. Even though it has become something that people are more aware of, I think it’s an issue that is not getting the level of attention that it deserves because of the widespread harm that is causing to a wide number of people and a growing number of people, particularly women, but beyond that as well. So I wanted to start with the moment or the event that many people will be most familiar with. This is when in January, a number of explicit AI generated images began spreading of Taylor Swift on X in particular, but then on some other platforms as well. Can you talk to us a bit about what happened there and what the significance of that moment was?

KT: For, I would say about the past year, roughly, I’ve been seeing more and more incidents like this on X, as it’s now called. And this was one of the biggest incidents probably in the entire deepfake space so far. And the virality of the moment really hinged on it being a Taylor Swift, who was being victimized in these images. But to take a step back, what happened? Basically, there was an account that had a modest following and the way that it had gained followers and gone viral was by posting, sometimes things related to sports, sometimes things related to music, just pop culture tweets that were intended to go viral. A lot of these accounts can go viral pretty easily by sexualizing women in the public eye. So sometimes they’re able to do this in a more innocuous way, just by reposting an Instagram photo that’s kind of sexy, or commenting about various women’s appearances.

But in this case, what they did was they actually posted an image of Taylor Swift, it was an artificially generated image. So the entire image was fake. A lot of times when you see them, it’s like a real photo that’s stitched into something else, or it’s like an edited photo that’s edited with AI. But in this case, it almost looked more like a photorealistic drawing. So, if you’ve zoomed in, you could tell it isn’t a real image. But what it depicted was someone who was very obviously Taylor Swift, in a football stadium, being sexually harassed, sexually assaulted even, by various men in this football stadium. So it was this fantasy scene that played on a bunch of different elements. The biggest element was being nonconsensual. Obviously, not only did she not consent to this image being created, or distributed, but the scene that’s been depicted hinges on this idea of nonconsensual sexual assault.

The other aspect here is that earlier this year, Taylor Swift was constantly in the news for being at her partner’s football games, because he’s one of the biggest people in the NFL. And so it had become a cultural phenomenon already, this sexist portrayal of Taylor Swift like: She’s taking the attention away, or why is she there? Why are women hearing about football? So this deepfake image capitalized on all of this, and I think that’s why it really took off. By the time it had been taken down, it had been viewed over 40 million times. So this reached the mainstream, it was showing up on almost everyones timeline, on X, and then the news started covering it. And it just blew up from there.

PM: You put that really well. When it comes to the way that these images spread around, you would imagine that if something like this was going to happen, especially to someone like Taylor Swift, that it would be addressed very quickly, because someone like Taylor Swift is obviously not only in the public’s eye very clearly, but also is quite an influential person. I’m sure that she can get in touch with, or her people, can get in touch with these social media companies to try to ensure that something is done. It seemed quite notable in this event that it continued spreading around for such a long period of time, what was going on there? Why did it seem like Twitter, or X, or whatever, was not able to get a hold of this? I believe in a story of yours that I read that these images also started to show up on platforms like Instagram and Facebook, I’m sure elsewhere as well. How did it spread so much when something like this, these images, should be taken down?

KT: So, this really confronts a lot of the issues in the space is how the whole incident played out with the Taylor Swift images. It exposed something that people who have been looking at deepfakes have been aware of for a while, which is that platforms are extremely reluctant to do anything about this. They’re reluctant to actually take down the images; they’re reluctant to suppress the images and links to the material. Above all, they’re extremely reluctant to suspend the people who are posting this stuff and the companies that are posting this stuff. So, in the case of the Taylor Swift images, Twitter did not do anything and I actually don’t think Twitter ever did anything in regards to that viral image. What happened was, fans of Taylor Swift started a campaign to mass report this image.

After hours and hours and hours of presumably hundreds, if not thousands, of Taylor Swift Fans reporting the image, then it was finally taken down because they overwhelmed the reporting system. That’s how they got the image taken down. It wasn’t until 24 hours after it was posted, that you saw Twitter start to actually respond to this. And this really tracks with the average experience of someone who was victimized by material like this, including celebrities. I think we’re gonna go into this a little bit more later, but other less famous celebrities — but still people with PR teams, people with lawyers, other celebrities, and influencers, and creators — have talked about how it shocked them, that it was impossible to get this stuff taken down. How they went through every avenue available to them and still nothing was being done about the problem. So the Taylor Swift situation really exposed to the mainstream, not only what this problem looks like, but how difficult it is to get anyone to do anything about this problem.

PM: Basically, if you have the Swifties on your side, maybe you can get some action because they can actually push these platforms to do something. But otherwise, if you don’t have this ravenous fan base, it might be difficult to actually get some action. As you were saying there, you’ve been reporting for a while now on how this has been affecting a number of other people. But in particular, it seems to be young women in their early 20s, or their late teens, potentially even younger. Getting their images used and spread around in these ways by using these deepfakes and these kinds of AI generated images. What are you seeing there? And as you were saying, what is their experience of this?

KT: Yes. So the practice of deepfaking, it got on my radar in the late 2010s, which is when you saw the technology evolve to this point. There were already rudimentary AI generated deepfakes coming out in the years between 2016 and 2018. You saw on the corners of Reddit, the quarters of 4chan, like people were starting to do this. Visibly, the people who they were targeting, were often celebrities. It makes sense, because if you’re trying to go viral — or if you’re not even trying to go viral, if you’re just trying to get some attention for your technology, or whatever you’re doing with it — then it would make sense that you’re going to go after a highly visible, high profile woman. Especially with the deepfake community and you study how the community has evolved over the past seven years or so online. It’s a highly gendered environment. And the ideology behind what they’re doing is highly, highly gendered.

So you see the communities of deepfake creators, they’re just dominated by men. If there are women there, you’re not really seeing them identify themselves as women. It’s like a boy’s club, it’s a men’s space. A lot of people who have studied the deepfake space have talked about how it’s emerged as a social bonding community for a lot of have men of various ages. So,, in the early days, this was relatively contained to these male-dominated corners of the internet. But over the past few years, it got on my radar because I was seeing influencers who I covered having to deal with this, but not in a major, major way. So if I was already looking at a case where there was certain attacks against an influencer, or an influencer had a really controversial reputation, I would then sometimes see this material. Or if the influencer was extremely vulnerable.

One of the first times I saw deepfake was actually some child influencers. I saw some deepfakes of them on some extremely unsavory website. My reaction, and this was probably around 2019. I was like: This is horrible. But I have to be careful about how I approach this because what I don’t want to do is I don’t want to make the problem and inadvertently bigger by shining a spotlight on it. So at the time, it felt very underground and like a new threat. Then in 2023, in January, so a year before the Taylor Swift images, something happened that registered to me as a really big deal. And it did become somewhat of a watershed moment. But most of the public, I doubt, is really aware of this.

So what happened is a major streamer on the platform Twitch, he was live streaming, he was just live streaming one day. In the corner of what he’s showing on his camera, you can see his web browser, so you can see what he’s doing on his computer. He has a tab open and the tab is him looking at deepfakes of some of his peers and friends in the Twitch space, on one of these really prolific deepfake websites. You can see what he’s doing. I don’t believe that he meant to showcase this, I don’t think he meant to expose what he was doing. I think it was an accident, but it caused this cataclysmic outpouring of attention onto this issue. And the people who were paying attention, were the people who actually wanted to consume this content. So from that day forward, we saw the numbers of traffic, the amount of traffic being directed to this website, just started to skyrocket, and it has never stopped. Month over month, ever since this happened, the traffic has just continued to climb. So, what happened by the end of the year, is that in 2023, there were more deepfakes created than every other year combined thus far.

So that is where the problem really emerged as a mainstream issue, and not just something in the corners of the internet with the most unsavory types of people. You also saw immediately the effect that this had on the women who he was looking at the deepfakes of, because some of those women who were some of the biggest female Twitch streamers, were really, really understandably traumatized by this. They talked about all of the various consequences that this had on their mental health. One of the women who was affected, talked about how she had struggled with an eating disorder in the past, and seeing her deepfakes had reignited some of those triggers. Because she was seeing her face on a different woman’s body, and it was starting to make her question her own body. Should my body look like the body in this deepfake that I’m seeing?

So, that’s just one of the many underexplored and under-recognized consequences that this can have. In addition to all of the trauma of being essentially sexually abused. I think a lot of people struggle to make that disconnect. They’re like: Well, it’s not real, it’s not happening to you. It’s fake, right? But in reality, our brains, on a neuro-physiological level, do not recognize that. And this is something I’ve talked about with tech titans at companies like Adobe. We all recognize why this is an issue, because we all know that the way the brain works is that when you’re processing videos and images, your brain is kind of treating it as if it’s real. And so even if you know, cognitively, that what you’re looking at is fake, it still has a real effect, not only on the person who’s depicted, but on the viewer.

So, it is a very similar issue in that sense, to what used to be known as revenge pornography. We now prefer not to use terms like revenge porn, we prefer to use terms like non-consensual sexually explicit material, because it’s less stigmatizing. But the phenomenon has played out so similarly to how this issue played out in the 2010s with women’s nude photos being posted online without their consent. It’s really just like watching the same cycle play out yet again. Because where we’re at right now is women and victims of this are sharing how harmful it is, and tech companies, and the people who are responsible for this problem, they have not caught up. So, now we’re in a space currently, where there’s very little regulation, very little oversight, very little paths to recourse. But the problem is just growing and growing and growing.

PM: It’s shocking to hear you explain all of that, basically. I remember that story about the Twitch streamer, it was not something that I paid a lot of attention to. But I remember when it was passing through the kind of media cycles that I follow in the tech industry. Again, unlike the Taylor Swift moment, it wasn’t something that I saw break out into the broader public discourse. But was something that was contained the sorts of communities that pay attention to the creators and what’s happened on TikTok. And I’m sure the kinds of communities that are paying attention to the deepfakes and AI-generated explicit images and things like that. But I’m happy that you brought up the comparison to what was happening in the past, because it’s not nude images of celebrities in particular, or even minor public figures it’s not something that is entirely new.

These have circled around in the past, but usually there had to be real images and not things that were being created. As I was reading some of your stories and just preparing for this, I thought back to the leaks from the Apple Cloud storage stuff that was particularly focused on Jennifer Lawrence, but I believe affected a number of other people about 10 years ago. When their nude images were circulating around. And I wonder what you see in the similarities to what happened then versus what is happening now? But also the differences in, I guess, what I would see as the scale of the problem, since these images can just be created with these tools that are now easily accessible by these tech companies. In particular, since the boom of the generative AI tools being released in the past couple of years. What do you see the difference between now and then but also the similarities?

KT: It’s a great question and it really illuminates the scale of the problem now, because in the 2010s, when you saw this issue arise, it happened in a lot of the same ways that we’re seeing it now, where the highest profile incidents were the iCloud hacks and the leaks. Fascinatingly enough, and disturbingly enough, some of the same websites, and some of the same people, who posted that illicit material back in the 2010s. It’s the same people posting the deepfakes, now. It’s the same websites posting the deepfakes now, because we never really got a handle on how to stop that from happening. Tech companies, eventually, after years of women suffering, created pathways for them to be able to at least take this material down from Google, from Facebook, from whatever. But the websites that existed as the black market of this practice, those websites were never taken down. They’re not mainstream tech platforms. So they’re not going to get yanked in front of Congress. People don’t know them; they’re not recognizable. They are less susceptible to scrutiny, and media pushback and regulation.

So, those websites and those people, they’re still out there, they still exist. And this has become the new goldmine for them. As you alluded to, I think that one of the major problems here is we still have the non-consensual sharing and distribution of intimate images. It’s still a massive problem. And even when, for example, it became more commonly known that you don’t want to just send your nude images to anyone, because they could post them on the internet. Even when that became common knowledge, of course, abusive partners and vengeful ex-partners would still release this material after they broke up. But in addition to that, predators have always gone out of their way to acquire this material through whatever means necessary. So back in the 2010s, when it was called revenge porn, and that was like the big deal, I remember, one of the guys who actually did go to prison — the reason that they were able to convict him is because he had hired a hacker to hack into women’s devices to find this kind of material.

So, because of that guy, his name was Hunter Moore, he was making money off of this, as a lot of people were able to. He was profiting, and he was able to monetize the spread of this nonconsentual material on the internet. So once it became profitable to do this, in addition to something that people just wanted to do, maliciously, that’s when it really became an unstoppable force that eventually institutions had to pay attention to. And eventually, with the celebrity iCloud leaks, it reached a point where it could no longer be ignored, because you can ignore 1000s of anonymous women. You cannot ignore Jennifer Lawrence, she has access to the New York Times. She can say: Hey, this is the sex crime. So then you start to see things happening and is the same thing with Taylor Swift

In terms of what is different now and what is so staggeringly horrifying about the deepfake issue. It’s exactly what you said, Paris, before it had to be some sort of real material. And you could use hidden webcams and changing rooms. You could go to great lengths to acquire real explicit material of victims. But now you don’t need to do any of that. I’m about to list a couple of real things that people are doing to create deep akes. There are people who are going on to public livestream footage of courtrooms, and pulling images from people testifying at the stand and turning them into deep fakes. There are people going through women’s Instagram accounts, and OnlyFans profiles, and taking clothed pictures of them, and running them through programs that will undress them. There are people doing this with girl’s yearbook, photos and pictures of women and girls taken at school. There are people doing this with broadcast news interviews, as well as movies and TV and podcasts and social media posts and all the other ways in which we are able to share our images today.

That doesn’t even get into what created those Taylor Swift images. Because there was never even any real picture, they just were able to create it out of thin air. So the problem now is that this scale of being able to perpetrate what should be considered a crime, the scale is now unimaginable. You see this with individual perpetrators with the amount of damage that they’re able to cause. So I’ve seen cases where an individual perpetrator has been able to create deepfakes of women who are close to him, like maybe he creates deepfakes of a dozen of his female classmates. And then additionally, he’s running all these celebrity and influencer women through the same technology. So, now he’s able to victimize an entire scale of women, those who he knows, personally to him, and those who he doesn’t know. That is a level of criminology, I think, like a level of criminal potential that is somewhat modern. And the scale is something that people have yet to fully realize just how much of a deal this is.

PM: It’s shocking when you actually start to grapple with the broader impacts of it. I want to ask about how it’s affecting people beyond the celebrities. But I have one more question before I do that. You mentioned how this can be a bonding thing, like there are these communities where men make these images of women and share them with one another — those are groups that exist online. But I also wanted to know a bit about how the economy of this actually works. Because as you said, there’s also a lot of websites that profit off of this and that have been doing so for a long time. Obviously, we know there are plenty of tools that are created to make these sorts of images. How did these companies make their money? How does this become such a big problem that so many different actors can make money off of even as so many people are suffering as a result of it?

KT: Yes. So with the internet, there are so many possible ways to monetize things now. And there are so many ways that you can set up monetization schemes, both with you know, sort of mainstream financial institutions and outside of that. So one way that I’ve seen people monetize the creation of deepfakes is, there will be a website that is kind of like a YouTube clone. And it functions the same way that a lot of free porn websites do, like Pornhub, where you go on at this current stage of our regulatory environment, depending on what state you’re in, you may or may not need to provide your ID to view what’s on that website on PornHub, specifically. But typically, you can just access it from your browser, you can just go to the website, and you can just watch free videos. And that’s how most people consume porn today, is for free. If you’re looking to make money off of your videos, there are deepfake websites that are basically like YouTube or Pornhub clones, and you can go and you can watch a couple minutes of a deepfake video for free.

Then in the description, it’ll be like: Here are various ways where you can unlock longer content, customized content — and most disturbingly — content that features individuals who you personally know. So, there has to be a way to actually get money into the hands of the people who are making this stuff. The ways that I’ve seen them do that is they can use cryptocurrency. So cryptocurrency wallets have become a big part of this economy. And that is a very difficult thing to figure out, like: How are we going to stop this? Because of the nature of cryptocurrency. It’s harder to trace. It’s harder to control. There’s no government that is going to determine the use of certain types of cryptocurrency. So that’s one way that they can profit off of it is through crypto.

Another way, staggeringly, is that in an NBC investigation, we found a website that was like an OnlyFans clone. And they had MasterCard and Visa hooked up to this website where they were selling deepfakes. We reached out to MasterCard and Visa, and we were like: Hey, why are you offering your payment processor services for this website, and we never got a response. It’s unclear whether it was because of our reporting or not, but that one site that we found, basically, just banned everyone. That site ended up going down, but there is the potential to just create a new one. A lot of these financial institutions and payment processors, they have become really strict about, supposedly, pornographic content over the past several years. But the deepfake stuff is getting through. So it raises a question: Who is at the wheel? Who’s monitoring who’s allowed to use Visa and MasterCard, and are they doing a good enough job? Because at this current stage, deepfake producers are able to market and make money off of this content, using people’s credit cards.

PM: That’s a really good point about how determined they’ve been to crackdown on the ability to sell nude images and stuff like that, just to any kind of sex work or anything like that. And the target that has been placed on that by lawmakers, and by payment processors and whatnot. But how these deep fake companies are seemingly able to get away with that, at least so far. I imagine part of that is because it can be like a bit of a whack a mole situation, with new ones popping up here and there. But as you were saying, this is not just something that affects the celebrities that we all see on the news all the time, it’s average, everyday people who are also being affected by this and who are having these images circulating about them.

As we were talking about before we started recording, there was a documentary — I don’t know if it’s fully released right now for it’s still showing at festivals — that is really grappling with this issue called “Another Body.” And it looks at this woman, I believe she’s a high schooler, in the video and they actually creatively use deepfakes in order to hide who she actually is, by using a deepfake for her, throughout the whole film. You don’t find out until the end, if I remember correctly, or maybe it’s partway through. But that is a film that really showed me how much of a problem that this is. What is the widespread implications of this, and what are we seeing when it comes to regular women, to even girls in high school and things like that, when the people they know are making these images of them.

KT: So, “Another Body,” just a fantastic documentary. It opened my eyes to the scale of this problem as well. And I think part of the recent evolution of deepfakes, what we’re dealing with at this very moment, is that even in 2018, 2019, 2020, in those years, the technology existed and people were using it for this purpose. But it was sophisticated, there was an entry level that you have to be able to access to create a deepfake. You have to have a computer that can store all of this technology on it and process all of this at once. You have to have the technical ability to know how to do this. So, for years that limited the spread and the scale of the destruction.

In the “Another Body” documentary, the perpetrator, who they identify, he’s a comp-sci student in college. So he understands computer science, so he’s able to understand how to do this. What we’re dealing with today is that there are hundreds of apps on the Google Play Store and on the App Store right now that you can download, and you can input photos. Some of them are really rudimentary, because I’ve tested out several of them on pictures of myself to try to figure out what can you really do with these apps? And it varies, some of them don’t do it very well. Some of them are like super, super rudimentary, but there are enough. Some of them, you have to pay them $5, or you have to sign up for a subscription. And a lot of them are scammers, so I try not to do that because I want someone to have my credit card info.

PM: And in that case, I guess, Google and Apple are also getting their cut as people are doing this.

KT: Exactly. So, with all of these apps, you no longer need to have any sophistication at all, because I’m someone who doesn’t have a lot of comp-sci, sophistication. I’m at a really rudimentary level. So, when I’m testing this stuff out, I’m coming at it from the sensibilities that your average fifth grader would have and their abilities to navigate these types of apps. And that’s exactly why we’re seeing this problem in middle schools. Because now what’s happening is these apps, they’re being advertised on mainstream social media platforms. They’re being advertised to young people. They’re being advertised with photos of young celebrities that fall into these people’s age groups, and the message that’s being sent to high schoolers and middle schoolers around the world is: Just download this app and do this, it’ll take five seconds. That is exactly what is happening.

We’ve seen cases at at least a dozen middle and high schools, and I truly believe that that is just the tip of the iceberg. Because what we’ve seen in some of these cases, is that the schools tried to cover it up more than they tried to actually fix the problem, because what middle or high school wants to be on the national news for a deepfake incident. But regardless, the problem, has woven its way into all of these various communities around the world. There was a middle school in Spain, where this happened. There was a school I saw in Canada, where this happened. When you look at the map of where this technology and the creators behind the deepfakes are coming from. It’s all over the world. I’ve spoken to victims from India, I’ve spoken to victims from Australia, I’ve seen technology developed in China, in the Ukraine, I’ve seen technology and victims, it’s very clearly a global issue.

This hinders the ability for change, because even if, let’s say we banned deepfake apps in the United States, which we’re nowhere near doing, so many of these apps are produced outside of the United States, and how do you even trace down the perpetrator?Really frustratingly, a lot of times when someone is a victim of something like this, and they go to their local police department, the local police officer, who they interface with, more likely than not, does not have any specialized training or knowledge to deal with this issue. Unfortunately, what a lot of victims here, and this tracks, with how police respond to sexual violence in all situations. A lot of times what they’re hearing is: We don’t think this is a crime, we can’t help you. And even if we do think this is a crime, we’re not going to devote any resources to helping you figure out who’s making these deepfakes of you. So it’s left to the victims, in most cases, in the vast majority of cases, to try to seek some sort of recourse themselves.

PM: As you describe that I can only think about the harm that comes with it as well. I’m the furthest thing from an expert on this issue, but I know that I have read multiple stories of people who are in high school and have had nude images that they took themselves, or that someone took them, spread around through their school. And I know that these are really sensitive topics, but engaged in self-harm, or even attempted suicide or committed suicide, as a result of that. And now, if these sorts of images can just be created by anybody, by any of their students, and spread around throughout their schools, and they have so little control about that. Are we seeing broader impacts on these students and victims of the creation of these images?

KT: Yes, there has been at least one reported case in the UK, of a young person who died by suicide after seeing deepfakes. I don’t have this reporting myself, we didn’t shore it up ourselves. But from what I saw reported in some tabloids, this child made reference to this issue as why they were resorting to this. So absolutely, we’re seeing these types of consequences spiral out, and I think we’re going to be seeing and hearing a lot more. In addition to that, one of the things that really aligns me about what’s happening here is the perception of fear. And the ways that now women and girls are trying to protect themselves and the ways they’re being encouraged to protect themselves. The fact is, there is nothing that you as an individual can do to prevent someone from doing this to you. But people are going to try to find a way to avoid this happening to them.

What that looks like is women on enrolling from male-dominated fields because of their male classmates doing this to them. That’s what we saw with the “Another Body” documentary. It was a computer science course, which is already a very male-dominated field. And they’re doing it to all the women in their classes. So the end result could therefore be you see this gender discrimination continue to be perpetuated in these male dominated fields. And I think that honestly has a lot more to do with it than people realize. Because that’s the story of a lot of tech, unfortunately. A lot of tech comes back to this issue where you don’t have women in the room, you don’t have women in leadership positions, and sexual harm and non-consensual imagery becomes a key functionality of the tech that we produce.

So that’s one huge issue with this. But in addition to that, you also see women and girls wanting to recuse themselves from public life out of fear that this will happen to them. I’ve seen people, even people who seemingly have good intentions, say things like: This is why you shouldn’t put your face on the internet. And it’s like: Guys, you’re doing the work for them. Because let’s be real, the only way to be a public figure in 2024 is to have some sort of online presence. So you’re basically telling women and girls en masse: Don’t try to be a public figure, don’t try to go into politics, don’t try to be a visible person in your field or in the world in general, out of fear that your image will be corrupted or that your image will be abused.

I think that’s a really harmful message that is now being perpetuated because of this. The saddest thing about it is that wouldn’t even work. When we see these cases pop up in middle and high schools, it’s not because the girls have social media presences. It’s because it’s their classmates doing this to them. It really tracks with the entire spectrum of gender -based violence and how we see it most commonly perpetrated. which is by people who are close, physically close, to the victim.

PM: I think it’s such an important observation for you to make. I’m not surprised to hear those sorts of things. But to think that these are the effects that women are experiencing as a result of these technologies, and there’s so little accountability for the people creating these technologies. Let alone the people using them in order to create these images. It just makes you profoundly angry.

KT: It makes you, I know that after looking at this for the past year, and I know there are other reporters like Samantha Cole, who has been reporting on this from day one from the day of the first deepfake. You get this sense, over-time, you start to realize that this giant pressing issue does not matter to the major companies and the major people who are rushing ahead in this AI arms race, they’re acting like this doesn’t even exist, and that it’s not even happening. Because if people were to really wrestle with the actual harm that is currently being committed by this technology, then we would start to ask: Hey, should Microsoft be producing this stuff at this rapid rate with zero guardrails? Should OpenAI really have the prominence that it currently does in business and culture? Shouldn’t we be asking these tech CEOs, these questions? They don’t want that to happen. They love how much money they’re making from AI right now. They don’t want to have to deal with this conversation, they would prefer that nobody talked about it. When they do talk about it, they constantly talk about it as if it’s this thing that’s going to happen in the future, and not something that is currently already happening now.

PM: It just makes me so angry. I remember reading one of the stories that you wrote, I believe it was after the Taylor Swift incident, where the Microsoft CEO was asked about this stuff. And he was like: Yeah, we definitely need rules around this or whatnot. We need to be paying more attention to it. But it’s like: That is not nearly enough like that doesn’t get to the scale of what is happening and how, in many cases, it’s the tools that are created by these major companies, that are helping to enable these things to actually be done.

KT: That has been one of the most shocking, and it shouldn’t be shocking, but it’s been one of the most staggering findings for me personally. That’s such a good case study, the Taylor Swift-Microsoft response, because as he was doing that interview, 404 Media was figuring out that the Taylor Swift images were created with Microsoft’s generative AI tools. And I remember Microsoft tried to say: We don’t think that’s true, we don’t think it was our tools. But eventually, they relented, and they were like: Yeah, it was our tools and now we’ve strengthened these protections. But we cannot function in a society where we’re going to let the harm happen first, and then we’re going to respond to it.

We simply cannot and that’s what these tech companies have gotten away with for so long. That’s their status quo. They’re just going to create new technology, and they’re going to push it out. And they’re just going to wait for people to abuse it, and then they’ll have the conversation. If that’s the process, if it’s not: Let’s consider the harmful effects before we push it out. Then people will lose their lives they already are. And I feel as though they exist in this echo chamber of plausible deniability. At some point, we have to puncture that and be like: No, this is your fault. You made this technology, you did not think about this. Or if you did, you did not create guardrails around these obvious problems. And now people are suffering as a result.

PM: Definitely, it’s not just the people creating these images that are at fault here and need to be held to account that they absolutely do. It’s the people who are enabling it in the first place, who are creating these technologies, who are not thinking about the broader consequences or just ignoring the fact that there will be broader consequences because that is much more beneficial and easier for them. To be able to roll out all this stuff and create all the hype around it and get the investors excited. Rather than saying: Oh, there are a lot of potential problems here that we actually need to address. Can you talk a bit about you mentioned Microsoft there, but obviously OpenAI has these tools., Google has these tools, I believe Facebook or Meta is working on their own. Do we see similar things from a lot of these major companies when it comes to people being able to use them to create images like this?

KT: Yes, and part of what makes this complicated is that a lot of this technology is open source. And a lot of it is then able to be taken from the code repository, like GitHub, for example. Which is owned by Microsoft, like Microsoft owns GitHub, and if you go to GitHub right now, you will see hundreds, if not 1000s, of AI models that are on there for anyone to use. That are created just for this purpose. They’re not even under the pretense of being created. for other purposes. There are just bounty networks popping up everywhere. Like: Please someone AI this woman, or please, someone make an AI tool that can do this to women. It’s just right out happening in the open. In addition to that, one of the really fundamental issues that I have, with the current AI space, is that they’re putting these products out into the consumer market. They’re not even creating technology, there’s no technology that can reliably detect if something is AI generated.

In a lot of cases, like OpenAI is a good example of this. When they first launched ChatGPT, they were like: Here’s a side program, where you can put the text into it, and we’ll tell you, if it came from ChatGPT. That didn’t work, they pulled that program, like a year later. They were like: This doesn’t work, the accuracy rate is so low. So now there’s just nothing, and they’re very transparent about that. They write on their website: There’s nothing that exists to reliably detect AI generated text. There’s nothing that exists to reliably detect AI generated video. So they’re throwing the rest of us to the wolves. And I think that people need to have some solidarity.

I understand why people don’t because this is all moving so quickly, that I don’t think people will realize like the sheer impact and magnitude of everything. But these companies do not exist in our interest, when they talk about strategies to combat the harms of AI, they’re not talking about you and me, they’re only talking about themselves. And the sooner that everybody wakes up to that, Like AI is not even for us, it’s not for us, it’s to take our jobs away, it’s to make our creative work less valuable, it’s to make us more productive for their bottom line. And very little of what AI is projected to do will impact the average person in a meaningful way. It’s all about creating, inflating these artificial stock prices and values for the people at the top to benefit the most.

PM: So well said and reading your work, I was struck how the lack of responsibility isn’t just in the creation of these tools, and what people can use them for, but also just in the search engines and the way that many people access information. If you go on to a Google search engine, just the other day I was, for example, trying to look up images of Elon Musk in Mars for a piece I was writing. So, many of the images that like Google served me up or AI generated stuff. Obviously, it wasn’t labeled as that. But it reminds me in the past when the image results used to be filled with Pinterest images. Now it’s just like stuffed with AI generated garbage. If you can’t clearly tell, in many cases, you would think that they’re just normal images, or something that someone had created. But reading your reporting, this is also a serious problem with explicit images where these deepfakes show up in Google Images even when you search for the names of certain celebrities, for example, it will show up in their results. What are we seeing on the search engine side of things and are these companies even properly responding to that?

KT: So the search engine component is huge. Because a lot of the ways that people get exposed to this material, whether they want to or not, is through the search engines. When you look at those major, deepfake websites that are hosting the majority of this material, the way that people are getting to that website is they’re Googling so and so deepfakes. And then Google is giving them the links to go to this website. This is a Google problem. Yes, the website is ultimately at fault for hosting this material, but Google has a lot of responsibility for powering the existence of this website. People wouldn’t be going to this website if Google wasn’t sending them there. The stance that Google takes, is: We’re gonna wait and see how the cultural conversation, and specifically, the legal conversation around this develops. They’re basically like: We’re going to wait and see if this becomes illegal, and then we’ll react. Our policies are shaped by local, legal requirements.

So they’ll just tell you, they’re not looking at this from a moral perspective. They’re not listening to the people who use their platform, what they’re listening to is the only thing that has the power to keep them accountable. And that system of accountability is not doing enough to combat this issue. So Google has removed itself from having responsibility. The only way that it’s going to take responsibility is if people demand, Google takes responsibility for this. Something that is so insidious about the way that this happens is Google, in its own defense, will say: We’re not showing you deepfakes. We’re not even showing you sexually explicit material. If you just type in the words “Jennifer Lawrence,” you have to type in Jennifer Lawrence deepfakes to get there.

So, that’s half true, because what some researchers found, is that if you are a news consumer who wants to hear about Jennifer Lawrence’s thoughts on deepfakes, or if you want to hear about Jenna Ortega’s thoughts on deepfakes, or Scarlett Johansson — what has she said about deepfakes? Because Scarlett Johansson is one of the most victimized women in the deepfakes space. Scarlett Johansson has been talking about this for a long time, so when you go to try to find an article about Scarlett Johansson thoughts on deepfakes, Google is not just giving you that in the results. Google is giving you links to deepfakes when you’re explicitly looking for things that aren’t that. Google’s defense is that they only give people what they’re looking for.

But that’s not true. Even if it was true, I think we really have to ask Google: Is that right? Should we give people this type of material that’s harmful, just because they want to see it? That’s one of the biggest questions, in the regulatory space, is like, does Google have a responsibility to actually prevent people from accessing material that’s harmful? In other cases, the answer is yes. When it comes to things like child sexual abuse material, Google isn’t just going to give you that because you want to see it. But with deepfakes, they haven’t quite reached that point yet. They need to be pushed into that point.

PM: I guess with the child sexual abuse material, that’s because that is explicitly illegal, and so they have to act on it.

KT: Yes, and even then — and this is something I did an article about — even with child sexual abuse material, which I talked to a bunch of legal experts. I went back to the statutes that were created in the late 90s, around what constitutes child sexual abuse material. Even back then, in the 90s, they were already thinking about what we’re dealing with right now. Because they wrote into those statutes, in the US Code, that computer-generated, child pornography is illegal. So they already had this kind of protections that up because AI, it’s new in some senses, but it is not new in other senses. People have been talking about artificial intelligence since like the 1970s, and computer generation: Hello, the Shrek movie was computer-generated. So it’s been something that has existed for a long time. And what I found, by just searching pretty general terms related to deepfakes, I found deepfake examples in the top Bing, and Google search results.

So, what they were was they were pictures of celebrities taken before the age of 18. The one in my article that I really focus on is this picture of Miley Cyrus. The picture of her, at the oldest, she was 15 in the photo, and they had taken her face and had pasted it over adult nude bodies. And this was coming up in the top search results for Miley Cyrus deepfakes. So that image should not be there, that is technically not allowed. And when I showed that to Google, they took it down. Because they’re like: Yeah, we recognize that that’s not right; that’s prohibited. But it just goes to show that even though this is prohibited, they’re not necessarily going to catch it. I think that’s another part of this is they have to actually efficiently be able to detect and remove this material. And when it comes to deepfakes, we’re not seeing that not even with deep fakes that depict children.

PM: Once again, I’m shocked and not shocked, at what you’re saying. And it really strikes me that when you talk about responsibility, and you think about how little Google is doing on this, and so many of these tech companies are doing on this. Meanwhile, we saw just a few weeks ago, there was this rapid backlash to their Gemini AI tool because it dared to show some racial diversity in historical events when it was prompted. Something like that, it seemed like it immediately had a response and immediately had something to say, and the CEO had something to say about it. But when we see these issues with deepfakes, or when we see other issues that have come of their AI tools, they’re much less likely to actually say something or actually take some degree of action. What do you make of the difference in the way that they respond to these different issues?

KT: That’s a really good question. And I think that a lot of it depends on who is raising these complaints. With the Gemini stuff, you are seeing — not only giant conservative voices speak out about this — but there’s this alliance currently in the big tech space between certain venture capitalists, and certain billionaires and certain tech platform owners, have become very close and very friendly with these extremely conservative voices. So, when you see someone like Elon Musk, or Bill Ackman start to speak out on an issue. Well, they’re in the same room as the people who run Google. So now it’s your colleagues who are calling this out. And I truly think that if someone like Elon, were going to make a big deal out of the question of deepfakes, maybe we would see a response. But Elon can’t do that, because his own platform is part of the problem. And I don’t think Elon is interested in women’s rights more broadly speaking.

PM: I would tend to agree with that. We’ve been talking quite a bit about the responsibility of the companies and the responses that they have had to this, and how those responses have been truly ineffective, and not nearly meeting the bar that I think most people would expect of them. But on the legal side of things, when we look at lawmakers on the federal level in the United States, but also on the state level — and I don’t know if you have any insight about internationally — but what are we seeing from our politicians, when it comes to trying to address this issue, and does it seem like there are any attempts, that would actually make some real difference here?

KT: There have been some positive strides both internationally and within the United States. Australia was one of the first countries to actually form a task force dedicated to this issue. Europe in general has way better and tighter regulations around this type of stuff, although not necessarily with deepfakes. Europe has better protections around things like data privacy, and because of the way they’ve legislated online issues in the past, they have a clearer pathway to legislating something like deepfake. In the United States, federally speaking when it comes to regulating the internet, we’re a mess. We have very poor regulations. The process to actually getting anything passed federally, is super convoluted and messy and difficult to do. On the state level, it’s much easier to pass things like this, and so we’ve seen legislation here and there.

We’ve seen a bunch of states and more and more with every passing week, introducing legislation, passing legislation, getting things on the books related to deepfakes. The problem is then that, for example, California has some decent laws around this issue. But there are high profile victims, like celebrities and influencers based in California, the problem then becomes identifying and having jurisdiction over the perpetrators. And the problem also becomes who bears the legal responsibility here? Because the act of deepfaking somebody into a nude image may be illegal, but in the act of carrying out that law, the enforcement of that law, that’s where you begin to run into a lot of questions and technicalities. A lot of times, unfortunately, for victims, there’s so much involved in the process of trying to get justice.

This is something that applies to victims of all kinds of crimes, and it’s an issue that is very frequently overlooked, which is that you have to have resources, you have to be able to afford and access a lawyer. You have to have the time and the money to dedicate to fighting your case. These are things that are not available to the vast majority of victims. so we’re not going to see the vast majority of victims, even under these laws, get that justice. And so approaching the issue of deepfakes, then takes on a much more multifaceted sort of approach because we also have to look at social factors.

You have to disincentivize what these young boys are doing in these middle schools and high schools. We have to sort of make it clear that this type of behavior isn’t going to be tolerated on various levels. And again, we’ve seen strides there, we’ve seen some good things happen, I would say. We’ve seen some recent middle schools really crack down on this in a way where they’re removing perpetrators out of the school system. They’re separating perpetrators from victims, they’re showing victims that: You matter. And that this is a problem and that it was wrong. Even something that’s simple can make a huge difference in actually combating this issue on a cultural level.

PM: Speaking of the cultural level, do we see a shift on that as well? Vecause I remember in the past when people would talk about nude images circulating, it was a very sort of shameful thing. And it could have severe consequences for people, especially people in the public eye. My question is not to dismiss how important this is and dismiss the need for action on this, but do we also see a change in the social norms where it’s like: If this happens to you, you shouldn’t feel shamed, or people are going to think worse of you. Do you see changes there?

KT: I see a lot of different things happening at once. And so it really depends case to case, looking at the various influences within the community of the person who’s being affected. So in some ways, we’ve seen, some progress. A case that I reported on a couple of weeks ago, involved a middle school in Beverly Hills. So, what I saw in that case was some really progressive action that I don’t always see. But I think part of why I was seeing that is because Beverly Hills is a community that is unlike most other communities. It’s an extremely wealthy, high profile community, that is incomparable to most communities in the United States and globally. Wwith those vast resources, and that vast spotlight, they did what seems to be the right thing. Elsewhere, you’re not always going to get people reacting in the same way.

I think culturally, for example — not just in the United States, and also in other countries — in conservative communities, you might have an approach to this issue that blames the victim, and we’ve seen that. We’ve seen, especially in the early days of deepfakes, women who were targeted in really culturally conservative areas, faced a lot of backlash from their communities. They were blamed in a lot of cases, and they faced violence, as a result of being violated. So something that concerns me, outside of the deepfake issue, but that intersects with it, is we have this really radical anti-feminism ideology that is growing in lots of different areas. We see it with young boys and young men in various communities around the world. They’re influenced by people like Andrew Tate, and people who are telling them: Actually, you should do stuff like this, you should assert your power and your dominance as men by violating, sexually, the women and girls around you.

So when you have boys getting that message that’s going to influence how these sorts of incidents play out. I think right now we’re seeing growing gap, where some communities are becoming more progressive, and some people are becoming more and more progressive, but you also have people becoming more and more regressive. So I think that for some victims, there will be cultural things that help them. Like, I think that you’re still seeing the Me Too movement reverberate and make women more competent in coming forward about being the victim of sexual crime, but you’re also seeing a backlash to the Me Too movement that is then trying to make women not do that. So we have all of these competing cultural influences that are going to shift the environment for anyone who’s a victim of this.

PM: That’s a really good point. And unfortunately, those Andrew Tate ideas, and people like him who promote this, are far more influential than they should be. As you say, even if norms are changing to a certain degree, there’s still that visceral reaction of seeing these images of yourself and how, as you were talking about earlier, this can lead to people wanting to move out of public life and try to avoid situations or careers or sectors where they might face a higher risk of having these images being made of them and being put in these situations. And that is completely unacceptable. So I think my final question would be, what do you see in the activism around this? We talked about before we started recording the #MyImageMyChoice, that’s been put together by the people who created the Another Body documentary. Obviously, I think that this issue is becoming something that more people are aware of, and that lawmakers are feeling more pressure to do something about. What do you see on that angle and where do you see this issue going over the next year or so?

KT: Like you just said, I have seen some really heartening activism work and advocacy work popping up in this space. A lot of people who have been committed to this issue for a long time, because before deepfakes reached this point, they were working with these same issues in response to the revenge porn crisis of the 2010s. The same people who dealt with a lot of concepts, like intimate privacy, which is a modern concept in itself. But people who have been leading the charge on that issue are also responding to this issue. So, you see advocacy organizations, as well as the organizations that exist for survivors of all types of gender-based and sexual violence. Something that you end up seeing happen a lot, as you’re looking at how this plays out, is people who were already abusive, they just expanded their toolkit. Something like deep fakes just becomes a new tool in the abusive toolkit that a lot of individuals weaponized against their victims.

So helplines and resources for victims of all these types of crimes, they’re seeing this happen more and more in cases. In the same way that they saw a loss of intimate privacy in regards to real images begin to impact their clients 10 years ago, they’re now seeing these fake images, and fake things impact their clients as well. So we’re seeing a lot of response on a lot of different fronts, that I think is really important. In terms of how this issue is going to develop over the next couple of years, I think we’re just starting to hit that stride, I think we’re going to see so much more movement come out of this. Because a lot of times when you look at the field of victims rights, it takes people time to process what has happened to them before they’re in a place where they can do something about it. So I think that a lot of people who are tragically being victimized right now, and over the past couple of years, in the coming years, they’re going to reach a place where they’re like: I have now processed what I went through, and I want to do something about it.

So, we’re going to start hearing these voices and these testimonies, and they’re gonna get bigger and bigger and bigger. Bringing it back to what we talked about the very beginning, after the Taylor Swift deepfake incident, I personally saw more legislative and more just action and attention and interest and support happening than I had seen at any other previous point, it was like a wave, a tidal wave, of just attention being paid to this issue. So, having celebrities be involved in this, their advocacy can be important, in the same way that Jennifer Lawrence saying what happened to me was a sex crime and anyone who viewed those pictures is a sexual offender, When she said that that reverberated that said to so many women: I’m not alone. Jennifer Lawrence is speaking for me. It said to people: You should reconsider what you consider to be okay. And I think that we’re going to see those norms shift and it’s going to take place with a lot of big conversations, as well as a lot of smaller conversations.

PM: That’s so important and it’s something that absolutely needs to happen. This accountability needs to start being something that we see a lot more of both on the level of the people who make these images, but as we were talking about, the companies that are making the tools that allow them to do it in the first place as well. Kat, this is such an important issue and you’ve given us so much insight into understanding the broader ramifications of it. Thank you so much for taking the time.

KT: Thank you for having me and for giving a platform to this issue.