Facebook Is the Zombie Internet

Paris Marx: Jason, Tech Won’t Save Us.

Jason Koebler: Hey, thanks for having me.

PM: Of course. Always great to talk to you, when I’m not reading the amazing work that you and your colleagues are doing at 404 Media all the time. But, we’re talking about this series of reporting that you’ve been doing recently on all the AI garbage that’s happening on Facebook. And to get started, I’m just wondering, broadly, how is this a topic that you decided to dive into and dedicate so much time into investigating and looking at? Because you’ve been reporting on it for a while now, I feel like there’s not many other people who have really gone or maybe nobody else who’s really on as in depth as you have on what’s Facebook with all this AI generated content over the past year or so. So how did you get into that?

JK: So, I think that the 2016 election was blamed on Facebook and there was Cambridge Analytica and all of this sort of thing and all of the tech journalism world was obsessed with Facebook and disinformation on Facebook, content moderation on Facebook, talking about Facebook and its policies and whether it’s swayed voters and things like this. And then time passes and Facebook starts to feel irrelevant. I am not on Facebook or until I started doing this reporting, I was not on Facebook. I mean, I had my account, but, I didn’t use it. I wasn’t checking; I wasn’t updating it.

And I feel like that happened for a lot of people. People moved on to Twitter; they moved on to Instagram, TikTok. And Facebook was just plodding along and aging in place more or less. And then toward the end of last year, I was on Threads. And there was a couple viral Threads where people were like: All of these versions of AI photos are going viral repeatedly. And I think that we’re talking now months later, people have probably seen shrimp Jesus and just super bizarre AI going viral on Facebook because it’s maybe the most relevant that we’ve seen.

Facebook itself as a platform has felt in quite some time. But the original photos that I was seeing going viral were not weird. What they were were wood carving images of dogs. There’s a few different artists on Facebook and on all social media where they basically carve dogs out of chainsaws. They get a big log and they painstakingly will make a statue of your dog for you in exchange for a lot of money. It’s quite expensive. It’s really time consuming. And they document this process on Facebook. They’re influencers and they’re also like artists. And what was happening was, there’s like this image of the person who does the wood carving kneeling next to a dog.

And I think it’s a golden retriever or something, but I don’t know. There’s different ones that he’s done. And it was, like 50 different versions of that same photo, but the dog was slightly different or the person was slightly different. So in some images, the person had a goatee; in some images, the dog was like a St. Bernard and some of them, it was like a German shepherd, etc. And they weren’t fantastic AI fakes, but they were AI fakes that were based on a real image. And so it looked pretty real. It was not bizarre at all. And it was going viral over and over and over again. And so it was a Thread post about this.

And I hopped on Facebook and I started like looking for them. And, one, I found a lot of them. And, two, I found a Facebook group called “Isn’t that AI?” and it was a group of like a couple hundred people who were documenting instances of AI going viral on Facebook. And there was one woman in this group who is from New Zealand. And she had created like a Google Sheets spreadsheet of all of the different versions of this dog that she had seen on Facebook and all of the places that it was being posted. And she had done this not just for the dog, but also for like a bread house, which was this guy who had like made a house out of bread. And then there were like 30 different versions of that.

There was images of children who had supposedly painted a picture. That’s dedication. Yeah, I was like: What’s going on here? She’s like: I’m very bored. And I’m obsessed with figuring out what’s going on. So I basically wrote an article about this group and these people who were documenting what was happening there. And I feel like from there, I got sucked into an algorithmic rabbit hole. In doing that story, I had looked at so much AI on Facebook that suddenly my entire feed became AI and remains AI to this day. So it’s led me down many rabbit holes and there’s been many stories since then.

PM: So I have a ton of questions based on that, but the first one to zoom back from the AI for a second is what are you making of Facebook these days? Because it feels it’s this social media platform that has just been left behind to a certain degree. Nobody really cares about it as much anymore; there isn’t so much discourse about it. But there are still a lot of people who use it. It’s just probably not the most online people and the people who are in the discourse all the time but like they’re still what at least hundreds of millions of people who regularly use that platform and it just feels like that reality is just left out there as the platform itself degrades constantly.

JK: So, I don’t want to dismiss the people who still use Facebook. I’m glad that you picked up on that because, one, it’s definitely not the most online platform. New memes don’t come from Facebook as we know them unless they’re really bizarre spam. But it’s still a massive, massive social platform. I don’t have the numbers in front of me. I don’t know if Facebook is still sharing them, but it’s still billions of users. It’s more than 1 billion and probably closer to 2 billion users, and many of them are in developing countries. It’s still the dominant social platform in places like Malaysia, Indonesia, even places like Hong Kong, which is very developed, still use Facebook a lot.

And then of course in the United States, it’s huge, but it’s with an aging demographic. And I think that we’ll probably talk more about what’s happening here, but a lot of the AI spam is coming from places like Vietnam, places like Malaysia, places like Indonesia, and it’s being injected into the feeds of your aunt and uncle and your old classmates who are still regularly posting all the time. So it’s still important. I think that it’s still a dominant marketplace is huge in the United States still.

And the extent to which I was still using Facebook was to buy a couch or something like that. And it’s also full of scammers and spam and weird stuff. And I think that it’s no longer the shiny toy or object out there in social media, but it’s still very big. And I think that the only way I can describe the experience of using it now is it feels like it’s a dead mall. To me, it’s a mall that still has some people who go there for very specific things, but most of the things happening there feel very scammy and like you feel very lucky if you’re able to like get in and out and buy the thing without getting sucked into a mid-level marketing scheme or something.

PM: I think that resonates a lot. I feel like for me, one of the things for a while, I didn’t have a Facebook account and like I have one now, but like you, I never really go on there very often at all. And I feel like the big thing that used to be was like a lot of events were posted to Facebook. And if you didn’t have Facebook, it was like you’re being left out of these sorts of things. There was a story that you wrote a few months ago that resonated with me. me where you were talking about how it was in relation to some of this AI stuff going on.

But you were talking about how Facebook is also a really complicated social platform in that it’s not just you go on there and post and you look at your newsfeed, but it has the marketplace and it has groups and it has pages and there’s like all these little pieces of it that can get really, what actually happens there can really get hidden in all these different pieces of the platform, which I hadn’t really considered before, but makes a lot of sense.

JK: It’s astounding. Google has launched so many products that has then sunset and everyone gets very mad when Google shuts a product down that they like. Facebook launched all those things, but then it didn’t shut them down. So you have the wall, the original thing; you update your status. There’s a news feed. You can have an event. You can have a group, but then you can also have a page, which is not a group. A group can have a page. A page can have an event. The event has a wall, and the event has like a separate area to upload images and things like that. I learned the other day that groups can also have files. And so you can just like upload PDFs and like MP3s and various files to groups. And a group can be either open or closed.

There’s just all of these different permutations that exist from lots of features that Facebook has launched over its 20 year history. And then, some of them are still used and some of them are not. I have found as I’ve been like spending more and more time on Facebook is that a lot of the stuff that breaks Facebook’s rules — the porn, the scammers, the spam — is abusing some feature of Facebook. It’s hidden and then it’s shared from where it’s hidden to a broader audience and then an algorithm picks it up and puts it in front of people.

I did a story about porn that was being posted, and it wasn’t just porn, it was malware. It tried to download a file onto your computer. If it was porn, whatever, it’s fine. I don’t really care. It’s against Facebook’s rules, but I don’t care. But it was porn that was trying to download files to your computer, and it was in an event that had a random string of characters as the name of the event. It was posted as a file in that event. That event was being thrown by a group. And then that event was being shared to a Taylor Swift fan page that had like hundreds of thousands of followers.

PM: That’s so wild. This is the story where I was reading you writing about this, because the thing that really stood out to me was on the one hand, obviously this stuff happening — t he fact that all of this is being posted on Facebook in the first place, because you talked about porn and scams and things like that in this story and how this is the type of content that would usually be or at least theoretically be moderated by Facebook. It breaks Facebook’s rules. It has traditionally been a platform where porn and nudity is not something that is allowed. And even women posting breastfeeding photos has been a real controversy on a platform like that.

Whereas, when it’s hidden within these groups and these layers of Facebook, not only can it be hidden away there and it still exists, but it feels like potentially because of the state of the platform, as you were talking about, that Facebook itself probably doesn’t really care that it’s happening there, as long as most people aren’t seeing it. Does it feel like that is the case?

JK: So in 2018, in the aftermath of Trump winning and the sort of panic about fake news and things like that, we did a series of stories back when I worked at Motherboard exposing what Facebook’s rules were, the non-public rules that they were giving to their content moderators. And the specific rules were like nuts. They were very highly pedantic and highly specific. And the one that continually stands out to me is — this is from internal Facebook documentation — it was like, sexual content is not allowed on Facebook. Nudity is allowed under some circumstances, and then it was a list of when nudity is allowed and breastfeeding is nominally allowed, although Facebook has like gone after it sometimes, and the specific example that they also had was you are not allowed to show an anus on Facebook.

But you are allowed to show an anus if it is photoshopped onto the face of a political leader as something that could be interpreted as like political speech. And so the example was they had an image in there where Kim Jong Un’s eyes had been replaced with two buttholes. And they were like, that is allowed because it is a commentary on Kim Jong Un. And then they had a second image where it was a dildo going into Kim Jong Un’s face and they’re like, that is sexual content. So that needs to be deleted and you need to ban the user.

And that sounds like incredibly crazy because it is like that that was an internal document at Facebook. But we publish a bunch of stories about this and then Facebook is like, you have to come to our office. We got called to the principal. Basically, we weren’t doing like a lot of access journalism, but they were like: We need to show you how complicated and crazy our platform is. And like you try to police this; you try to moderate it. So I went to Menlo Park. I went to their headquarters. And I spent two full days. Everything was on the record. I went to like a meeting with their content moderation teams. I talked to their executives. I talked to them about why they do this; how they do this, blah, blah, blah.

And I went in extremely skeptical and I came out being like, this is crazy. But it is very hard. I don’t know how one would do this. Because if you think about it, it’s like 2 billion people posting whatever they want. Some number of them are trying to break the rules. There’s all these edge cases. There’s all these different languages. Facebook has been credibly accused of facilitating genocide in Myanmar because, in part, it didn’t know how to moderate in Burmese. It didn’t have automated content systems that could read the ASCII for the actual Burmese characters, but it was letting people post in that language.

Basically, it was very complicated and I found that they were trying. I was like: This is very hard. It seems like they’re trying their best. They’ve hired like all these professors, all these former politicians, all these like former law enforcement, blah, blah, blah. So they do that and then time passes and I don’t know, at the time I couldn’t go type Taylor Swift into Facebook and find porn three seconds later. And I can do that now and my feed is taken over by AI generated spam you have like shrimp Jesus and all of these super bizarre things and you have scam and malware I’m getting ads for hard drugs on my Instagram, so on and so forth. I think Facebook tried hard for a little while, and they then probably saw that Elon Musk bought Twitter, fired the content teams there, content moderation teams there. Some advertisers left, but many of them came back.

And the bar right now is Mark Zuckerberg simply has to not overtly praise white supremacy. And Facebook is then seen as a safer platform for advertisers. And I truly think — and this is backed up by interviews that I’ve done with content moderation professors and people who have formerly worked at Facebook — I think that they are seeing what they can get away with. And so they have left the platform to fend for itself, more or less. Unless it is 911 very, very, very bad, Facebook is not gonna remove it. And I mean, what I mean by that is child sexual abuse material, things that are overtly illegal, I think that they probably don’t want on their platform. But, I don’t know, spam, scams, AI content, I don’t think that they care.

PM: I feel like when you talk about that distinction between Facebook and Twitter (X), you can even sort of see that in the coverage of Zuckerberg and Musk, as well, where Musk is getting this quite negative coverage. There’s still some boosterish stuff about him. Media organizations are still writing about his tweets, all that sort of stuff. But Zuckerberg is taking this real shift where he’s really into UFC and MMA and he’s training and his hair is a bit longer and he’s wearing chains and baggier clothes now and it’s like: Whoa, Mark Zuckerberg has changed. Still running the company the same way, I’m sure, but like he’s being treated in this very different way because he’s not Elon Musk right now.

JK: I think that’s right. I think that people are trying to say that Zuck is cool now because of his chain and so on and so forth. And I think that it comes from a good, well-meaning place, but if you hop on Threads and you say, I think Twitter is not that bad, or I think Threads is bad, people will jump down your throat and it’s crazy to me because in my experience, I don’t see like the overt Nazi stuff that you might see on X. But I see way, way worse stuff on Facebook, constantly. I see horrible things on Instagram. I don’t know if you scrolled through Reels lately, but we were talking about banning TikTok right now. Reels is one of the most unhinged places on the internet.

I feel like Zuckerberg is getting a pass. And I don’t know if that’s because, he got hauled in front of Congress several times a few years ago and we were mad at him for a little while. And it’s like, well, now, he likes surfing and MMA, so he’s cool. I don’t really get it. I’d be curious to know what you think, but I just think that there’s not much scrutiny on Meta, the company right now, unless it’s schadenfreude associated with the Metaverse failing and the weird stuff on Facebook. But it’s not like our democracy is dying vibes in the way that it was after the 2016 election.

PM: It feels like we had our moment of grilling Zuckerberg and getting mad at Zuckerberg and then, someone else rose to really take the spotlight as Elon Musk bought Twitter and we were like: Oh, this is the evil social media person now. And Zuckerberg was able to recede into the background and just let him take the limelight. But I would have to think a bit more like if I have a real deep take on why that is, but I feel like surface level, that seems to explain it.

But, talking about Facebook, let’s dig back into this AI piece of things. I was wondering, you have been looking at this for quite a number of months now. You’ve been following these pages; you’ve been looking at what’s coming up. What are the trends that you’re seeing in the AI content that’s being posted to Facebook? Does it seem like there are certain things that We’re more popular at different eras and the type of AI content has changed over time or is it just like this massive, massive stuff that’s always there?

JK: I do think that there’s different ethics, if you will, like any other type of content on the internet. And it’s actually not what I would have expected because the early stuff was way more realistic. It was trying to fool people. It was people claiming: Hey, I made this art. Hey, here’s a beautiful log cabin. Here is a stunning nature scene. And then things just got more and more surreal and bizarre. There was the whole era of children in Africa who had built a car out of plastic bottles, was a thing that I kept seeing, which it’s funny because it’s hard to explain with my voice. But if you see it, you’ll be like: Okay, that’s what that is.

There was like a lot of that type of content. There’s like a lot of military adjacent stuff, like support the troops vibes, a lot of American soldiers hanging out with hot women with gigantic breasts, but sometimes they have like three arms or like 17 legs.

PM: I feel like there’s a trend of like those sort of like AI generated women and they’re looking for husbands or something like that that get posted. And more recently they seem to be more and more deformed. I saw one recently that was a woman’s head on a pair of legs, still posted in the same format. And I was like: What is going on here?

JK: Exactly. And then, I mean, Jesus. Jesus has persisted. Take all of the things I previously said, where it’s like troops, children who are starving. There’s like a lot of: It’s my birthday; please support me. But then, there’s a lot of images where they just like stick AI generated Jesus or multiple of him into the image. These things seem popular for a little while, and then they move on to something else. As we’re recording this, and I sort of, I’m not sure how much people care. Still, it’s hard to keep doing similar versions of the same story, but I’m currently deciding whether I want to talk about the fact that we’re actually moving back toward a disinfo vibe.

There was quite some time where it was like, these are just like weird images and these are just images of people who are ostensibly 105 years old. So amazing! Click; enjoy. And now I’m seeing a lot of like science stuff where it’s an image of space, but it’s not actually the planet Jupiter. And then there’ll be some fake explanation for the phenomenon. I’ve seen a lot of like beautiful waterfalls and archeological discoveries, like archeological discoveries of like the biggest whale ever. But then you look and 17 blurry people pulling out a clear monster out of the ground somewhere.

But it seems to be trying to do like news again. I say it, but these are people who are making these things. It does not have a life of its own. It’s not sophisticated in any way. I would think that they are using Stable Diffusion or Midjourney, generating these things in huge numbers and then posting variations of them every five minutes across thousands of pages.

PM: That’s interesting. It especially seems interesting to me because we don’t have news on Meta platforms in Canada anymore. It’s not allowed. So it’s like this stuff seems like it could fill the void of actual news. But it also brings to mind, even in the earlier days of the internet, those conspiracy theory videos about these weird creatures being found and stuff would still be popular and go viral in their own way. Obviously it was a more limited number of images because there was a lot more work that came into putting something like this together. So I guess the scale can be different. Do you see it as like distinctly different than what was happening in the past? Or how would you think about that?

JK: I do think that it’s different, and here is why: most of the pages that I’ve seen that are doing this have a mix of AI content and what’s clearly just stolen stuff, regular run of the mill spam, which is just screenshots of things that were previously viral or whatever. And I think that Facebook’s algorithm dings reposts more or less. I think that there is a finite number of previously viral images that you can just post on Facebook and hope that it goes viral. You can only post the same uplifting photo of a dog or something so many times before people start not engaging with it and maybe the algorithm can tell. The metadata suggests like: Hey, this has been posted in the past. So it’s not going to be valued as like new content.

And what I think is happening is that people are creating so many different slight variations of the AI content that is treated as new every time that it’s posted. And it’s a black box, so we don’t know. But I think that Facebook is treating this stuff as new every time, even if it’s just 5,000 different variations of the same image, just slightly different. So I think that this spam is more effective, essentially. I don’t think that it’s necessarily like more dangerous necessarily. It depends on what it is. But a lot of the AI stuff that I’ve seen has been annoying and space filling. I look at it as truly just like spam or junk that clogs your feed versus trying to trick people. A lot of them do have scams associated with them too. But as far as disinfo goes, I don’t think it’s necessarily too much different from what we’ve seen, at least so far on Facebook.

PM: That makes sense. And when you started to say disinfo, I thought immediately you were going to bring up political stuff. But it even seems that is something where there’s a lot of focus on it. These AI tools can be used to create these deep fakes and make it seem like someone’s saying something that they didn’t say. But there’s far less attention on on that piece of it, where you’re just putting out totally bullshit stuff about space or nature or science or whatnot that people might end up falling for if it’s not something that they’re aware of if it looks real enough, if it seems interesting enough. I would wonder about the broader impact of that. And I have no idea what it could potentially be, unfortunately. .

JK: It’s funny because we’re talking about Facebook. Let’s say this was Facebook 2016, where every news outlet is making their living off of Facebook traffic, and then it gets taken over by this. Suddenly you’ve crowded out this traffic source for like legitimate news. But Facebook has become so useless as a platform more broadly that it’s like: Oh, like the feeds are taken over by junk. Who cares? I think that like, we’re having the same conversation about Google search and AI content that’s indexed by Google, etc. And seen as this existential threat for news outlets, for independent websites for creators, so on and so forth.

I think the argument should also persist on Facebook that it’s not great if artists, news outlets, your mom, my cousin, random people who are trying to reach their friends and family are trying to being drowned out by just like 17 million different versions of the same Jesus AI. I’m just like obsessed with it because it’s so weird and I’ve watched it evolve over time. But I do think that if Facebook felt more relevant and urgent as a platform, then it would be worse.

PM: I’m not on Facebook looking at it all, but I’m very much enjoying reading everything that you are finding on it. I want to dig into this further, but I have one quick question on this before we do: shrimp or crab Jesus? I would imagine in an earlier version of the internet, there would be a fake religion created around shrimp Jesus, and people would be jokingly revering the shrimp Jesus. Do you see any of that or is it just: Oh, there’s these AI images everywhere?

JK: I mean, I’ve seen a lot of jokes and shit posts that yes: I welcome shrimp Jesus or crab Jesus as my Lord and savior vibes like that. But it’s not like the pasta, the flying pasta monster.

PM: That’s what I was thinking of. Flying spaghetti monster!

JK: Yeah, yeah, yeah. I haven’t seen as much of that and that’s a bummer because like shrimp Jesus is pretty iconic. I think you could possibly argue that it is one of the most popular, iconic pieces of AI art that has been created thus far.

PM: I feel we need a 404 Media shrimp Jesus merch.

JK: That’s great.

PM: There’s an idea for you.

JK: Yeah, thank you. Thank you [both laugh].

PM: So you’ve been talking about everything that is going on on Facebook, and I feel like we’ve been having this broader discussion recently, as all this AI stuff has been taking off about the state of the Internet and the state of the web itself, the reemergence of this dead internet theory, this idea that the internet has been taken over by bots, that now it’s full of all this AI garbage content and the humanness of the internet is increasingly being pushed out because there’s so much of this trash and non-living stuff that’s going on there. You have framed it in a different way as being what you think is happening is more like a zombie internet, especially looking at Facebook. Can you talk to us about the distinctions between the two and what you see the zombie internet as being?

JK: So the dead internet theory is basically the idea that everything that you interact with on the internet is full of bots, and you can take it to an extreme where there are some folks who are like: And how do I know that anyone on the internet is real? I’m the only person on the internet. Everything else is bots.

PM: How do I know that you’re not a chat bot that I’m speaking to?

JK: Exactly. Exactly [laughs]. And I think it’s a very compelling argument or theory. Every time I do an article like this, people say dead internet theory, dead internet, dead internet. And it’s like: I get it. It’s a thing that you can say very quickly and resonates, I think, but it felt like it wasn’t complete. I didn’t think that what I was seeing on Facebook was dead internet. It felt like worse than that to me. And what I mean by this is there’s an AI image that I saw on Facebook that was a deck, like a wooden outside deck that’s attached to a house. And it was like a normal picture of a deck that had been run through an AI generator like image to image generator, and it had added like some posts and some supports and it had taken away others, and it was massively viral on Facebook. I just happened to find it.

And there were like hundreds of people arguing about whether the deck was up to code, whether that deck was safe, essentially. And I was like: The deck isn’t real. The deck is not real! These people are arguing over whether the deck is safe or not. I’m like, the deck is not real. And that was a moment for me where I was like, there is so much time being wasted by real human beings arguing about stuff like bot content or automated content. And so for me to say like the dead internet it’s all bots. Yeah, there’s tons and tons of bots. A lot of them are bots. A lot of the ways that these are getting surfaced into people’s feeds are like inauthentic bot behavior where there’s tons of likes, tons of comments, tons of people just like saying amen or prayer hands emoji or like adding a heart.

But the effect that that is happening is it’s boosting that into people’s feeds and they’re then engaging with it and they’re wasting their time on it, and it depends on how bleak and dystopian you want to get about it. But people are essentially talking to no one about nothing, depending on what you’re looking at. And then I also saw a few accounts that were posting these things where it was like, some of this behavior seems real to me and some of it seems not.

And what I mean by that is if I click through to people’s profiles, I would see that they were posting on their own profiles what looked to be real images of real family members. There was like back and forth comments and like: Hey, look, I was at this wedding or I loved this food that I ate the other day. And it was like, that’s a person. I feel pretty confident saying this is a person, but then they were also commenting, amen on 900 AI generated images. And so like that real person’s account had been hijacked, perhaps without them even knowing it to like engage with this stuff. And I guess that’s why it feels like a zombie internet to me. It feels more parasitic and it’s very hard to tell what is real, what is not, who is doing what. It feels like the zombie internet, not, not the dead internet.

PM: What really stands out to me, when I hear you describe that, is these are a lot of people who probably went online and Facebook was part of their original experience of what the Internet was. And sure, they used Google and they went and explored other things. And I’m sure they use YouTube and all that stuff. But they were never really like using forums and all these more spread out earlier forms of what communication looked like online. And they went online and they didn’t have this technical prowess or anything like that. It was just like, everyone’s going online. This is what you do.

And one of the things that’s part of that is the sign up on Facebook. And then you talk to all your friends and your family there and all those sorts of things. And even though the internet has changed, that’s still how a lot of people use the internet. And now on the one hand, they’re just stuck on this platform that feels like it is degrading and that Meta just wants to get as much profit out of as possible. So it keeps changing the experience and making the experience worse.

But this is how these people use the Internet. They don’t have another like way or platform to move to to keep in touch with all these people. So they stay there. And then on the other hand, as you talk about these accounts being being hijacked, maybe if their account is being used in this way, they wouldn’t necessarily realize it anyway, or they would know that something is going on, but they don’t know what to do about it. So they just keep using their account in the way that they can, even knowing that other things are happening. And it’s this, I don’t know, terrible situation where you imagine, theoretically, that the web would be a better thing and that we imagine it in these, I don’t know, more positive ways but it’s almost like this is the reality of at least part of what the internet is today.

JK: And I mean, you’re absolutely right. And something that it’s been gnawing at me and that I have like nodded at in some of my articles, but haven’t been able to unpack because it’s impossible to say the extent to which this is happening, is that a lot of the places where this stuff is being created and shared from our countries that were part of Facebook Free Basics, which was a program by Facebook to essentially allow people in developing countries to use Facebook, but not the other parts of the internet. And what I mean by that is you basically got a free data plan on your phone. But that free data could only be used on Facebook and the effect of this was that billions of people, probably, ended up creating Facebook accounts as one of their first things that they did on the internet and we don’t really know the extent to which that legacy has had any sort of impact. It’s like the program doesn’t exist anymore.

I wrote a few articles about why it was not that great of a thing because it violates net neutrality. It creates this situation where people think that Facebook is the internet because it’s the only thing that they can access. But it also creates a lot of very good Facebook users. If you can only use Facebook, you’re going to find out every feature that Facebook has. I did this story where people in Angola a long time ago, and by a long time ago, I mean in 2017 or something, they had access to Facebook and they had access to Wikipedia, because Wikipedia had a similar feature called Wikipedia Zero.

And so what people in Angola were doing was they were using their free Wikipedia and their free Facebook to upload pirated movies onto Wikimedia Commons. And then they were sharing links to those movies, in hidden Facebook groups. And so they had created a version of Netflix that use these two programs. The article I wrote about was like the Wikimedia community was very mad that these people were abusing their platform that they were trying to keep as an encyclopedia. But when people don’t have access to the entire internet, they’re going to figure out how to use it. And I think that my theory is that we’re seeing some of the impacts of that now, when we talk about all of the ways that people are hiding crazy stuff on Facebook. A lot of this stuff is happening in places where Free Basics was popular.

PM: That’s such a fascinating story. I’m going to have to go back and read it now because that sounds so cool. I’m wondering about the monetary side of this as well, because, okay, we’re talking about all this AI generated content flooding Facebook and people’s accounts getting hijacked and Facebook not doing a whole lot to stop it necessarily.

I’m wondering, on the one hand, does Facebook feel like it is making money through allowing all this AI content to filter around the platform and all these potential bot accounts to be using it. And then on the flip side are people who are taking over these accounts and are people who are sharing these AI generated content, images and stuff, are they making money somehow through doing this? I don’t really understand that piece of it.

JK: This has become a little bit more clear over time because at first it was like: We don’t know. Are they doing this for disinfo? Are they doing this for money? I don’t understand. Best theory at the moment is that people are building up as large of a following as they can, because some of these images have gone massively viral. We’re talking some of the most popular images on Facebook. So they’re building followers to their pages and then they can sell those pages if they want. There’s a gray market for: I have a Facebook page with 3 million followers; give me X amount of money for it. So that’s one thing that can happen.

Another thing that’s happening is a lot of them are linking off of Facebook to AI generated spam, quote “news websites” that’s plagiarized nonsense. And when you go there, there’s just like 1 zillion Google ads. So they’re driving traffic off platform and then collecting the small amount of programmatic advertising that they can get.

There’s also some that are selling products, so they’ll do drop shipping stuff. They’ll go viral with a AI generated image of a cute dog. And then in the comments, they will be selling a dog leash. I’m not sure how lucrative any of this is, but it clearly must be working because I’m seeing just tons and tons of it. And then Rene DiResta, who was at the Stanford Internet Observatory and did a n academic study into some of this stuff, found that some of it was linking to malware and credit card stealing websites and things like that. That’s not the majority of these pages, but a lot of them ultimately do try to get people off of Facebook and to a place where you’re prompted to buy something or give someone your credit card.

PM: That makes a lot more sense to understand how people are making money off of it and I guess it works for Facebook because all of this AI content fuels engagement and that presumably means more people are looking at ads. Is that that side of it?

JK: This is very interesting and I think important at, I think, the first quarter earnings call, Zuckerberg talked about tweaks that Facebook had made to its algorithm. Essentially, TikTok was eating Facebook’s lunch and Instagram’s lunch with its “For You” page. And every social media platform is now chasing some version of that “For You” page where you just log on and you see popular stuff that’s tailored to you in some way, shape or form. So for a very long time, you went on Facebook and you would scroll the news feed and you would see things that your friends and family would post or the pages that you liked posted, so on and so forth.

But sometime last year, Facebook started adding recommended posts, and these recommended posts have nothing to do with anything that you have proactively gone out and liked. It’s just popular stuff from around Facebook that the algorithm thinks that you like, but it’s not from pages that you proactively said, show me more of this. And Zuckerberg said something like a third of all posts that people see on Facebook are now delivered through this algorithm. And that it’s driving more engagement on the platform. I’d be very curious to know whether this persists. But what I suspect is happening is a lot of people are just obsessive Facebook users. And because of this algorithm, there’s a lot more stuff for them to see because it’s just pulling stuff from anywhere on the platform versus: Oh, your friends didn’t post that much. You can now do Facebook and listly, whereas maybe that wasn’t the case in the past.

PM: That makes sense. I don’t use TikTok. I don’t use Facebook regularly, but I found on Instagram recently that there’s a few times where I’ve fallen into this rabbit hole of watching these different videos that it recommends to me. And after a few minutes or 10 minutes, I’ll be like: What am I doing? Why am I watching this? But, I’m sure it works for it.

JK: Or like: What is this? Or where did it come from? What accounts are these? So that’s what’s happening. And it also has the effect of a person in Ohio or Arkansas can end up seeing AI generated content about Jesus made by some random person in Brazil or Thailand, where that wasn’t the case before. Not that that’s necessarily a bad thing, but I think it has globalized Facebook in a way that it wasn’t previously, which I think probably has the effect of like content that can transcend language is probably performing better than it used to. Because it’s like: Oh, this crazy shrimp Jesus image, you don’t need to speak any specific language to understand what’s going on there. It’s Jesus, but he’s a shrimp.

PM: Our new God, of course! [laughs] There’s one other potential effect of this. I wanted to ask you about speaking about Facebook in particular, and honestly, this broadens it out as well. You talked about in one of the stories that you wrote about how we’ve been in this era of generative AI for about a year and a half now and people are running into these things and finding that they’re being fooled by it to the degree that increasingly people are looking at things that are real and thinking that they must be a generated because they look a little off for a little different or something doesn’t seem right about it.

Do you think that there’s a broader risk in that piece of this, where now you have these things that are real, but because there’s so much AI generated garbage floating around and people are running into that now they think even the real stuff is fake as well, because it can be so hard to tell it all apart?

JK: I think some people don’t want to be fooled on the internet. They really don’t want to be seen by their friends and family as an idiot who fell for a deepfake or someone who fell for fake news or an AI generated image. And so people really have their guards up and I fully understand why that’s the case as a bit of an experiment and not scientific, but I posted on my own Facebook page and I was like: Tell me if you are getting fed AI generated content and screenshot it and send it to me.

And I don’t know, probably a third of the people who responded, sent me images of things that I was like: This is real I know this is real, but they thought that it was AI generated. And so I do think it’s a risk. I think it’s a risk that people are going to see things that are real and they’re not going to believe them because they know that AI exists. Maybe they’re not that fluent in what it looks like. And they’re just like: I don’t want to be an idiot. I don’t want to fall for the AI. So this beautiful drawing made by an actual art student, I’m going to call that fake.

PM: I think that makes a lot of sense, especially when you’re not as into these things as you’re not following them all the time. You might not have the technical background to understand what’s actually going on there. And so, you’re worried about falling for these things that look ridiculous, especially when you’re seeing it on the news all the time. And it’s playing into potentially these political narratives and stuff. It’s like: I’m not the kind of person who’s going to get fooled by this. So, I’m going to be extra certain.

JK: I spend all day, every day looking at this stuff, constantly for years. And sometimes I’m like: I don’t know. I don’t know. And so if you have heard about it, but you’re not, obsessively zooming in on images. I mean, some of them are very obvious. Most of them are very obvious, but a lot of them aren’t. And so I totally understand why people can’t tell the difference always.

PM: No, I definitely agree. I want to end by zooming out our conversation a bit more. We’ve been talking a lot about Facebook, in particular, and what is happening on Facebook, but obviously this is not just a Facebook problem. We’re seeing more and more of this AI generated stuff on other social media platforms, whether it’s Twitter or you wrote about seeing it on LinkedIn as well. You’ve also written a story about these free, ad-supported streaming platforms using AI generated or AI content that’s made with the support of AI. I guess we can put it that way.

Obviously, we had this big scandal recently with Google introducing its AI overviews onto the platform and everything that happened there. And there’s this broader conversation that’s happening about the state of the Google Search platform, because it’s having such a difficult time dealing with all of this AI generated content that is on the web and parsing it and what’s going to happen with it, building on this broader issue of SEO content that’s been there for a while, and that’s been flooding Google search. What do you make of the state of the internet at the moment and what generative AI is doing to it?

JK: You’ve done incredible work on this topic, as well. I think that it’s the most important story of the moment, which is why we keep writing about it and the reason I say that is there is an energy, not just to the tech companies wanting to shove this into every product and every corner of the internet, but also into the backlash of it, and I think that, like crypto before, like the metaverse, all of these other trends, there is so much money being spent trying to force people to use this. And I think that unlike crypto and unlike the metaverse, there is a wow factor associated with some of the AI stuff that we’re seeing.

And there’s especially a thirst from CEOs and companies to find efficiencies and to fire workers or replace workers. And if not to fire them, to make them way more efficient require them to create much much much more stuff. And I think that the backlash has been such that if people keep calling attention to it, if they keep saying we don’t want this, if we keep saying there is value in the human written word or the human generated art. Hate to even say generated! They’re human made art, human music, human, all this stuff.

At the very least, I think that we can create a system where other people value that art and purposefully seek it out. I think that the entire internet has not been taken over by AI generated content yet, but it is very easy to see how this could extend to every platform, every service, everywhere that we go on the internet. I think that if we keep calling it out and making these companies embarrassed to roll out half-assed products and to have their platforms taken over by horrible AI stuff, that is worthwhile.

I will also say that I think that what’s happened on Facebook is what happens when a company doesn’t care and doesn’t try. Facebook is been around for 20 years. It is one of the oldest social media platforms that is still relevant today. If not the oldest, I can’t think of anything else that’s still going. And in many ways, I think it’s a peek into the future. It’s what happens when you allow these platforms to decay. If you continue to allow Google to decay, if you continue to allow Twitter to decay, if you continue to allow YouTube to decay, this is what we’ll have.

You log onto the internet and you won’t be able to find anything and everything that you find will be one person’s 90 million images that they generate and post it all at once and finding stuff will be a lot harder. I think that there is an opportunity for people like us, like 404 Media and like Tech Won’t Save Us, who are humans doing real work! Tell your friends about us. I think that the social media platform for 404 Media is like, The group chat or email or RSS or these old technologies where where we can find people directly and our best scale play is word of mouth. Please tell your friends about us! That is what we want.

We’re on all of these platforms, but I think that the hope that we will publish a podcast or write an article or do Do a social media post that will go mega viral and will suddenly save journalism. And, it’s not going to happen, but what will happen is like: Oh, these people are real people who are doing real work. Please support them. So I think that there’s both a challenge and an opportunity if you want to, six in one hand, half a dozen in the other. There’s a backlash and the internet is getting way worse. But I think that people don’t like what is happening. And that presents an opportunity for human made

PM: I definitely agree with you. There was already this growing skepticism and annoyance with this platform economy and the state of the internet that had built up, post Dot-com crash, in the 2000s is when a lot of these platforms really started to build and take off and try to cement themselves. And then that really accelerated in the 2010s, of course, but it feels like a lot of those platforms have been decaying or, or have had a lot of problems for a while now. But they were so dominant that you couldn’t displace them.

But it really feels over the past few years, whether it is the effect of the AI generated content on the social platforms and what has been happening with content moderation and the people leading them, or the fact that, say, Amazon is getting filled with all this garbage, fake stuff and whatnot, these listings that are really terrible products that are misleading. And now of course how the AI is messing with the Kindle Books and what’s available on the platform more generally. It feels like this whole platform ecosystem is starting to crumble, but it’s hard to see exactly where else people go or what replaces it. And maybe that’s just natural in a time of decay. You don’t know what is going to be the phoenix that comes out of it at the other end. But, I certainly I agree with you that I feel hopeful that there are opportunities to build something better.

But then I also wonder about, again, as we’ve been talking about all those hundreds of millions or billions of people who are still on a platform like Facebook, and can they make that move? Or how are we thinking about what the future of their internet is going to look like? And like to bring people like that along with us. And I feel like that is, I think there’s an opportunity, but I think that that is a challenge for us in a good way to think about how, whatever this future is going to be is something that is inclusive of the types of people who just came online with these big platforms and weren’t there before as well, which I think is an exciting prospect.

JK: I think you’re absolutely right. I think also, even if you think about it cynically, let’s say I wanted to sell a bunch of ads. Let’s say I’m a company and I want to sell a bunch of trinkets. All of these companies are paying Facebook and Google tons of money to have their ads on AI generate content written for an algorithm that’s being consumed by bots. They’re throwing so much money in the trash because they are just advertising on dead and spam websites that are being fed primarily, there have been studies on this. I don’t have them in front of me, but it’s like a lot of the programmatic ads on the internet are fed to no one. They’re just fed to other bots. It’s scam traffic, so on and so forth. That’s the business model of Facebook. That’s the business model of Google. They’re fabulously wealthy companies.

They sell way more ads than any company I’ve ever worked for in journalism has ever sold. And I think that’s why we’re seeing influencers rise, like the YouTube influencer, the podcaster, etc. This is nothing new. I’m not saying anything new. But people are seeking out Joe Rogan. I’m not saying it’s good. It’s good, bad, everything in between. But I think that a lot of these YouTubers are also moving off platforms, even. They’re trying to own their audience. They’re trying to reach them directly. People are fed up with only being able to reach people via algorithm. I don’t know how you do the lifeboat situation and get people off of Facebook and say: Hey, type this URL into your web browser instead, but I think that that’s the future. I don’t know, I’ll print out our articles and staple them to the nearest tree and hope that people read them, I guess. Might have as much luck as tweeting it.

PM: I’ve talked to Nora Kenworthy recently, and she talked about how in the early days of GoFundMe, they actually had a thing that you could print off that would have the URL for your GoFundMe page that you could put up around your neighborhood. And I never knew that that was a thing, but maybe that is the future of how we do media, as well, printing these things off and sticking them on the light pole or whatever around the neighborhood.

Jason, it’s always fascinating to talk to you. I’ve been loving reading about all this. So it’s great to talk further about it and to dig into it with you. And I’ll be looking forward to the future crazy things that you find in the AI generated world on Facebook. So thanks so much for taking the time and coming on the show.

JK: Thanks so much. I love the show. I love coming on. Very thankful.