Not long ago, it seemed like the conversation surrounding online moderation couldn’t get more intense. Then came the Facebook Files, followed shortly by the Facebook Papers, whose findings took the debate to a whole new level. Online moderation has now joined climate change and gun control as a top issue in the US. This makes sense: moderation touches on a host of other major concerns, like extremism, polarization, science denialism, and more.
But at this pivotal moment, it’s important to acknowledge that moderation, while essential, can only get us so far. Perhaps we can agree that social media has made polarization worse, but it didn’t invent the problem: fundamentally, this is an offline issue. To truly combat what Francis Fukuyama has called “epistemic relativism”—i.e., the erosion of shared truths about reality in our society—we must look outside the walls of social media platforms, in civic education and digital literacy. That said, it is still worth exploring what can be done within the platforms to stem these ill effects.
Fixing our conversation is a massive task. We are in a predicament without easy solutions (or, in some cases, any clear solutions at all). This may go a long way towards explaining another recent trend: social media platforms, together with civil society groups, working to de-platform voices seen as responsible for spreading disinformation.
These actions have affected a range of communities: domestic extremists, anti-vaccine advocates, adherents to the QAnon conspiracy theory. On the surface, silencing these groups makes sense: if someone is peddling dangerous falsehoods, why not pressure the platforms to stop them? But there’s a problem here. Removing a group or public figure might make a platform more pleasant to its users in the short term, but recent evidence suggests it may also make the rest of the world a bit more dangerous in the long run.
You can see a good demonstration of this phenomenon in a paper from last summer. It examined what happened when Reddit banned two controversial communities—the r/Incels and r/TheDonald subreddits. What the researchers noticed was troubling. After getting banned, r/TheDonald’s members reformed on a new website, thedonald.win. There were fewer members than there had been on Reddit, but these users were increasingly radical—”more toxic, negative, and hostile” about democrats and leftists. Later, in the weeks leading up to January 6th Capitol attack, users would go on to share maps of the Congressional tunnel systems and advocate violence.
We can draw a number of conclusions here. When large, potentially dangerous groups are deplatformed, they end up in places with little to no moderation—alternative social networks like Gab or Parler, or private group chats on Telegram. Here, things can spiral out of control, as true believers egg one another on towards increasingly radical views. Whatever the downsides of platforms like Facebook and Twitter, they at least partner with global consortiums of fact-checkers and civil society organizations to provide fact-based counternarratives to views like these. In the darkest corners of the internet, these views stand unchallenged. (And unlike a Facebook page, a network of private Signal chats can be difficult for law enforcement to monitor.)
All of this creates an environment where new users are ripe for radicalization: nothing binds people together more than a shared sense of persecution. In the long run, this kind of deplatorming could very easily lead to increased—and lethal—collaboration between radicalized communities online.
This is not in any way to suggest that hate speech should be protected on any platform. Nor is it a wholesale argument against “de-platforming.” But we must stay cognizant of the risks inherent in pushing entire communities into the shadows. January 6th was, in part, one consequence of this tendency; and there will inevitably be others, if we don’t keep these risks in mind.
“De-platforming” is often presented as a simple choice: either kick the offending party off of your platform or let them stick around and poison the well. It shouldn’t be that way—and often it really can’t be as the definition of “offending party” is highly relative. For example, Spotify’s employees might take offense to Joe Rogan’s presence on their platform, but millions of listeners clearly find his views acceptable. Instead of removing these actors and groups, we need to find ways to incentivize and elevate positive user contributions to these conversations. As we now know, Facebook has done just the opposite—often, toxic forums were advertised to users on their News Feeds. To begin with, we must make sure social media platforms’ algorithms respect quality over engagement, as the most engaging material is often the most inflammatory. As Tristan Harris and others have noted, freedom of speech does not need to guarantee freedom of reach.
The ethics of moderation are hotly contested and in constant flux: no universal consensus exists. It will take the coordinated work of nonprofits, government agencies and the platforms themselves to arrive at some workable standard—to find the ideal balance between respect for the right to free speech and the safety of every user of a given platform.
We can bemoan this epistemic relativism – the erosion of ground truth – that got us into this situation, but we can’t wish it away: no algorithm tweak or clever moderation tool can correct the mistakes of history. What moderation can do is help establish community norms. Learning how to live with each other on these platforms—doing what we can to prevent further radicalization while working to stamp out harassment and hate speech—is among our most urgent tasks. And as long as the platforms continue to prioritize engagement metrics over safety, we will continue to fail at it.
