Overreliance on Automated Filters Would Push Victims Off of the Internet
In all of the debate about the Stop Enabling Sex Traffickers Act (SESTA, S. 1693), there’s one question that’s received surprisingly little airplay: under SESTA, what would online platforms do in order to protect themselves from the increased liability for their users’ speech?
With the threat of overwhelming criminal and civil liability hanging over their heads, Internet platforms would likely turn to automated filtering of users’ speech in a big way. That’s bad news because when platforms rely too heavily on automated filtering, it almost always results in some voices being silenced. And the most marginalized voices in society can be the first to disappear.
Section 230 Built Internet Communities
The modern Internet is a complex system of online intermediaries—web hosting providers, social media platforms, news websites that host comments—all of which we use to speak out and communicate with each other. Those platforms are all enabled by Section 230, a law that protects platforms from some types of liability for their users’ speech. Without those protections, most online intermediaries would not exist in their current form; the risk of liability would simply be too high.
Section 230 still allows authorities to prosecute platforms that break federal criminal law, but it keeps platforms from being punished for their customers’ actions in federal civil court or at the state level. This careful balance gives online platforms the freedom to set and enforce their own community standards while still allowing the government to hold platforms accountable for criminal behavior.
When platforms rely too heavily on automated filtering, it almost always results in some voices being silenced. And the most marginalized voices in society can be the first to disappear.
SESTA would throw off that balance by shifting additional liability to intermediaries. Many online communities would have little choice but to mitigate that risk by investing heavily in policing their members’ speech.
Or perhaps hire computers to police their members’ speech for them.
The Trouble with Bots
Massive cloud software company Oracle recently endorsed SESTA, but Oracle’s letter of support actually confirms one of the bill’s biggest problems—SESTA would effectively require Internet businesses to place more trust than ever before in automated filtering technologies to police their users’ activity.
While automated filtering technologies have certainly improved since Section 230 passed in 1996, Oracle implies that bots can now filter out sex traffickers’ activity with near-perfect accuracy without causing any collateral damage.
That’s simply not true. At best, automated filtering provides tools that can aid human moderators in finding content that may need further review. That review still requires human community managers. But many Internet companies (including most startups) would be unable to dedicate enough staff time to fully mitigate the risk of litigation under SESTA.
So what will websites do if they don’t have enough human reviewers to match their growing user bases? It’s likely that they’ll tune their automated filters to err on the side of extreme caution—which means silencing legitimate voices. To see how that would happen, look at the recent controversy over Google’s PerspectiveAPI, a tool designed to measure the “toxicity” in online discussions. PerspectiveAPI flags statements like “I am a gay woman” or “I am a black man” as toxic because it fails to differentiate between Internet users talking about themselves and making statements about marginalized groups. It even flagged “I am a Jew” as more toxic than “I don’t like Jews.”
See the problem? Now imagine a tool designed to filter out speech that advertises sex trafficking to comply with SESTA. From a technical perspective, creating such a tool that doesn’t also flag a victim of trafficking telling her story or trying to find help would be extremely difficult. (For that matter, so would training it to differentiate trafficking from consensual sex work.) If Google, the largest artificial intelligence (AI) company on the planet, can’t develop an algorithm that can reason about whether a simple statement is toxic, how likely is it that any company will be able to automatically and accurately detect sex trafficking advertisements?
Despite all the progress we’ve made in analytics and AI since 1996, machines still have an incredibly difficult time understanding subtlety and context when it comes to human speech. Filtering algorithms can’t yet understand things like the motivation behind a post—a huge factor in detecting the difference between a post that actually advertises sex trafficking and a post that criticizes sex trafficking and provides support to victims.
This is a classic example of the “nerd harder” problem, where policymakers believe that technology can advance to fit their specifications as soon as they pass a law requiring it to do so. They fail to recognize the inherent limits of automated filtering: bots are useful in some cases as an aid to human moderators, but they’ll never be appropriate as the unchecked gatekeeper to free expression. If we give them that position, then victims of sex trafficking may be the first people locked out.
At the same time, it’s also extremely unlikely that filtering systems will actually be able to stop determined sex traffickers from posting. That’s because it’s not currently technologically possible to create an automated filtering system that can’t be fooled by a human. For example, say you have a filter that just looks for certain keywords or phrases. Sex traffickers will learn what words or phrases trigger the filter and avoid them by using other words in their place.
New developments in AI research will likely make filters a more effective aid to human review, but when freedom of expression is at stake, they’ll never supplant human moderators.
Building a more complicated filter—say, by using advanced machine learning or AI techniques—won’t solve the problem either. That’s because all complex machine learning systems are susceptible to what are known as “adversarial inputs”—examples of data that look normal to a human, but which completely fool AI-based classification systems. For example, an AI-based filtering system that recognizes sex trafficking posts might look at such a post and classify it correctly—unless the sex trafficker adds some random-looking-yet-carefully-chosen characters to the post (maybe even a block of carefully constructed incomprehensible text at the end), in which case the filtering system will classify the post as having nothing to do with sex trafficking.
If you’ve ever seen a spam email with a block of nonsense text at the bottom, then you’ve seen this tactic in action. Some spammers add blocks of text from books or articles to the bottom of their spam emails in order to fool spam filters into thinking the emails are legitimate. Research on solving this problem is ongoing, but slow. New developments in AI research will likely make filters a more effective aid to human review, but when freedom of expression is at stake, they’ll never supplant human moderators.
In other words, not only would automated filters be ineffective at removing sex trafficking ads from the Internet, they would also almost certainly end up silencing the very victims lawmakers are trying to help.
Don’t Put Machines in Charge of Free Speech
One irony of SESTA supporters’ praise for automated filtering is that Section 230 made algorithmic filtering possible. In 1995, a New York court ruled that because the online service Prodigy engaged in some editing of its members’ posts, it could be held liable as a “publisher” for the posts that it didn’t filter. When Reps. Christopher Cox and Ron Wyden introduced the Internet Freedom and Family Empowerment Act (the bill that would evolve into Section 230), they did it partially to remove that legal disincentive for online platforms to enforce community standards. Without Section 230, platforms would never have invested in improving filtering technologies.
However, automated filters simply cannot be trusted as the final arbiters of online speech. At best, they’re useful as an aid to human moderators, enforcing standards that are transparent to the user community. And the platforms using them must carefully balance enforcing standards with respecting users’ right to express themselves. Laws must protect that balance by shielding platforms from liability for their customers’ actions. Otherwise, marginalized voices can be the first ones pushed off the Internet.
