Automated systems fight ISIS propaganda, but at what cost?

eGLYPH was developed by Hany Farid, professor of computer science at Dartmouth College, and is modeled on the PhotoDNA algorithm that Farid created with Microsoft in 2008 to combat child pornography. Both PhotoDNA and eGLYPH are capable of recognizing images even if they’ve been resized or altered, but eGLYPH extends the technology to video and audio. (YouTube’s ContentID system uses similar technology to identify copyright-infringing videos.) eGLYPH is in its final stages of testing, and will be made freely available to social media companies. The CEP is also compiling a database of extremist content, called the National Office on Reporting Extremism (NORex), which it hopes will become a "comprehensive" resource for researchers and social media companies.

Mark Wallace, the CEO of the CEP and a former ambassador to the UN under President George W. Bush, describes eGLYPH as a "game-changer." He believes the algorithm, if adopted widely, could help stem the spread of terrorist propaganda and dissuade radical groups from posting extremist content in the first place.

"If an extremist group knows that the moment they try to post a video online, that it will be immediately removed and it won’t have that viral reach, perhaps it’s no longer compelling for them because they can no longer accomplish their propaganda aims," Wallace says.

Lawmakers in Europe and the US have called on tech companies to more proactively police terrorist propaganda over the past year, following a spate of high-profile attacks. Last month, a panel of British lawmakers said in a report that Facebook, Twitter, and other tech giants are "consciously failing" to combat ISIS online, echoing previous statements from leaders in France and Germany. "No group has been as successful at drawing people into its message than [ISIS]," Michael Steinbach, executive assistant director at the National Security Branch of the FBI, said in a July interview. 

Far-right extremism has become a growing concern, as well. Germany has pressured Facebook and other companies to more swiftly remove xenophobic content and other material targeting refugees; and a study published last week by George Washington University’s Program on Extremism found that neo-Nazis and American white nationalist groups continue to thrive on Twitter, even as ISIS’ influence has waned on the social network.

Tech companies have said they're committed to combatting extremism on their platforms, pointing to statistics on content removal and suspensions. Last month, Twitter announced that it has suspended 360,000 accounts for promoting terrorism since mid-2015, including 235,000 accounts since February.

The industry appears to be moving toward automation, as well. In a blog post announcing the suspensions, Twitter added that it has begun incorporating automated technologies, including "proprietary spam-fighting tools," to supplement its reporting system. In June, Reuters reported that Facebook and YouTube have quietly begun implementing automated hashing systems, though Wallace says that discussions with major social media companies about adopting eGLYPH are still ongoing.

Some experts say that automation could help limit the reach that ISIS and other groups have on social media, though there are questions over how such technologies would be implemented. Unlike child pornography, which is more easily identifiable, extremist content covers a broad spectrum, and laws governing its dissemination vary from country to country.

"No one would argue that we should allow beheading videos, but there are a lot of questions around what constitutes extremist or problematic content," says John Horgan, a professor of psychology at Georgia State University who has extensively studied ISIS and other terrorist groups.

Some worry that the algorithm would inadvertently block propaganda videos or newsworthy content. Although many news outlets have not broadcast ISIS beheading videos, some have aired parts of the clips that do not show any violence. If those clips bear the same "fingerprint" as the original video, the algorithm may automatically remove them from Facebook and Twitter.

"A lot of news organizations have broadcast portions of those videos, and have made a decision that those excerpts are newsworthy," says Vivek Krishnamurthy, a clinical instructor at Harvard Law School’s Cyberlaw Clinic. "Will those uses that are more innocent also come down automatically?"

Farid acknowledges that child porn and ISIS propaganda are "two totally different beasts," and that news videos or other non-malicious media may be inadvertently swept up by his algorithm. But he says mainstream media outlets could be whitelisted, noting that the technology can be calibrated to target any content that violates a company’s terms of service. "If Facebook came today and said we no longer want photos of kittens on our website, we could do that," Farid says.

Some find that prospect troubling. Digital rights groups have long expressed concerns over allowing Facebook and Twitter to be arbiters of free speech, and they fear that an automated system would lead to more widespread censorship. Jillian York, director for international freedom of expression at the Electronic Frontier Foundation, cautioned in an email that "automation always comes with false positives," adding that "sometimes it's not merely the technology, but also the implementation or implementers, that are flawed." Joe McNamee, executive director of the Brussels-based advocacy group European Digital Rights (EDRi), said in an interview that efforts to automate content removal may actually hinder counter-speech, which a recent Google-backed study found to be an effective way of sparking online debate.

"Is it really proportionate to scan/filter every single upload from every single European, to make sure it is legal?" EDRi wrote in a blog post in July. "If Europe takes the lead in mass surveillance and filtering of their citizens’ uploads to the internet, what hope [is there] for the open and democratic internet elsewhere in the world?"

The bigger question, perhaps, is whether removing extremist content will have any impact on radicalization. Politicians have linked online radicalization to recent terrorist attacks, as President Barack Obama did after the Orlando shooting in June. But Horgan says the connection is still fuzzy.

"Most people, I suppose, would rightfully assume that we have some sort of baseline understanding of the role that specific content plays in the radicalization process," he says. "Well guess what? We don’t." The hard drives of American jihadis, he adds, often contain more innocuous content — "porn, games, cat videos" — whereas the more gruesome material "is probably more important for the fanboys and supporters" than it is for operational members.

Still, Horgan acknowledges that the algorithm could play an important role in "reducing the footprint of terrorist propaganda," which he says would be a significant move "in a grand strategic and psychological sense." Farid expects the cat-and-mouse game to continue, as well. If major social media companies do adopt his algorithm, he has no doubt that extremist groups will try to work around it. And if they succeed, he says he’ll continue to tweak the technology.

"No matter what technology you develop, the adversary will develop a counter technology," Farid says. "I don’t consider that a problem, I consider that a reality."