In one video, British far-right leader Tommy Robinson talks directly to camera. In another, a user of the popular Minecraft video game builds a replica of the Auschwitz concentration camp. A third shows graphic images of an ISIS militant about to behead two Japanese men.
Combined, these extremist videos — of which more than 1,000 were found in June alone — garnered millions of views on TikTok, the Chinese-owned social network. Four out of five of these videos were not removed before they were flagged to the company by the Institute for Strategic Dialogue, a think tank that tracks online extremism, which on Tuesday published a report on the rise of extremism on TikTok.
“TikTok offers a platform for this explicit content to reach new audiences and potentially reach younger audiences, which was quite worrying,” said Ciaran O’Connor, a researcher for the nonprofit, which monitors the rise of white supremacist, anti-Semitic and other extremist material.
“TikTok has shown signs that they have grown up fast and they’ve learn the lessons from other platforms about the possible pitfalls,” he added. “But this research shows they still have a way to go.”
The social network is not alone in finding it difficult to stop extremists from promoting hate on its global platform.
Facebook, Twitter and Google’s YouTube similarly have struggled to stop a tidal wave of violent and harmful material from spreading online despite increased efforts to remove everything from videos from the Proud Boys, an American white supremacist group, to propaganda from the Taliban. The companies have employed tens of thousands of contractors to manually monitor fringe groups and have turned to machine learning tools to automatically flag the most heinous material.
Still, the latest research from the Institute for Strategic Dialogue highlights how these groups — spread across the European Union and the United States — have taken advantage of TikTok’s user-friendly features, including the ability to create eye-catching viral content within seconds on a smartphone, to bring their hateful messages to millions of mostly-young users.
That includes relying on the social media company’s so-called duet function, which allows people to splice their own videos alongside those of others, to promote neo-Nazi and fascism imagery, as well as attacks aimed at George Floyd, the Black man who was killed by U.S. police officers last year. Extremists groups have also piggybacked on the widespread use of music on TikTok to splice white supremacist and anti-Muslim songs into their viral videos, based on Tuesday’s report.
Weaponizing TikTok features
In response, TikTok said that it had removed almost all of the extremist videos and user accounts that the researchers had discovered, and that it had removed more than 90 percent of content, in the first quarter of 2021, that broke the company’s content policies within 24 hours of such material being posted.
“TikTok categorically prohibits violent extremism and hateful behavior, and our dedicated team will remove any such content as it violates our policies,” a company spokesperson said in a statement, adding that it welcomed the Institute for Strategic Dialogue’s research.
To check how much extremist content was shared on the Chinese-owned platform, the think tank started with a database of keywords used by these groups, and then searched TikTok for accounts that promoted white supremacist, anti-Semitic, racist or other harmful content. In total, the researchers found 491 accounts that had shared a combined 1,030 extremist videos between June 4 and June 30.
That represents a mere rounding error to the millions of posts on TikTok each day. But the level of sophistication of extremists’ tactics, according to O’Connor, the researcher, allowed these videos to sidestep the company’s efforts to clamp down on hateful material from spreading within its users. That included using proxies to promote hateful messages without explicitly using banned words or images on TikTok.
Instead of posting images of Adolf Hitler, for instance, neo-Nazis peppered the social network with photos and videos of Oswald Mosley, the British fascist leader from the 1930s who had close ties to the Third Reich. To attack the feminist movement, TikTok users also routinely shared images of Ted Kaczynski, known as the Unabomber, who has become a poster child for extreme misogyny online.
Yet it was the extremists’ use of features only available on TikTok that set apart their ability to share harmful content compared to other platforms like Facebook and Twitter.
In one example, a TikTok account used the platform’s so-called switch function, which allows people to use others’ videos in their own content, to attack the LGBTQ+ community by combining a video about someone describing how they had told friends and family about their sexual orientation with their own false claims that the LGBTQ+ community had higher levels of suicide compared to the rest of society.
“The way that these featured on Tiktok are being used by all creators does lend itself to extremist creators,” O’Connor said. “These seem to be naturally created trends. But the challenge for TikTok is how do you effectively police against extremists taking advantage of them?”