Content and Character:
AI as an Asset and Adversary in UGC Moderation

As we stand on the cusp of a new technological era, rapid developments in AI are not just reshaping business, recreation, communication, and social interaction; they fundamentally redefine digital transformation. As the media constantly reminds us, AI’s advanced capabilities and expanding applications herald both a blessing and a potential curse, introducing a new set of complexities and threats we must address head-on.

Businesses, governments, and other organizations that use and encourage user-generated content are particularly vulnerable. In an era of viral marketing and social influencers, UGC has become an integral part of many business strategies. From social media platforms to online marketplaces to “traditional” commercial endeavors, UGC can significantly enhance customer engagement, brand loyalty, and revenue. However, it also provides new tools that devious characters can use to spread malicious content.

Malicious actors constantly evolve and refine their tactics and leverage advanced technologies to manipulate UGC and exploit online platforms. AI has allowed them to extend their reach and amplify their impact.

Stealthy, Mass-Produced Disinformation

AI empowers rapid-fire, mass-produced, hard-to-detect misinformation, inflammatory content, and other attacks on our platforms.

In the past, a coordinated disinformation campaign might have required a team of people working around the clock to create and spread false narratives — writers, artists, printers, and others. Nowadays, one person with even rudimentary communication and technical skills can access AI tools that match or exceed that output.

The digital age enables even more widespread and coordinated disinformation. The 2016 US presidential election saw Russian operatives use social media to disseminate lies and sow discord among American voters. More recently, users shared false news stories, deep-fake videos, misleading memes, and pseudoscience, influencing public opinion about the COVID-19 virus and vaccines and potentially undermining public health.

The sheer scale and speed at which malicious content can now be generated and disseminated presents one of AI’s most immediate and visible impacts on the threat radar. It is not hard to imagine a disgruntled employee, trailing political candidate, or bigoted loudmouth generating thousands of unique pieces of harmful content — each directed at impressionable targets — in just a few hours.

AI Moderation vs. AI Malfeasance

While villains use AI as a spear to penetrate social media, news sites, and digital platforms, heroes can employ similar technology as a shield to deflect these underhanded efforts. AI also provides powerful solutions to detect and counteract the spread of misinformation and maintain trust in information ecosystems:

• Content Verification: AI’s natural language processing (NLP) algorithms can analyze the semantic structure of text to identify patterns consistent with fake news. Fact-checking AI tools cross-reference claims with reliable databases and flag discrepancies in real-time. Moreover, AI can quickly scan large volumes of data, enabling rapid identification of suspicious content that might otherwise slip through human scrutiny.
  
• Deepfake Detection: AI-informed detection tools can analyze the subtle imperfections ingrained in AI-generated and manipulated videos that are imperceptible to the human eye. These tools examine facial movements, eye-blink patterns, and audio inconsistencies to differentiate between genuine and deepfakes created to embarrass and slander people or slant the way we perceive events.
  
• Network Analysis: AI excels at analyzing complex information flow. By mapping how false information spreads through social networks, AI can identify its origins and compromised nodes. Machine learning detects abnormal patterns in sharing and interactions, flagging potential sources of misinformation for platform early intervention.
  
• Behavioral Analysis: Users who propagate false information often exhibit common characteristics and tactics. AI can analyze users’ posting habits, interaction routines, and network connections to find accounts that exhibit suspicious activity. This helps platforms uncover bot accounts and foil coordinated disinformation campaigns for monitoring or removal.
  
• Speed and Accuracy: AI-driven content moderation tools can find and filter out harmful content faster and more reliably than their human counterparts. As machine learning becomes more sophisticated, AI can consistently categorize and assess content based on predefined guidelines. Automating the moderation process lets platforms manage a massive influx of content more efficiently.