Unknown

Key Concepts

• HCI (Human-Computer Interaction)
• Security (Confidentiality, Integrity, Availability)
• Threat Models
• Killchain
• Online Hate and Harassment
• Misinformation and Disinformation
• Utility and Reciprocation
• Toxic Content
• Relevant Content
• Non-Consensual Sexual Deepfakes
• Underground Economy
• Face Sets
• GPU (Graphics Processing Unit)
• Bulletproof Hosting Providers
• Perspective API
• Originating Content
• Propagating Content

Introduction

The speaker discusses their journey from traditional network security research to addressing online hate, harassment, misinformation, and disinformation. They explore the parallels between these issues and traditional security problems, arguing that online abuse can be viewed as a form of attack.

Security vs. Online Abuse: A Shift in Perspective

• Traditional Security Focus: Confidentiality, integrity, availability, threat models with defined attackers (e.g., nation-states, financially motivated actors).
• Challenge: Online hate and harassment didn't initially fit this model. The research community was resistant to considering it a security problem.
• New Perspective: Framing online hate and harassment as attacks with aggressors and targets.
• Similarities: Both involve targeting individuals or groups, and there are models for understanding attackers.
• Example: Spam and phishing are abuse problems that are treated as security problems. Online censorship can also be viewed as a security issue.

Applying Security Principles to Online Abuse

• Killchain Analogy: Disinformation campaigns can be seen as attacks against users' cognition, with attackers and targets.
• Real-World Example: Meta's killchain for combating influence operations is similar to traditional cybersecurity killchains.
• Convergence: Traditional cyberattacks (e.g., ransomware) are increasingly combined with disinformation and online harassment.
• Key Question: Can cybersecurity techniques be used to tackle online abuse and misinformation?

Case Study 1: Supporting Journalists on Social Media

• Problem: Journalists, especially those from marginalized identities, face a high volume of hateful messages on social media, which is critical for their work.
• Existing Tools: Blocking tools are often ineffective and conflict with journalistic norms of engaging with the audience.
• Key Insight: Journalists need to manage toxic content, not just filter it out.
• Two Axes:
  • Toxicity (toxic vs. non-toxic)
  • Relevance (relevant vs. irrelevant)
• Solution: Focus on surfacing relevant content, even if it's toxic.
• Unexpected Finding: Some journalists want to see certain toxic content (e.g., death threats) to ensure their physical safety.

Case Study 2: The Underground Economy of Deepfakes

• Platform: Mr. Deepfakes, a large open marketplace for non-consensual sexual deepfakes.
• Scale: 40,000 videos, 1.5 billion views.
• Pricing: $50-$200 for a deepfake video.
• Key Finding: Beyond sexual gratification, a significant motivation is creating "art" for the community.
• Community Aspects: Tutorials, support, and a welcoming environment.
• Limiting Factors: Availability of "face sets" (data for training) and GPU hardware.
• Tools: DeepFaceLab and FaceSwap are popular tools.
• Cat-and-Mouse Game: Google tries to ban deepfake libraries on Colab, and the community finds ways to bypass the protections.
• Emergence of Bulletproof Hosting: Expect bulletproof hosting providers for GPUs to emerge.

Case Study 3: Abuse on Reddit: A Global Perspective

• Data: Analysis of abuse across Reddit over time.
• Key Statistics:
  • 50% of Reddit accounts interact in a thread with hate or harassment.
  • 3% of accounts post toxic content.
  • These 3% account for 1/3 of all comments.
  • 90% of toxic comments are one-offs.
• Personas of Attackers:
  • Occasional posters (one-offs) - 70% of toxic comments
  • Moderate posters
  • Serial abusers - 4% of accounts
• Implication: Different solutions are needed for different types of abusers.

Case Study 4: News and Disinformation

• Approach: Analyzing every news story on the internet to uncover connections.
• Insights: Identifying who is successful at originating and propagating stories.
• Finding: There is not necessarily a strong correlation between site popularity and who is good at originating content.

Conclusion

• Success in Applying Security Principles: The security community has made progress in understanding and addressing online abuse and misinformation.
• Importance of a Global Perspective: Zooming out and analyzing broader ecosystems reveals patterns that are missed when focusing on individual incidents.
• Nuance is Key: There is nuance in what constitutes abuse, how users interact, and the trustworthiness of news sources.
• Need for Gray-Area Thinking: The security community needs to be comfortable with the gray areas of online abuse and misinformation, as these are often the hardest problems to solve.
• Call to Action: The security industry should tackle these problems as a whole.