Why AI is threatening student privacy--and how we can do better | Mary Mason | TEDxWUSTL

Key Concepts

AI-powered safety monitoring systems, student mental health, data privacy, FERPA, Fourth Amendment, unintended consequences, interdisciplinary approach, false positives, chilling effect.

Main Topics and Key Points

Introduction: The Case Study

• A hypothetical scenario is presented where a college student is flagged as a potential school shooter due to AI-powered safety monitoring systems analyzing their emails, web searches, and social media posts.
• The student's online activity (searching for guns for a class assignment and jokingly saying they wanted to "kill" a professor) triggers a red flag, leading to questioning by campus police and potential long-term consequences for their academic and professional future.
• This scenario highlights the potential risks and unintended consequences of using AI to monitor students' online activities.

The Rise of Safety Monitoring Platforms

• The use of safety monitoring platforms in schools (K-12) and colleges is increasing, especially after the pandemic-induced shift to online and hybrid learning.
• These platforms scan online activities to identify students at risk of self-harm, cyberbullying, or potential violence.
• Companies like Bark, Securly, GoGuardian, and Gaggle are major players in this market, serving over 50,000 schools across the US.

The Promise and Peril of AI in Education

• AI-powered monitoring systems are presented as a "solution in a box" for educators struggling with the rising number of students facing mental health issues and limited resources.
• These systems offer unprecedented visibility into a student's mental health and well-being.
• However, the accuracy and effectiveness of these systems are questioned, with a report by the Future of Privacy Forum stating that self-harm and violence monitoring systems are an "unproven technique."

Data Privacy and Legal Concerns

• Concerns are raised about where schools store mental health alerts (both true and false), who has access to them, how long they are retained, and whether they are included in students' educational records.
• The Family Educational Rights and Privacy Act (FERPA) is discussed as a federal law protecting student educational records, but it only applies to schools receiving federal funding.
• The Fourth Amendment of the US Constitution is mentioned, raising questions about whether monitoring software on school-issued laptops constitutes an unreasonable search and seizure.

Unintended Consequences and Potential Harms

• Vulnerable students could be further harmed if strong guardrails and privacy policies are not in place.
• Students could be unfairly targeted by educators or fellow students due to bias around mental health.
• Law enforcement or social services could needlessly get involved.
• Students could face potential disciplinary actions.
• The "chilling effect" of constant monitoring could discourage students from using their laptops for learning.
• The example of Securly being subject to a class-action lawsuit for selling student data to third parties without consent is cited.

The Need for an Interdisciplinary Approach

• The current situation is described as a triangle of intersecting bodies: tech companies, educators, and students.
• Educators are implementing these products without students and parents knowing the full extent of their use, potentially leading to a loss of student privacy.
• The speaker emphasizes the need for an interdisciplinary team (education, medical, legal, and technology) to develop and implement AI solutions in education responsibly.
• This interdisciplinary lens is crucial for ensuring that AI is used ethically and effectively in the educational setting.

Reimagining the Case Study with an Interdisciplinary Team

• The opening scenario is revisited, but this time, instead of campus police, a trained crisis counselor contacts the student.
• The counselor determines it's a false alert, and the interdisciplinary team reviews the incident to learn from it and modify the algorithm.
• This revised scenario illustrates how an interdisciplinary approach can lead to a more positive outcome, avoiding the negative consequences of the initial scenario.

Notable Quotes

• "Safety monitoring platforms powered by artificial intelligence, they have become the solution in the Box. They are that answer that educators are looking for with the skyrocketing number of students suffering from mental health issues in our strained school systems."
• "The self harm in violence monitoring systems are an unproven technique for effectively identifying and assisting students flagged" - Future of Privacy Forum
• "Our solution in a box can now become Pandora's Box."

Technical Terms and Concepts

• AI-powered safety monitoring systems: Software that uses artificial intelligence to analyze student online activity and identify potential risks.
• FERPA (Family Educational Rights and Privacy Act): A federal law protecting the privacy of student educational records.
• Fourth Amendment: Protects against unreasonable searches and seizures.
• Chilling effect: The discouragement of certain behaviors due to fear of surveillance or reprisal.
• False positive: An incorrect identification of a risk or threat.

Logical Connections

The video begins with a specific example to illustrate the potential problems with AI monitoring. It then broadens to discuss the prevalence of these systems, the reasons for their adoption, and the concerns surrounding their use. The discussion then narrows to specific legal and ethical considerations, before returning to the initial example to propose a better outcome through an interdisciplinary approach.

Data, Research Findings, or Statistics

• 69% of public schools have reported an increase in the number of students seeking mental health services (National Center for Education Statistics).
• Millions of students attend schools with no school counselors, nurses, psychologists, or social workers (American Civil Liberties Union report, 2019).
• Bark, Securly, GoGuardian, and Gaggle have engaged over 50,000 schools K through 12 across the country.

Synthesis/Conclusion

The video argues that while AI-powered safety monitoring systems hold promise for addressing student mental health issues, their implementation raises significant ethical, legal, and practical concerns. The lack of proven effectiveness, potential for unintended consequences, and risks to student privacy necessitate a more cautious and interdisciplinary approach. The speaker advocates for a collaborative effort involving educators, medical professionals, legal experts, and technologists to ensure that these systems are used responsibly and effectively, protecting students' rights and well-being.