Zeynep Demirbas
8th Grade, Transit Middle School
East Amherst, NY
Evaluating the Reliability of Large Language Models for Stress Detection
View Poster
Project Background
Zeynep has been fascinated by ChatGPT since it was released in 2022. She was especially interested in how some users and businesses felt it could be trusted—even with mental health. One of Zeynep’s family friends is a psychologist. “She mentioned how health insurance companies were exploring large language models (LLMs) as cheaper 24/7 alternatives to human therapists,” Zeynep says. “She was concerned about her job safety.” Zeynep wondered if today’s LLMs like ChatGPT could replace therapists and decided to start by asking if the programs could detect stress.
Tactics and Results
Zeynep wanted to compare several LLMs to see if they could detect stress in human text. She used a Dreaddit dataset, which is a group of 3,553 Reddit posts that humans have labeled with stress or no-stress, and gave the data to four different LLMs: Bidirectional encoder representations from transformers (BERT), a version of BERT called MentalBERT for mental health, Random-Forest (a basic machine learning technique), and Chat-GPT4o. She asked the models to identify stress and score their confidence. Zeynep found that MentalBERT performed best, identifying stress precisely 82.4 percent of the time. BERT performed at 79.4 percent. ChatpGPT-4o identified stress precisely only 73.7 percent of the time, worse than the basic machine learning Random-Forest, which doesn’t understand language or context at all. “My project shows that LLMs are currently unreliable and unsafe to deploy as diagnostic tools,” Zeynep says.
Beyond the Project
Zeynep is on the track team and holds school records in shot put. She plays the viola in a chamber group and likes to play chess online, playing with people around the world. She also helps her mom make traditional Turkish meals. “One of my favorite memories is back when I was learning to make ‘sarma,’” she says. “We laughed as I struggled to roll grape leaves.” Zeynep would like to be a computer scientist. “I enjoy writing programs, but I’m even more excited by what they mean in the real world.”
