Research Methods

User Experience Research

To understand the problem space and validate that this was an area worth researching, the UXR team conducted 3 short exploratory interviews each to understand students’ perspective on the WarnMe system through the categories of interpreting, impact, and needs. The initial interviews consisted of questions that then guided the researchers on what topics they wanted to do a deeper dive into for official interviews for the study and the diary study once interviews were completed.

Screening and recruitment were completed via google form through word of mouth, flyers, posts in group chats, and posts on social media. The only criteria was to be a UCB student, either as an undergraduate student, graduate student, or PhD student. Community members could also be accepted for the interview process. 10 interviews were conducted for approximately 30-45 minutes each in the topic areas of students’ comprehension of WarnMes, students' campus safety practices, interpreting and emotional impact of WarnMes on students, and the needs of students regarding WarnMes.

Once interviews concluded, a diary study was conducted. There were 11 participants in the diary study that responded to a google form for 9-11 WarnMes across two weeks. The WarnMes that the participants completed a form about were either sent out by the university during the two week time frame or were past WarnMes dating back to 2 years ago that the researchers chose to use in case not enough WarnMes were sent out by the university during the two weeks. The topic areas covered by the diary study were purposefully similar to the interview topics: comprehension of the WarnMe that was sent out, emotional impact that the WarnMe had on them, and how the WarnMe affected their needs.

To analyze the interviews, the researchers individually coded their own interviews to pull out themes for each question as well as quotes. Descriptions were made for the themes found and entered into individual spreadsheets for each researcher. From there, as a group, the researchers grouped the themes that sounded similar and placed them in categories based on the research questions.

Coding WarnMes and Natural Language Processing

The project involved analyzing WarnMe messages collected from August 2021 to October 2023 through public records requests. A codebook was developed to code the messages based on multiple categories of codes: location granularity, scene descriptions, and personal identifiers used for victims and suspects.

Four annotators each coded 200 of the 315 messages with all four annotators coding 100 of the messages. From those 100 messages Fleiss' kappa, a statistical measure for assessing the reliability of agreement between multiple annotators, was calculated. The inter-annotator agreement across all code categories showed moderate agreement between the annotators: a weighted average Fleiss' kappa of 0.4828. The codes with the highest agreement were selected for each WarnMe, and the frequency of the various code categories were analyzed.

The NLP analysis began with data preprocessing to refine the dataset by removing URLs, non-standard characters, and excess whitespace. A sentiment analysis pipeline was then implemented using the sentiment-analysis module from Hugging Face, categorizing each comment as positive or negative, and results were documented in the dataset.

We manually annotated the entire data the human way based on six emotions (joy, love, anger, sadness, fear, and surprise) which provided a baseline to evaluate the model's performance. A pre-trained model (bhadresh-savani/distilbert-base-uncased-emotion) from Hugging Face was employed to analyze emotions in the comments. The model achieved 80.65% accuracy, excelling in identifying sadness and anger but struggled with surprise and love. This indicates that there is room for improvement in recognizing less frequent or subtly expressed emotions.

To enhance sentiment analysis, three machine learning models were tested:

• Naive Bayes Classifier with TfidfVectorizer: This model showed limited success due to its simplicity.
• Support Vector Machine (SVM) with TfidfVectorizer: This model performed slightly better but still struggled with the subtleties of emotional content.
• Neural Network with GloVe Embeddings: This model, which incorporates LSTM layers and dropout layers, demonstrated a significant improvement in capturing complex emotional sentiments, validated through rigorous hyperparameter tuning and early stopping measures.

The study underscores NLP technologies' evolving complexity and capability in automated emotion detection from text data, highlighting the potential for more accurate and nuanced sentiment analysis in future research.