User Experience Interviews

User interviews were a primary research method used to understand the student experience with University of California, Berkeley’s emergency mass notification system Emergency Alert System, WarnMe. The stories and needs of users were collected to paint a picture of how WarnMe is perceived and applied to students’ sense of safety on-campus. The objectives of the interviews include:

1. How does the WarnMe Emergency Alert System at UC Berkeley impact student perceptions of safety, emotions, and behaviors?
2. How can the system enhance its effectiveness in addressing safety precautions and communication?

The research team recruited 10 participants. The participant group was composed of full-time students at the University of California, Berkeley during the Spring 2024 semester. Participants were recruited through posters, social media marketing through a student-led news publication ad, and student community groups. Participants completed an interest questionnaire, including sharing their school email to ensure student enrollment. The group only included students who were already subscribers of WarnMe (all UCB students are automatically enrolled through their university email account). Each participant was interviewed one-on-one with a research team member for 45 mins to one hour. The interview questions are listed and the interviews conducted were semi-structured.

After interviews were completed, the research team used grounded coding to review the interview data for themes and topics. Those themes and topics were then categorized into buckets that directly informed the key findings produced for this project.

Themes

1: Oz of WarnMe

Theme Description and Background:

How users conceptualize the administrators and authors of WarnMe messages influences how they engage with the system and their sentiments towards it. The motivations of this conceptualization and expectations of these internally-constructed figures has an impact on how student users evaluate the WarnMe system and the users feelings and trust towards the system.

Student users at the University of California, Berkeley are automatically enrolled in its campus emergency alert system, WarnMe. It’s a resource that allows the university to send urgent messages that impact campus safety. The system is administered as a federal requirement of the Clery Act by university staff. During interviews, users commented on their assumptions of who they imagine administers and/or composes WarnMe messages. Understanding this conceptualization or mental image of an unknown figure determining when the university community will, or will not, be notified may impact how students engage with the system and use it to make safety decisions.

Issues/User Needs:

Two key student user needs informed the development of this theme. Interviews with student users referenced not only the messages they received but voiced frustrations at who may be composing them or responsible for the alert system.

“A form of information warfare” One student described how the university engaged in “information warfare” by not reporting all events that impacted students and being selective in when, how, or what was sent. To users there was doubt if the university representative responsible for the message was concerned with student safety or using the system to pursue institutional objectives that deprioritized student safety. Users surfaced descriptions of a removed and institution-protecting administrator was mentioned during interviews. This disrupted a sense of trust or covenant that student, or community, safety was the highest priority in administering the WarnMe alert system. This largely negative orientation to administrators is an issue that needs to be addressed in how WarnMe is administered, if student users are going to trust and successfully apply urgent campus safety information.

Message motivation misalignments One user questioned the motivations of WarnMe administrators: “the police saw the video go viral and then spun up the WarnMe to get ahead of the backlash.” In this disclosure the student user is highlighting two things: the university is concerned about optics over student safety and that there are faster ways for students to understand incidents in context to evaluate university response for safety concerns. Students are then pushed to other channels of safety communication it creates distrust in the system and forces students to rely on more insecure channels. One user further describes their experience being in a building directly next to the shooting on February 9th: “When you get information from What’s app groups, it’s speculation at most. [To get from WarnMe], information as to what happened or if you’re in the area: “stay clear” [would help]. It didn’t need to have a lot of detail, not even [needing to mention] a shooting at that point. But at minimum: information to stay away from the area and in doors. It needs to happen at the moment the danger was present. Hearing two hours later, is too late. I don’t even need information at that point.”

Student Quotes:

• Referring to February 9th, 2024 Shooting: "I thought it was funny that the police saw the video go viral and then spun up the WarnMe to get ahead of the backlash. I’d like to not think that’s what happened, but it seemed like it was"
• "It was a WarnMe about somebody using a water gun on someone. It was weird for sure to read that. It was weird because, it was good that incidents are reported. But using a water gun? iIt’s a children’s toy, not too dangerous. A WarnMe about water guns seemed unnecessary."
• “We even get them for protest activity on campus, which I think is strange that we are warned by protest activity. It muddles the situation, and people stop reading the texts.”
• “In my head there’s some administrator (nameless administrator). I don’t have a sense of the order of operations of how WarnMe messages are sent. The police being involved is more clear, but with the protest there aren’t police involved and so who is sending the message is unclear to me."
• “There are different degrees of threats and someone walking around with a gun vs a protest an administrator thinks is too unruly are not the same”

Solutions and Recommendations:

• Allow for increased student input: develop ways of understanding the student user experience through community input and leadership.
  • Coordinate with student groups to understand how to further develop the WarnMe system and message content. This will allow the university to center student needs outside of exclusively prioritizing their interpretation of the Clery Act.
  • Set up regular cadences of collecting student feedback through surveys and interviews. Feedback will allow for continued iteration to develop an emergency alert system that student users are more likely to use and trust.
  • Develop a comment or community input link that is shared with each WarnMe message to allow for input on that message and its contents.
  • UCB-supported and student-led review of the WarnMe message composition guides, developed by the university, and response time analysis.

2: Bombardment of Inconsistent Language: The Effects of Constant Discrepancies that Impact Safety Decisions

Description and Background:

This theme found in the interviews encompasses the inconsistency seen by users of the WarnMe system that ultimately affect how they comprehend WarnMe messages and how it affects their general safety decisions. These inconsistencies are attributed to who writes the WarnMe causing issues with the formatting of the notifications, often providing rare, useless, or vague descriptions; and a lack of centralized information for future reference of an incident.

The intention of WarnMes should be to provide a consistent and reliable way for students to make accurate safety decisions. This is hindered by discrepancies of how WarnMe is presented to the students. This can be seen through the way WarnMes use both military time and regular time, differences in the amount of descriptions provided, and lack of updates from the campus/police on an incident that has occurred. This ultimately places users in a place where they have to decide if they can use the information provided to make accurate safety decisions while around the campus and off-campus and whether they turn to other news sources or first-hand accounts for information to aid their decisions.

Issues/User Needs:

Users of the WarnMe system stated that WarnMe often does nothing for them and that it is “nice to have and not need, but it’s pretty middling.” Many users often don’t include WarnMe in their safety decisions even if they check the WarnMe notifications they receive. One incident that occurred on campus, the “Banana incident,” was brought up by half the interviewees when describing a notable incident that occurred on campus, mainly due to the fact that they believed it wasn’t worth reporting on and gave “odd details.” It is incidents such as that one that make the more peculiar reports stand out and one interviewer described it as breaking free from the “desensitization” they feel from the constant influx of WarnMes, which is inherently grim. Lastly, users expressed issues with the way text was presented in the email titles, email descriptions, and text notifications that left them with an “unsettling feeling.”

Student Quotes:

• "It doesn’t impact [my overall experience] alot. When immediate safety and security was at risk, it was lacking. My general sense of safety is not supremely increased by the Warnme or safety measures."
• "The information they pass is highly perishable. If I had to go back and read a WarnMe, I wouldn’t know why."
• “You get random ass notifications, you would think they could give information that is actually helpful to students”
• "The police are transparent when it is easier to them but not for the sake of the students"
• "No need to bombard people in terms of font/language"
• "Language of WarnMe is written in a threatening way"

Solutions and Recommendations:

• Transparent Method and Centralized Information Location: Provide case numbers for incidents within WarnMe report and a website to track incidents that have occurred
  • By providing a case number, users can have an easy and efficient way to stay updated on a case since updates currently are not sent out through WarnMe.
  • A status bar could also be of use to track how far along a case is which could be included on a website that houses information about every WarnMe that has been sent out for users to be able to search for WarnMes for as far back as they would like, instead of just through their email of when they started school.
  • Example: “In shipping they have a status update bar, something similar to that would give me some piece of mind of where they are in the process (the assault, next steps, etc.)”
• Consolidated Method of Writing WarnMe Messages: More intentional and consistent about descriptions provided for an incident and a general, centralized way to write WarnMes
  • As there are inconsistencies with the way WarnMes are written, there should be a standard report for the WarnMe to follow to provide the max amount of description as possible, including specific language that is consistent across similar incidents as to not confuse users when trying to comprehend a WarnMe message.
  • Example: "The information they pass is highly perishable. If I had to go back and read a WarnMe, I wouldn’t know why."

3: The Illusion of Safety: Evaluating the Perceived Security and Actual Effectiveness of the WarnMe System

Description and Background:

In this theme we tried to investigate the gap between the perceived security provided by the WarnMe alert system and its actual effectiveness in real-world scenarios. We will explore how discrepancies in timeliness, relevance, and clarity of communications can create a false sense of security, leading to increased anxiety and confusion among the campus community. This gap between expected and actual performance of the alert system can lead to compromising the perceived safety and preparedness of the community.

WarnMe alerts are designed to inform the campus community about potential threats in real-time. However, the effectiveness of these alerts is often compromised by delays and the lack of actionable information. For instance, one student noted, "Delayed communication: When you get information from WhatsApp groups, it’s speculation at most... But at minimum: information to stay away from the area and indoors. It needs to happen at the moment the danger was present. Hearing two hours later, is too late."

Issues / User Needs:

Feedback from users highlights significant concerns about the system's current state. A student remarked, "The first one I got was confusing. It didn’t take long to figure out what it was about: some sort of emergency thing. It was confusing at first and then it was interesting." This indicates a need for clearer, more direct communication. Another user stated, "I feel that given how important locations are, it doesn’t do a good job of sharing where locations are (like a map)."

Student quotes, and excerpts:

• "I wish there was some sort of option to unsubscribe or 'don’t show me crimes like this again.' I want to know about shootings or armed standoffs." — UC Berkeley student.
• "It would be helpful to have a Google widget: a 3in x 3in map. That would make more sense to me in this context. One of the first pieces of information you get: the date and time is “2400 Block.” This would be confusing for anyone not from Berkeley or the US." — UC Berkeley student.
• "The gist I get from the WarnMes: they’re more tactical level reports, without a lot of context given." — UC Berkeley student

Solutions and Recommendations

1. Immediate and Accurate Information Delivery: Implementing a more robust real-time alert system that prioritizes the speed of notifications to ensure community members receive timely updates about potential threats.
   • Utilize a dedicated, real-time alert system with capabilities for rapid incident verification and develop protocols for rapid verification of incidents to reduce delays significantly.
   • Example: "Need to know about ongoing and dangerous events. Information Needed: how to make immediate safety decisions."
2. Enhanced User Customization and Relevance: Allowing users to customize the types of alerts they receive, focusing on those they count most relevant to their personal safety and reducing notification fatigue.
   • Develop a user interface that allows individuals to filter the types of alerts they receive, based on their personal safety preferences.
   • Example: "How can I manage the notification system? So I’m not bombarded with 'Berkeley is crime-ridden Gotham.'"
3. Integration with Localized Mapping and Detailed Descriptions: including interactive maps in alerts to provide clear, visual representations of affected areas, enhancing users' understanding of their proximity to potential threats.
   • Implementing interactive maps and improving the specificity of descriptions in alerts, particularly regarding the nature of the threat and recommended actions for safety.
   • Example: "Better geographic location sharing, like a static or interactive map that people can open and orient themselves about where it happened."
4. Complementary Information Channels: Establishing partnerships with local social media platforms and community forums to create supplemental channels of communication that can offer immediate updates and community-driven insights.
   • Partner with local social media and community forums for real-time, or promote the availability and advantages of direct SMS alerts alongside traditional email notifications to enhance the perceived urgency and visibility of important messages.
   • Example: "Mainly WhatsApp groups. It’s the best strategy. Social media applications."

Why the need to evaluate perceived vs. actual safety in Emergency Communication?

The theme and subsequent solutions address critical research questions about the effectiveness of emergency communication systems in promoting actual safety and security. By focusing on real-time updates, user customization, detailed mapping, and leveraging community platforms, the proposed solutions aim to transform WarnMe into a more reliable and trusted resource for campus safety, aligning perceived safety with actual safety outcomes.