skip to main content
10.1145/3290605.3300748acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Put Your Warning Where Your Link Is: Improving and Evaluating Email Phishing Warnings

Published: 02 May 2019 Publication History

Abstract

Phishing emails often disguise a link's actual URL. Thus, common anti-phishing advice is to check a link's URL before clicking, but email clients do not support this well. Automated phishing detection enables email clients to warn users that an email is suspicious, but current warnings are often not specific. We evaluated the effects on phishing susceptibility of (1) moving phishing warnings close to the suspicious link in the email, (2) displaying the warning on hover interactions with the link, and (3) forcing attention to the warning by deactivating the original link, forcing users to click the URL in the warning. We assessed the effectiveness of such link-focused phishing warning designs in a between-subjects online experiment (n=701). We found that link-focused phishing warnings reduced phishing click-through rates compared to email banner warnings; forced attention warnings were most effective. We discuss the implications of our findings for phishing warning design.

Supplementary Material

ZIP File (pn8432.zip)
We have included a pdf of our survey script.

References

[1]
Alessandro Acquisti, Idris Adjerid, Rebecca Balebako, Laura Brandimarte, Lorrie Faith Cranor, Saranga Komanduri, Pedro Giovanni Leon, Norman Sadeh, Florian Schaub, Manya Sleeper, et al. 2017. Nudges for privacy and security: understanding and assisting users' choices online. ACM Computing Surveys (CSUR) 50, 3 (2017), 44.
[2]
Devdatta Akhawe and Adrienne Porter Felt. 2013. Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness. In USENIX security symposium, Vol. 13.
[3]
Hazim Almuhimedi, Florian Schaub, Norman Sadeh, Idris Adjerid, Alessandro Acquisti, Joshua Gluck, Lorrie Faith Cranor, and Yuvraj Agarwal. 2015. Your location has been shared 5,398 times!: A field study on mobile app privacy nudging. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, 787--796.
[4]
Bonnie Brinton Anderson, C Brock Kirwan, Jeffrey L Jenkins, David Eargle, Seth Howard, and Anthony Vance. 2015. How polymorphic warnings reduce habituation in the brain: Insights from an fMRI study. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2883--2892.
[5]
Anti Phishing Working Group (APWG). 2016. Global Phishing Report 2015--2016. http://docs.apwg.org/reports/APWG_Global_Phishing_ Report_2015--2016.pdf
[6]
Anti Phishing Working Group (APWG). 2018. Phishing Activity Trends Report Q1 2018. Technical Report.
[7]
Lujo Bauer, Cristian Bravo-Lillo, Lorrie Cranor, and Elli Fragkaki. 2013. Carnegie Mellon University Warning Design Guidelines. Technical Report Carnegie Mellon University-CyLab-13002. Carnegie Mellon University.
[8]
André Bergholz, Jan De Beer, Sebastian Glahn, Marie-Francine Moens, Gerhard Paaß, and Siehyun Strobel. 2010. New filtering approaches for phishing email. Journal of computer security 18, 1 (2010), 7--35.
[9]
Cristian Bravo-Lillo. 2014. Improving Computer Security Dialogs: An Exploration of Attention and Habituation.
[10]
Cristian Bravo-Lillo, Lorrie Cranor, Saranga Komanduri, Stuart Schechter, and Manya Sleeper. 2014. Harder to Ignore? Revisiting Pop-Up Fatigue and Approaches to Prevent It. In 10th Symposium On Usable Privacy and Security (SOUPS 2014). USENIX Association, Menlo Park, CA, 105--111. https://www.usenix.org/conference/soups2014/ proceedings/presentation/bravo-lillo
[11]
Cristian Bravo-Lillo, Lorrie Faith Cranor, Julie Downs, and Saranga Komanduri. 2011. Bridging the Gap in Computer Security Warnings: A Mental Model Approach. IEEE Security & Privacy 9, 2 (2011), 18--26.
[12]
Cristian Bravo-Lillo, Saranga Komanduri, Lorrie Faith Cranor, Robert W Reeder, Manya Sleeper, Julie Downs, and Stuart Schechter. 2013. Your attention please: designing security-decision UIs to make genuine risks harder to ignore. In Proceedings of the Ninth Symposium on Usable Privacy and Security. ACM, 6.
[13]
José Carlos Brustoloni and Ricardo Villamarín-Salomón. 2007. Improving security decisions with polymorphic and audited dialogs. In Proceedings of the 3rd symposium on Usable privacy and security. ACM, 76--85.
[14]
Madhusudhanan Chandrasekaran, Krishnan Narayanan, and Shambhu Upadhyaya. 2006. Phishing E-Mail Detection Based on Structural Properties. (2006), 7.
[15]
Lorrie Faith Cranor. 2012. Necessary but not sufficient: Standardized mechanisms for privacy notice and choice. J. on Telecomm. & High Tech. L. 10 (2012), 273.
[16]
Rachna Dhamija, J Doug Tygar, and Marti Hearst. 2006. Why phishing works. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, 581--590.
[17]
Julie S Downs, Mandy Holbrook, and Lorrie Faith Cranor. 2007. Behavioral response to phishing risk. In Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit. ACM, 37--44.
[18]
Serge Egelman, Lorrie Faith Cranor, and Jason Hong. 2008. You've been warned: an empirical study of the effectiveness of web browser phishing warnings. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1065--1074.
[19]
Adrienne Porter Felt, Alex Ainslie, Robert W Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, and Jeff Grimes. 2015. Improving SSL warnings: Comprehension and adherence. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2893--2902.
[20]
Adrienne Porter Felt, Robert W Reeder, Hazim Almuhimedi, and Sunny Consolvo. 2014. Experimenting at scale with google chrome's SSL warning. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2667--2670.
[21]
Ian Fette, Norman Sadeh, and Anthony Tomasic. 2007. Learning to detect phishing emails. In Proceedings of the 16th international conference on World Wide Web. ACM, 649--656.
[22]
Kristin Firth, David A. Hoffman, and Tess Wilkinson-Ryan. 2017. Law and Psychology Grows Up, Goes Online, and Replicates. (2017).
[23]
Gmail Help Forum. 2018. Suspicious link issue! - Google Product Forums. https://productforums.google.com/forum/#!msg/gmail/h_ yeYefHFWk/Fj8o4q3HAQAJ
[24]
Simson Garfinkel and Heather Richter Lipford. 2014. Usable security: History, themes, and challenges. Synthesis Lectures on Information Security, Privacy, and Trust 5, 2 (2014), 1--124.
[25]
Anti Phishing Working Group. 2017. Global Phishing Survey: Trends and Domain Name Use in 2016. https://docs.apwg.org/reports/APWG_ Global_Phishing_Report_2015--2016.pdf
[26]
Marian Harbach, Markus Hettig, Susanne Weber, and Matthew Smith. 2014. Using Personal Examples to Improve Risk Communication for Security & Privacy Decisions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2647--2656.
[27]
Jason Hong. 2012. The state of phishing attacks. Commun. ACM 55, 1 (2012), 74--81.
[28]
The Radicati Group Inc. 2017. Email Statistics Report, 20172021. http://www.radicati.com/wp/wp-content/uploads/2017/01/ Email-Statistics-Report-2017--2021-Executive-Summary.pdf
[29]
Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. 2009. A nutrition label for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security. ACM, 4.
[30]
Soyun Kim and Michael S Wogalter. 2009. Habituation, dishabituation, and recovery effects in visual warnings. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 53. Sage Publications Sage CA: Los Angeles, CA, 1612--1616.
[31]
Ponnurangam Kumaraguru, Justin Cranshaw, Alessandro Acquisti, Lorrie Cranor, Jason Hong, Mary Ann Blair, and Theodore Pham. 2009. School of phish: a real-world evaluation of anti-phishing training. In Proceedings of the 5th Symposium on Usable Privacy and Security. ACM, 3.
[32]
Ponnurangam Kumaraguru, Steve Sheng, Alessandro Acquisti, Lorrie Faith Cranor, and Jason Hong. 2010. Teaching Johnny not to fall for phish. ACM Transactions on Internet Technology (TOIT) 10, 2 (2010), 7.
[33]
Phish Labs. 2017. 2017 Phishing and Threat Intelligence Report. https: //pages.phishlabs.com/rs/130-BFB-942/images/2017%20PhishLabs% 20Phishing%20and%20Threat%20Intelligence%20Report.pdf
[34]
Elmer Lastdrager, Inés Carvajal Gallardo, Pieter Hartel, and Marianne Junger. 2017. How Effective is Anti-Phishing Training for Children?. In Symposium on Usable Privacy and Security (SOUPS).
[35]
Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.
[36]
Eric Lin, Saul Greenberg, Eileah Trotter, David Ma, and John Aycock. 2011. Does domain highlighting help people identify phishing sites?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2075--2084.
[37]
Gang Liu, Guang Xiang, Bryan A Pendleton, Jason I Hong, and Wenyin Liu. 2011. Smartening the crowds: computational techniques for improving human verification to fight phishing scams. In Proceedings of the Seventh Symposium on Usable Privacy and Security. ACM, 8.
[38]
Samuel Marchal, Giovanni Armano, Tommi Grondahl, Kalle Saari, Nidhi Singh, and N Asokan. 2017. Off-the-Hook: An Efficient and Usable Client-Side Phishing Prevention Application. IEEE Trans. Comput. (2017).
[39]
Samuel Marchal, Kalle Saari, Nidhi Singh, and N. Asokan. 2015. Know Your Phish: Novel Techniques for Detecting Phishing Sites and their Targets. CoRR abs/1510.06501 (2015). arXiv:1510.06501 http://arxiv. org.offcampus.lib.washington.edu/abs/1510.06501
[40]
Federal Bureau of Investigation Internet Crime Complaint Center. 2017. 2017 Internet Crime Report. (2017), 29.
[41]
Kenneth Olmstead and Aaron Smith. 2017. Pew Research Center. http://www.pewinternet.org/2017/03/22/ what-the-public-knows-about-cybersecurity/
[42]
Kenneth Olmstead and Aaron Smith. 2017. Pew Research Center. http: //www.pewinternet.org/2017/01/26/americans-and-cybersecurity/
[43]
Sameer Patil, Roberto Hoyle, Roman Schlegel, Apu Kapadia, and Adam J. Lee. 2015. Interrupt Now or Inform Later?: Comparing Immediate and Delayed Privacy Feedback. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1415--1418.
[44]
Sameer Patil, Roman Schlegel, Apu Kapadia, and Adam J. Lee. 2014. Reflection or Action?: How Feedback and Control Affect Location Sharing Decisions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 101--110.
[45]
PhishTank. 2018. Join the fight against phishing. Technical Report. https://www.phishtank.com/
[46]
Florian Schaub, Rebecca Balebako, Adam L Durity, and Lorrie Faith Cranor. 2015. A design space for effective privacy notices. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015). 1--17.
[47]
Stuart E. Schechter, Rachna Dhamija, Andy Ozment, and Ian Fischer. {n. d.}. The Emperor's New Security Indicators. In Security and Privacy, 2007. SP'07. IEEE Symposium On (2007). IEEE, 51--65. http://ieeexplore. ieee.org/abstract/document/4223213/
[48]
UC Berkeley Information Security and Policy. 2018. Phishing | Information Security and Policy. https://security.berkeley.edu/resources/ phishing
[49]
Steve Sheng, Bryant Magnien, Ponnurangam Kumaraguru, Alessandro Acquisti, Lorrie Faith Cranor, Jason Hong, and Elizabeth Nunge. 2007. Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish. In Proceedings of the 3rd Symposium on Usable Privacy and Security - SOUPS '07. ACM Press, Pittsburgh, Pennsylvania, 88.
[50]
Joshua Sunshine, Serge Egelman, Hazim Almuhimedi, Neha Atri, and Lorrie Faith Cranor. 2009. Crying Wolf: An Empirical Study of SSL Warning Effectiveness. In USENIX Security Symposium. 399--416.
[51]
Symantec. 2018. Internet Security Threat Report.
[52]
Janice Y Tsai, Serge Egelman, Lorrie Cranor, and Alessandro Acquisti. 2011. The effect of online privacy information on purchasing behavior: An experimental study. Information Systems Research 22, 2 (2011), 254--268.
[53]
Verizon. 2018. 2018 Data Breach Investigations Report. Technical Report 11th Edition.
[54]
Melanie Volkamer, Karen Renaud, and Paul Gerber. 2016. Spot the Phish by Checking the Pruned URL. Information and Computer Security 24, 4 (Oct. 2016), 372--385.
[55]
Melanie Volkamer, Karen Renaud, Benjamin Reinheimer, and Alexandra Kunz. 2017. User experiences of TORPEDO: TOoltip-poweRed Phishing Email DetectiOn. Computers & Security 71 (2017), 100 -- 113.
[56]
Jaclyn Wainer, Laura Dabbish, and Robert Kraut. 2011. Should I open this email?: inbox-level cues, curiosity and attention to email. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 3439--3448.
[57]
Rick Wash and Molly M. Cooper. 2018. Who Provides Phishing Training?: Facts, Stories, and People Like Me. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, Montreal QC, Canada, 1--12.
[58]
Liu Wenyin, Ning Fang, Xiaojun Quan, Bite Qiu, and Gang Liu. 2010. Discovering Phishing Target Based on Semantic Link Network. Future Generation Computer Systems 26, 3 (March 2010), 381--388.
[59]
Colin Whittaker, Brian Ryner, and Marria Nazif. 2007. Large-Scale Automatic Classification of Phishing Pages. (2007), 14.
[60]
Michael S Wogalter, Vincent C Conzola, and Tonya L Smith-Jackson. 2002. based guidelines for warning design and evaluation. Applied ergonomics 33, 3 (2002), 219--230.
[61]
IBM X-Force. 2018. IBM X-Force Threat Intelligence Index 2018. https: //www.ibm.com/security/data-breach/threat-intelligence
[62]
Guang Xiang, Jason Hong, Carolyn P Rose, and Lorrie Cranor. 2011. Cantina+: A feature-rich machine learning framework for detecting phishing web sites. ACM Transactions on Information and System Security (TISSEC) 14, 2 (2011), 21.
[63]
Weining Yang, Jing Chen, Aiping Xiong, Robert W. Proctor, and Ninghui Li. {n. d.}. Effectiveness of a Phishing Warning in Field Settings. ACM Press, 1--2.
[64]
Yue Zhang, Jason I. Hong, and Lorrie F. Cranor. 2007. Cantina: A Content-Based Approach to Detecting Phishing Web Sites. In Proceedings of the 16th International Conference on World Wide Web - WWW '07. ACM Press, Banff, Alberta, Canada, 639.

Cited By

View all
  • (2025)Phishing scams on social media: An evaluation of cyber awareness education on impact and effectivenessJournal of Economic Criminology10.1016/j.jeconc.2025.1001257(100125)Online publication date: Mar-2025
  • (2024)FakeBehalfProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698970(1243-1260)Online publication date: 14-Aug-2024
  • (2024)What drives SMiShing susceptibility? a U.S. interview study of how and why mobile phone users judge text messages to be real or fakeProceedings of the Twentieth USENIX Conference on Usable Privacy and Security10.5555/3696899.3696920(393-411)Online publication date: 12-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
May 2019
9077 pages
ISBN:9781450359702
DOI:10.1145/3290605
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Check for updates

Badges

  • Honorable Mention

Author Tags

  1. phishing
  2. privacy
  3. security
  4. usability
  5. warning design

Qualifiers

  • Research-article

Conference

CHI '19
Sponsor:

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)288
  • Downloads (Last 6 weeks)24
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Phishing scams on social media: An evaluation of cyber awareness education on impact and effectivenessJournal of Economic Criminology10.1016/j.jeconc.2025.1001257(100125)Online publication date: Mar-2025
  • (2024)FakeBehalfProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698970(1243-1260)Online publication date: 14-Aug-2024
  • (2024)What drives SMiShing susceptibility? a U.S. interview study of how and why mobile phone users judge text messages to be real or fakeProceedings of the Twentieth USENIX Conference on Usable Privacy and Security10.5555/3696899.3696920(393-411)Online publication date: 12-Aug-2024
  • (2024)Examining the Effectiveness of Speech and Earcon Alerts for Aiding Phishing Email DetectionProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/1071181324127551868:1(209-214)Online publication date: 26-Aug-2024
  • (2024)Eyes on the Phish(er): Towards Understanding Users' Email Processing Pattern and Mental Models in Phishing DetectionProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688465(15-29)Online publication date: 30-Sep-2024
  • (2024)From Victims to Defenders: An Exploration of the Phishing Attack Reporting EcosystemProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678926(49-64)Online publication date: 30-Sep-2024
  • (2024)Utilizing Large Language Models with Human Feedback Integration for Generating Dedicated Warning for Phishing EmailsProceedings of the 2nd ACM Workshop on Secure and Trustworthy Deep Learning Systems10.1145/3665451.3665531(35-46)Online publication date: 2-Jul-2024
  • (2024)Better Together: The Interplay Between a Phishing Awareness Video and a Link-centric Phishing Support ToolProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642843(1-60)Online publication date: 11-May-2024
  • (2024)Understanding Users' Interaction with Login NotificationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642823(1-17)Online publication date: 11-May-2024
  • (2024)PellucidAttachment: Protecting Users From Attacks via E-Mail AttachmentsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.327903221:3(1342-1354)Online publication date: May-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media