A Human in Every APE: Delineating and Evaluating the Human Analysis Systems of Anti-Phishing Entities

Abstract

We conducted a large-scale evaluation of some popular Anti-Phishing Entities (APEs). As part of this, we submitted arrays of CAPTCHA challenge-laden honey sites to 7 APEs. An analysis of the “click-through rates” during the visits from the APEs showed strong evidence for the presence of formidable human analysis systems in conjunction with automated crawler systems. In summary, we estimate that as many as 10% to 24% of URLs submitted to each of 4 APEs (Google Safe Browsing, Microsoft SmartScreen, Bitdefender and Netcraft) were likely visited by human analysts. In contrast to prior works, these measurements present a very optimistic picture for web security as, for the first time, they show presence of expansive human analysis systems to tackle suspicious URLs that might otherwise be challenging for automated crawlers to analyze.

This finding allowed us an opportunity to conduct the first systematic study of the robustness of the human analysis systems of APEs which revealed some glaring weaknesses in them. We saw that all the APEs we studied fall prey to issues such as lack of geolocation and client device diversity exposing their human systems to targeted evasive attacks. Apart from this, we also found a specific weakness across the entire APE ecosystem that enables creation of long-lasting phishing pages targeted exclusively against Android/Chrome devices by capitalizing on discrepancies in web sensor API outputs. We demonstrate this with the help of 10 artificial phishing sites that survived indefinitely despite repeated reporting to all APEs. We suggest mitigations for all these issues. We also conduct an elaborate disclosure process with all affected APEs in an attempt to persuade them to pursue these mitigations.