Abstract
Collections of Web-based resources are often decentralized; leaving the task of identifying and locating removed resources to collection managers who must rely on http response codes. When a resource is no longer available, the server is supposed to return a 404 error code. In practice and to be friendlier to human readers, many servers respond with a 200 OK code and indicate in the text of the response that the document is no longer available. In the reported study, 3.41% of servers respond in this manner. To help collection managers identify these “friendly” or “soft” 404s, we developed two methods that use a Naïve Bayes classifier based on known valid responses and known 404 responses. The classifier was able to predict soft 404 pages with a precision of 99% and a recall of 92%. We will also elaborate on the results obtained from our study and will detail the lessons learned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bush, V.: As we may think. The Atlantic (1945), http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/3881/
Logasa Bogen, P., Pogue, D., Poursardar, F., Li, Y., Furuta, R., Shipman, F.: WPv4: a re-imagined Walden’s paths to support diverse user communities. Presented at the Proceeding of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, Ottawa, Ontario, Canada (2011)
Shipman, F., Hsieh, H., Maloor, P., Moore, J.M.: The visual knowledge builder: a second generation spatial hypertext. Proceedings of the twelfth ACM conference on Hypertext and Hypermedia - HYPERTEXT 2001, Arhus, Denmark, pp. 113–122 (2001)
McCown, F., Marshall, C.C., Nelson, M.L.: Why web sites are lost (and how they’re sometimes found). Commun. ACM 52, 141–145 (2009)
Francisco-Revilla, L., Shipman, F., Furuta, R., Karadkar, U., Arora, A.: Managing change on the web. Presented at the Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries, Roanoke, Virginia, United States (2001)
Francisco-Revilla, L., Shipman, F., Furuta, R., Karadkar, U., Arora, A.: Perception of content, structure, and presentation changes in Web-based hypertext. Presented at the Proceedings of the 12th ACM Conference on Hypertext and Hypermedia, Arhus, Denmark (2001)
Logasa Bogen, P., Francisco-Revilla, L., Furuta, R., Hubbard, T., Karadkar, U.P., Shipman, F.: Longitudinal study of changes in blogs. Presented at the Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, Vancouver, BC, Canada (2007)
Klein, M., Shipman, J., Nelson, M.L.: Is this a good title? Presented at the Proceedings of the 21st ACM Conference on Hypertext and Hypermedia, Toronto, Ontario, Canada (2010)
Klein, M., Nelson, M.L.: Evaluating methods to rediscover missing web pages from the web infrastructure. Presented at the Proceedings of the 10th Annual Joint Conference on Digital Libraries, Gold Coast, Queensland, Australia (2010)
Harrison, T.L., Nelson, M.L.: Just-in-time recovery of missing web pages. Presented at the Proceedings of the Seventeenth Conference on Hypertext and Hypermedia, Odense, Denmark (2006)
Klein, M., Ware, J., Nelson, M.L.: Rediscovering missing web pages using link neighborhood lexical signatures. Presented at the Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, Ottawa, Ontario, Canada (2011)
Dalal, Z., Dash, S., Dave, P., Francisco-Revilla, L., Furuta, R., Karadkar, U., Shipman, F.: Managing distributed collections: evaluating web page changes, movement, and replacement. Presented at the Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries, Tuscon, AZ, USA (2004)
Johnson, D.B.: Enabling the reuse of World Wide Web documents in tutorials. University of Washington (1997)
Park, S.-T., Pennock, D.M., Giles, C.L., Krovetz, R.: Analysis of lexical signatures for finding lost or related documents. Presented at the Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland (2002)
Baeza-Yates, R., Pereira, I., Ziviani, N.: Genealogical trees on the web: a search engine user perspective. Presented at the Proceedings of the 17th International Conference on World Wide Web, Beijing, China (2008)
Ashman, H.: Electronic document addressing: dealing with change. ACM Comput. Surv. 32, 201–212 (2000)
Ashman, H., Davis, H., Whitehead, J., Caughey, S.: Missing the 404: link integrity on the World Wide Web. Presented at the Proceedings of the Seventh International Conference on World Wide Web 7, Brisbane, Australia (1998)
Davis, H.C.: Hypertext link integrity. ACM Comput. Surv. 31, 28 (1999)
Davis, H.C.: Referential integrity of links in open hypermedia systems. Presented at the Proceedings of the Ninth ACM Conference on Hypertext and Hypermedia: Links, Objects, Pittsburgh, Pennsylvania, United States (1998)
Kahle, B.: Preserving the Internet. Scientific American 276, 82–83 (1997)
Koehler, W.: Web page change and persistence—a four-year longitudinal study. J. Am. Soc. Inf. Sci. Technol. 53, 162–171 (2002)
Spinellis, D.: The decay and failures of web references. Commun. ACM 46, 71–77 (2003)
Phelps, T.A., Wilensky, R.: Robust Hyperlinks Cost Just Five Words Each. University of California at Berkeley (2000)
Park, S.-T., Pennock, D.M., Giles, C.L., Krovetz, R.: Analysis of lexical signatures for improving information persistence on the World Wide Web. ACM Trans. Inf. Syst. 22, 540–572 (2004)
Bar-Yossef, Z., Broder, A.Z., Kumar, R., Tomkins, A.: Sic transit gloria telae: towards an understanding of the web’s decay. Presented at the Proceedings of the 13th International Conference on World Wide Web, New York, NY, USA (2004)
Jatowt, A.: Web page summarization using dynamic content. Presented at the Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers and Posters, New York, NY, USA (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Meneses, L., Furuta, R., Shipman, F. (2012). Identifying “Soft 404” Error Pages: Analyzing the Lexical Signatures of Documents in Distributed Collections. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds) Theory and Practice of Digital Libraries. TPDL 2012. Lecture Notes in Computer Science, vol 7489. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33290-6_22
Download citation
DOI: https://doi.org/10.1007/978-3-642-33290-6_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33289-0
Online ISBN: 978-3-642-33290-6
eBook Packages: Computer ScienceComputer Science (R0)