Skip to main content
SpringerLink
Log in

CDX Summary: Web Archival Collection Insights

  • Conference paper
  • First Online:
Linking Theory and Practice of Digital Libraries (TPDL 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13541))

Included in the following conference series:

Abstract

Large web archival collections are often opaque about their holdings. We created an open-source tool called, CDX Summary, to generate statistical reports based on URIs, hosts, TLDs, paths, query parameters, status codes, media types, date and time, etc. present in the CDX index of a collection of WARC files. Our tool also surfaces a configurable number of potentially good random memento samples from the collection for visual inspection, quality assurance, representative thumbnails generation, etc. The tool generates both human and machine readable reports with varying levels of details for different use cases. Furthermore, we implemented a Web Component that can render generated JSON summaries in HTML documents. Early exploration of CDX insights on Wayback Machine collections helped us improve our crawl operations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://archive.org/download/mediacloud.

  2. 2.

    https://www.webcomponents.org/.

References

  1. Alam, S.: Archive profiler: scripts to generate profiles of various web archives (2014). https://github.com/oduwsdl/archive_profiler

  2. Alam, S.: MemGator: A memento aggregator CLI and server in go (2015). https://github.com/oduwsdl/MemGator

  3. Alam, S.: Web ARChive (WARC) file format (2018). https://www.slideshare.net/ibnesayeed/web-archive-warc-file-format

  4. Alam, S.: MementoMap: a tool to summarize web archive holdings (2019). https://github.com/oduwsdl/MementoMap

  5. Alam, S.: CDX summary (2021). https://github.com/internetarchive/cdx-summary

  6. Alam, S.: CDX summary in PyPI (2022). https://pypi.org/project/cdxsummary/

  7. Alam, S.: CDX summary web component (2022). https://www.npmjs.com/package/@internetarchive/cdxsummary

  8. Alam, S., Nelson, M.L.: MemGator - a portable concurrent memento aggregator: cross-platform CLI and server binaries in go. In: Proceedings of the 16th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 243–244. JCDL 2016 (2016). https://doi.org/10.1145/2910896.2925452

  9. Alam, S., Nelson, M.L., Van de Sompel, H., Balakireva, L.L., Shankar, H., Rosenthal, D.S.H.: Web archive profiling through CDX summarization. In: Proceedings of the 19th International Conference on Theory and Practice of Digital Libraries, pp. 3–14. TPDL 2015 (2015). https://doi.org/10.1007/978-3-319-24592-8_1

  10. Alam, S., Nelson, M.L., Van de Sompel, H., Balakireva, L.L., Shankar, H., Rosenthal, D.S.H.: Web archive profiling through CDX summarization. Int. J. Digit. Libr. 17(3), 223–238 (2016). https://doi.org/10.1007/s00799-016-0184-4

    Article  Google Scholar 

  11. Alam, S., Nelson, M.L., Van de Sompel, H., Rosenthal, D.S.H.: Web archive profiling through fulltext search. In: Proceedings of the 20th International Conference on Theory and Practice of Digital Libraries, pp. 121–132. TPDL 2016 (2016). https://doi.org/10.1007/978-3-319-43997-6_10

  12. Alam, S., Weigle, M.C., Nelson, M.L.: Profiling web archival voids for memento routing. In: Proceedings of the 21st ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 150–159. JCDL 2021 (2021). https://doi.org/10.1109/JCDL52503.2021.00027

  13. Alam, S., Weigle, M.C., Nelson, M.L., Melo, F., Bicho, D., Gomes, D.: MementoMap framework for flexible and adaptive web archive profiling. In: Proceedings of the 19th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 172–181. JCDL 2019 (2019). https://doi.org/10.1109/JCDL.2019.00033

  14. AlSum, A., Weigle, M.C., Nelson, M.L., Van de Sompel, H.: Profiling web archive coverage for top-level domain and content language. In: Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries, pp. 60–71. TPDL 2013 (2013). https://doi.org/10.1007/978-3-642-40501-3_7

  15. AlSum, A., Weigle, M.C., Nelson, M.L., Van de Sompel, H.: Profiling web archive coverage for top-level domain and content language. Int. J. Digital Lib. 14(3–4), 149–166 (2014). https://doi.org/10.1007/s00799-014-0118-y

    Article  Google Scholar 

  16. Bornand, N., Balakireva, L., Van de Sompel, H.: Routing memento requests using binary classifiers. In: Proceedings of the 16th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 63–72. JCDL 2016 (2016). https://doi.org/10.1145/2910896.2910899

  17. Holzmann, H., Goel, V., Anand, A.: ArchiveSpark: efficient web archive access, extraction and derivation. In: Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, pp. 83–92. JCDL 2016 (2016). https://doi.org/10.1145/2910896.2910902

  18. Internet archive: CDX file format (2003). http://archive.org/web/researcher/cdx_file_format.php

  19. ISO 28500:2017: WARC file format (2017). https://iso.org/standard/68004.html

  20. Jackson, A.: Messy web archive collections (2014). https://twitter.com/anjacks0n/status/466690812269846528

  21. Klein, M., Balakireva, L., Shankar, H.: Evaluating memento service optimizations. In: Proceedings of the 19th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 182–185. JCDL 2019 (2019). https://doi.org/10.1109/JCDL.2019.00034

  22. Maurer, Y.: Summarize CDX(J) files for MIME analysis per 2nd-level domain (2021). https://github.com/ymaurer/cdx-summarize

  23. Meneses, L., Furuta, R., Shipman, F.: Identifying “Soft 404’’ error pages: analyzing the lexical signatures of documents in distributed collections. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) TPDL 2012. LNCS, vol. 7489, pp. 197–208. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33290-6_22

    Chapter  Google Scholar 

  24. Nelson, M.L., Van de Sompel, H.: Adding the Dimension of Time to HTTP. SAGE Handb. Web Hist. (2018)

    Google Scholar 

  25. Ruest, N., Lin, J., Milligan, I., Fritz, S.: The archives unleashed project: technology, process, and community to improve scholarly access to web archives. In: Proceedings of the 20th ACM/IEEE Joint Conference on Digital Libraries, pp. 157–166. JCDL 2020 (2020). https://doi.org/10.1145/3383583.3398513

  26. Sanderson, R., Van de Sompel, H., Nelson, M.L.: IIPC Memento Aggregator Experiment (2012). http://www.netpreserve.org/sites/default/files/resources/Sanderson.pdf

  27. Van de Sompel, H., Nelson, M.L., Sanderson, R.: HTTP framework for time-based access to resource states - memento. RFC 7089 (2013)

    Google Scholar 

  28. Van de Sompel, H., Nelson, M.L., Sanderson, R., Balakireva, L.L., Ainsworth, S., Shankar, H.: Memento: time travel for the web. Technical report arXiv:0911.1112 (2009). https://arxiv.org/abs/0911.1112

Download references

Acknowledgements

We thank various IA staff members for their help. Brewster Kahle for testing the CLI, Kenji Nagahashi for feedback on the web UI text, Brenton Cheng and Isa Herico Velasco for main site integration, Jason Buckner for Web Component help, and Jim Shelton for UX feedback.

Author information

Authors and Affiliations

  1. Wayback Machine, Internet Archive, San Francisco, CA, 94118, USA

    Sawood Alam & Mark Graham

Authors
  1. Sawood Alam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Graham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sawood Alam .

Editor information

Editors and Affiliations

  1. University of Padua, Padua, Italy

    Gianmaria Silvello

  2. Universidad Politécnica de Madrid, Madrid, Spain

    Oscar Corcho

  3. CNR-ISTI – National Research Council, Pisa, Italy

    Paolo Manghi

  4. University of Padua, Padua, Italy

    Giorgio Maria Di Nunzio

  5. Linnaeus University, Växjö, Sweden

    Koraljka Golub

  6. University of Padua, Padua, Italy

    Nicola Ferro

  7. Sapienza University of Rome, Rome, Italy

    Antonella Poggi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alam, S., Graham, M. (2022). CDX Summary: Web Archival Collection Insights. In: Silvello, G., et al. Linking Theory and Practice of Digital Libraries. TPDL 2022. Lecture Notes in Computer Science, vol 13541. Springer, Cham. https://doi.org/10.1007/978-3-031-16802-4_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16802-4_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16801-7

  • Online ISBN: 978-3-031-16802-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation