Abstract
Besides the main content, webpages often contain other complementary and noisy data such as advertisements, navigational information, copyright notices, and other template-related elements. The detection and extraction of main content can have many applications, such as web summarization, indexing, data mining, content adaptation to mobile devices, web content printing, etc. We introduce a novel site-level technique for content extraction based on the DOM representation of webpages. This technique analyzes some selected pages in any given website to identify those nodes in the DOM tree that do not belong to the webpage template. Then, an algorithm explores these nodes in order to select the main content nodes. To properly evaluate the technique, we have built a suite of benchmarks by downloading several heterogeneous real websites and manually marking the main content nodes. This suite of benchmarks can be used to evaluate and compare different content extraction techniques.
This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Ciencia, Innovación y Universidades/AEI under grant TIN2016-76843-C4-1-R and by the Generalitat Valenciana under grant PROMETEO-II/2015/013 (SmartLogic). Salvador Tamarit was partially supported by the Conselleria de Educación, Investigación, Cultura y Deporte de la Generalitat Valenciana under the grant APOSTD/2016/036.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Bar-Youssef et al. [4] defined a pagelet as a self-contained logical region with a well defined topic of functionality. Accordingly, webpages are composed of pagelets.
References
Adam, G., Bouras, C., Poulopoulos, V.: CUTER: an efficient useful text extraction mechanism. In: 2009 International Conference on Advanced Information Networking and Applications Workshops, pp. 703–708, May 2009
Alarte, J., Insa, D., Silva, J., Tamarit, S.: Automatic detection of webpages that share the same web template. In: ter Beek, M.H., Ravara, A. (eds.) Proceedings of the 10th International Workshop on Automated Specification and Verification of Web Systems (WWV 2014). Electronic Proceedings in Theoretical Computer Science, vol. 163, pp. 2–15. Open Publishing Association, July 2014
Alarte, J., Insa, D., Silva, J., Tamarit, S.: Site-level web template extraction based on DOM analysis. In: Mazzara, M., Voronkov, A. (eds.) PSI 2015. LNCS, vol. 9609, pp. 36–49. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41579-6_4
Bar-Yossef, Z., Rajagopalan, S.: Template detection via data mining and its applications. In: Proceedings of the 11th International Conference on World Wide Web (WWW 2002), pp. 580–591. ACM, New York (2002)
Baroni, M., Chantree, F., Kilgarriff, A., Sharoff, S.: Cleaneval: a competition for cleaning web pages. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2008), pp. 638–643. European Language Resources Association, May 2008
Burget, R., Rudolfova, I.: Web page element classification based on visual features. In: Proceedings of the 1st Asian Conference on Intelligent Information and Database Systems (ACIIDS 2009), pp. 67–72. IEEE Computer Society, Washington, DC (2009)
Cardoso, E., Jabour, I., Laber, E., Rodrigues, R., Cardoso, P.: An efficient language-independent method to extract content from news webpages. In: Proceedings of the 11th ACM Symposium on Document Engineering (DocEng 2011), pp. 121–128. ACM, New York (2011)
Ferraresi, A., Zanchetta, E., Baroni, M., Bernardini, S.: Introducing and evaluating ukWaC, a very large web-derived corpus of English. In: Proceedings of the 4th Web as Corpus Workshop (WAC-4), pp. 47–54 (2008)
Gottron, T.: Content code blurring: a new approach to content extraction. In: Proceedings of the 2008 19th International Conference on Database and Expert Systems Application, DEXA 2008, pp. 29–33. IEEE Computer Society, Washington, DC, September 2008
Insa, D., Silva, J., Tamarit, S.: Using the words/leafs ratio in the DOM tree for content extraction. J. Log. Algebr. Program. 82(8), 311–325 (2013)
Kohlschütter, C.: A densitometric analysis of web template content. In: Quemada, J., León, G., Maarek, Y.S., Nejdl, W. (eds.) Proceedings of the 18th International Conference on World Wide Web (WWW 2009), pp. 1165–1166. ACM, April 2009
Kohlschütter, C., Fankhauser, P., Nejdl, W.: Boilerplate detection using shallow text features. In: Davison, B.D., Suel, T., Craswell, N., Liu, B. (eds.) Proceedings of the 3rd International Conference on Web Search and Web Data Mining (WSDM 2010), pp. 441–450. ACM, February 2010
Kohlschütter, C., Nejdl, W.: A densitometric approach to web page segmentation. In: Shanahan, J.G., et al. (eds.) Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM 2008), pp. 1173–1182. ACM, October 2008
Li, Z., Ng, W.K., Sun, A.: Web data extraction based on structural similarity. Knowl. Inf. Syst. 8(4), 438–461 (2005)
Pasternack, J., Roth, D.: Extracting article text from the web with maximum subsequence segmentation. In: Proceedings of the 18th International Conference on World Wide Web, WWW 2009, pp. 971–980. ACM, New York (2009)
Qureshi, P.A.R., Memon, N.: Hybrid model of content extraction. J. Comput. Syst. Sci. 78(4), 1248–1257 (2012)
Reis, D.d.C., Golgher, P.B., Silva, A.S., Laender, A.H.F.: Automatic web news extraction using tree edit distance. In: Proceedings of the 13th International Conference on World Wide Web (WWW 2004), pp. 502–511. ACM, New York (2004)
Vieira, K., da Costa Carvalho, A.L., Berlt, K., de Moura, E.S., da Silva, A.S., Freire, J.: On finding templates on web collections. World Wide Web 12(2), 171–211 (2009)
Vieira, K., da Silva, A.S., Pinto, N., de Moura, E.S., Cavalcanti, J.a.M.B., Freire, J.: A fast and robust method for web page template detection and removal. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM 2006), pp. 258–267. ACM, New York (2006)
Wang, Y., Fang, B., Cheng, X., Guo, L., Xu, H.: Incremental web page template detection. In: Proceedings of the 17th International Conference on World Wide Web (WWW 2008), pp. 1247–1248. ACM, New York (2008)
Weninger, T., Henry Hsu, W., Han, J.: CETR: Content Extraction via Tag Ratios. In: Rappa, M., Jones, P., Freire, J., Chakrabarti, S. (eds.) Proceedings of the 19th International Conference on World Wide Web (WWW 2010), pp. 971–980. ACM, April 2010
Wu, S., Liu, J., Fan, J.: Automatic web content extraction by combination of learning and grouping. In: Proceedings of the 24th International Conference on World Wide Web, WWW 2015, pp. 1264–1274. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2015)
Yi, L., Liu, B., Li, X.: Eliminating noisy information in web pages for data mining. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data mining (KDD 2003), pp. 296–305. ACM, New York (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Alarte, J., Insa, D., Silva, J., Tamarit, S. (2018). Main Content Extraction from Heterogeneous Webpages. In: Hacid, H., Cellary, W., Wang, H., Paik, HY., Zhou, R. (eds) Web Information Systems Engineering – WISE 2018. WISE 2018. Lecture Notes in Computer Science(), vol 11233. Springer, Cham. https://doi.org/10.1007/978-3-030-02922-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-02922-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-02921-0
Online ISBN: 978-3-030-02922-7
eBook Packages: Computer ScienceComputer Science (R0)