Abstract
Rich Internet Applications are widely used as they are interactive and user friendly. Automated tools for crawling Rich Internet Applications have become needed for many reasons such as content indexing or testing for correctness and security. Due to the large size of RIAs, distributed crawling has been introduced to reduce the amount of time required for crawling. However, having one controller may result in a performance bottleneck resulting from a single database simultaneously accessed by many crawlers. It may also be vulnerable to complete data loss if a node failure occurs at the storage unit. We present a distributed decentralized scheme for crawling large-scale RIAs capable of partitioning the search space among several controllers in which the information is partially stored, which allows for fault tolerance and for the scalability of the system. Our results are significantly better than for non-distributed crawling, and outperforms the distributed crawling using one coordinator.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Mirtaheri, S.M., von Bochmann, G., Jourdan, G.V., Onut, I.V.: GDist-RIA Crawler: a greedy distributed crawler for Rich Internet Applications. In: Noubir, G., Raynal, M. (eds.) NETYS 2014, LNCS 8593, pp. 200–214. Springer, Heidelberg (2014)
Zhang, X., Wang, H.: AJAX crawling scheme based on Document Object Model. In: 4th International Conference on Computational and Information Sciences (ICCIS), China, Chongqing, pp. 1198–1201 (2012)
Choudhary, S., Dincturk, M.E., Mirtaheri, S.M., Moosavi, A., von Bochmann, G., Jourdan, G.V., Onut, I.V.: Crawling rich Internet applications: the state of the art. In: Conference of the Center for Advanced Studies on Collaborative Research, Toronto, Markham, pp. 146–160 (2012)
Benjamin, K., von Bochmann, G., Dincturk, M.E., Jourdan, G.V., Onut, V.: Some modeling challenges when testing rich internet applications for security. In: 1st International Workshop on Modeling and Detection of Vulnerabilities, France, Paris (2010)
Xiao, X., Zhang, W.Z., Zhang, H.L., Fang, B.X.: A forwarding-based task scheduling algorithm for distributed web crawling over DHTs. In: Proceedings of the 15th International Conference on Parallel and Distributed Systems (ICPADS), China, Shenzhen, pp. 854–859 (2009)
Paulson, L.D.: Building rich web applications with Ajax. Computer 38, 14–17 (2005). (IEEE Computer Society)
Cho, J., Garcia-Molina, H.: Parallel crawlers. In: Proceedings of the 11th International Conference on World Wide Web, WWW, Hawaii, Honolulu, vol. 2 (2002)
Schollmeier, R.: A definition of peer-to-peer networking for the classification of peer-to-peer architectures and applications. In: Proceedings of the IEEE 2001 International Conference on Peer-to-Peer Computing (P2P2001), Sweden, Linkping (2001)
Stoica, I., Morris, R., Karger, D., Frans Kaashoek, M., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for Internet applications. In: Proceedings of ACM SIGCOMM 2001, California, San Deigo (2001)
Brin, S., Page, L.: The anatomy of a large-scale hypertextual Web search engine. In: Proceedings of the 7th International Conference on World Wide Web, WWW, Australia, Brisbane, vol. 7 (1998)
Misra, J.: Detecting termination of distributed computations using markers. In: PODC’83, Proceedings of the 2nd Annual ACM Symposium on Principles of Distributed Computing, vol. 22, pp. 290–294. ACM, New York (1983)
Marini, J.: Document Object Model: Processing Structured Documents. McGraw-Hill/Osborne, New York (2002)
Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1, 269–271 (1959)
Loo, B.T., Owen, L., Krishnamurthy, C.S.: Distributed web crawling over DHTs. Technical report (2004)
Carzanig, A., Rutheford, M.: SSim: a simple Discrete-event Simulation Library, University of Colorado (2003). http://www.inf.usi.ch/carzaniga/ssim/index.html
Acknowledgments
This work is supported in part by IBM and the Natural Science and Engineering Research Council of Canada.
Disclaimer. The views expressed in this article are the sole responsibility of the authors and do not necessarily reflect those of IBM.
Trademarks. IBM, Rational and AppScan are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Ben Hafaiedh, K., von Bochmann, G., Jourdan, GV., Onut, I.V. (2014). A Scalable P2P RIA Crawling System with Partial Knowledge. In: Noubir, G., Raynal, M. (eds) Networked Systems. NETYS 2014. Lecture Notes in Computer Science(), vol 8593. Springer, Cham. https://doi.org/10.1007/978-3-319-09581-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-09581-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09580-6
Online ISBN: 978-3-319-09581-3
eBook Packages: Computer ScienceComputer Science (R0)