research-article

XIEv: dynamic analysis for crawling and modeling of web applications

Authors:

Manuel Leithner,

Dimitris E. SimosAuthors Info & Claims

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

Pages 2201 - 2210

https://doi.org/10.1145/3341105.3373885

Published: 30 March 2020 Publication History

Abstract

Researchers and practitioners in the fields of testing, security assessment and web development seeking to evaluate a given web application often have to rely on the existence of a model of the respective system, which is then used as input to task-specific tools. Such models may include information on HTTP endpoints and their parameters, available user actions/event listeners and required assets. Unfortunately, this data is often unavailable in practice, as only rigorous development practices or manual analysis guarantee their existence and correctness. Crawlers based on static analysis have traditionally been used to extract required information from existing sites. Regrettably, these tools can not accurately account for the dynamic behavior introduced by JavaScript and other technologies that are prevalent on modern sites. While methods based on dynamic analysis exist, they are not fully capable of identifying event listeners and their effects. This work presents XIEv, an approach for dynamic analysis of web applications that produces an execution trace usable for the extraction of navigation graphs, identification of bugs at runtime and enumeration of resources requested by each page. It offers improved recognition and selection of event listeners as well as a greater range of observed effects compared to existing approaches.

References

[1]

W3C. 2006. Document Object Model Events. Retrieved September 20, 2018 from https://www.w3.org/TR/2006/WD-DOM-Level-3-Events-20060413/events.html.

[2]

W3C. 2016. UI Events. Retrieved September 24, 2018 from https://www.w3.org/TR/uievents/.

[3]

Ali Mesbah, Arie van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes. ACM Trans. Web 6, 1, Article 3 (March 2012), 30 pages.

Digital Library

[4]

Ali Mesbah and Arie van Deursen. 2009. Invariant-Based Automatic Testing of Ajax User Interfaces. In Proceedings of the 31st International Conference on Software Engineering (ICSE'09). IEEE Computer Society, 210--220.

Digital Library

[5]

Ali Mesbah and Mukul R. Prasad. 2011. Automated cross-browser compatibility testing. In Proceedings of the 33rd International Conference on Software Engineering (ICSE '11). ACM, New York, NY, USA, 561--570.

Digital Library

[6]

Shabnam Mirshokraie and Ali Mesbah. 2012. JSART: JavaScript Assertion-based Regression Testing. In Proceedings of the 12th International Conference on Web Engineering (ICWE'12). Springer Berlin Heidelberg, 238--252.

Digital Library

[7]

Software Freedom Conservancy. 2019. Selenium - Web Browser Automation. Retrieved January 10, 2019 from https://www.seleniumhq.org/.

[8]

The Chromium Authors. 2019. Chrome DevTools Protocol Viewer. Retrieved March 20, 2019 from https://chromedevtools.github.io/devtools-protocol/.

[9]

W3C. 2005. W3C Document Object Model. Retrieved January 22, 2019 from https://www.w3.org/DOM/.

[10]

Giancarlo Pellegrino, Constantin Tschürtz, Eric Bodden and Christian Rossow. 2015. jÄk: Using Dynamic Analysis to Crawl and Test Modern Web Applications. In Proceedings of Research in Attacks, Intrusions and Defenses (RAID) Symposium (RAID 2015). Springer International Publishing, 295--316.

Digital Library

[11]

Steven Van Acker, Daniel Hausknecht, and Andrei Sabelfeld. 2017. Measuring login webpage security. In Proceedings of the Symposium on Applied Computing (SAC '17). ACM, New York, NY, USA, 1753--1760.

Digital Library

[12]

Milivoj Simeonovski, Giancarlo Pellegrino, Christian Rossow, and Michael Backes. 2017. Who Controls the Internet?: Analyzing Global Threats using Property Graph Traversals. In Proceedings of the 26th International Conference on World Wide Web (WWW '17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 647--656.

Digital Library

[13]

Constantin Tschürtz. 2015. jAEk. Retrieved April 20, 2019 from https://github.com/ConstantinT/jAEk.

[14]

Manuel Leithner and Dimitris E. Simos. 2018. DOMdiff: Identification and Classification of Inter-DOM Modifications. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI). IEEE Computer Society, 262--269.

[15]

The Chromium Authors. 2017. Chromium policy on JavaScript dialogs. Retrieved May 6, 2019 from https://developers.google.com/web/updates/2017/03/dialogs-policy.

[16]

w3af.org. 2013. w3af. Retrieved January 25, 2019 from http://w3af.org/.

[17]

Google Inc, Michal Zalewski, Niels Heinen, Sebastian Roschke. 2012. Skipfish. Retrieved January 25, 2019 from https://code.google.com/archive/p/skipfish/.

[18]

Free Software Foundation, Inc. 2017. Wget. Retrieved January 25, 2019 from https://www.gnu.org/software/wget/.

[19]

WordPress Foundation. 2003. WordPress. Retrieved February 13, 2019 from https://wordpress.org/.

[20]

Open Source Matters, Inc. 2005. Joomla Content Management System (CMS). Retrieved February 13, 2019 from https://www.joomla.org/.

[21]

MODX LLC. 2004. MODX. Retrieved February 13, 2019 from https://modx.com/.

[22]

Wikimedia Foundation. 2002. MediaWiki. Retrieved February 13, 2019 from https://www.mediawiki.org/wiki/MediaWiki.

[23]

phpBB Limited. 2000. phpBB. Retrieved February 13, 2019 from https://www.phpbb.com/.

[24]

Chris Boulton. 2002. MyBB. Retrieved February 13, 2019 from https://mybb.com/.

[25]

Bedirhan Urgun. 2014. Web Input Vector Extractor Teaser. Retrieved February 27, 2019 from https://github.com/bedirhan/wivet.

[26]

Sanjay K. Malik and Syed A. M. Rizvi. 2011. Information extraction using web usage mining, web scrapping and semantic annotation. In 2011 International Conference on Computational Intelligence and Communication Networks. IEEE Computer Society, 465--469.

Digital Library

[27]

Suhit Gupta, Gail Kaiser, David Neistadt, and Peter Grimm. 2003. DOM-based content extraction of HTML documents. In Proceedings of the 12th international conference on World Wide Web (WWW '03). ACM, New York, NY, USA, 207--214.

Digital Library

[28]

Josip Bozic, Bernhard Garn, Dimitris E. Simos and Franz Wotawa. 2015. Evaluation of the IPO-family algorithms for test case generation in web security testing. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE Computer Society, 1--10.

[29]

Allison Woodruff and Paul M. Aoki and Eric Brewer and Paul Gauthier and Lawrence A. Rowe. 1996. An investigation of documents from the World Wide Web. Computer Networks and ISDN Systems, 28, 7--11 (1996), 963--980.

Digital Library

Cited By

Rosa Martins FSeixas Pereira LDuarte C(2024)QualState: Finding Website States for Accessibility EvaluationProceedings of the 21st International Web for All Conference10.1145/3677846.3677851(96-105)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677851
Sigalov DGamayunov D(2024)Dead or aliveJournal of Information Security and Applications10.1016/j.jisa.2024.10374682:COnline publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1016/j.jisa.2024.103746
Sigalov DGamayunov D(2024)Finding Server-Side Endpoints with Static Analysis of Client-Side JavaScriptComputer Security. ESORICS 2023 International Workshops10.1007/978-3-031-54129-2_26(442-458)Online publication date: 12-Mar-2024
https://doi.org/10.1007/978-3-031-54129-2_26
Show More Cited By

Index Terms

XIEv: dynamic analysis for crawling and modeling of web applications

Recommendations

CHIEv: concurrent hybrid analysis for crawling and modeling of web applications

Researchers and practitioners in the fields of testing, security assessment and web development seeking to evaluate a given web application often have to rely on the existence of a model of the respective system, which is then used as input to task-...
Reverse engineering techniques: From web applications to rich Internet applications
WSE '13: Proceedings of the 2013 IEEE 15th International Symposium on Web Systems Evolution (WSE)

Web systems evolved in the last years starting from static websites to Web applications, up to Ajax-based Rich Internet Applications (RIAs). Reverse Engineering techniques followed the same evolution, too. The authors and many other WSE contributors ...
A Model-Based Approach for Crawling Rich Internet Applications

New Web technologies, like AJAX, result in more responsive and interactive Web applications, sometimes called Rich Internet Applications (RIAs). Crawling techniques developed for traditional Web applications are not sufficient for crawling RIAs. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

March 2020

2348 pages

ISBN:9781450368667

DOI:10.1145/3341105

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Tomas Cerny
Baylor University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Alessio Bechini
University of Pisa, Italy

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Österreichische Forschungsförderungsgesellschaft

Conference

SAC '20

Sponsor:

SIGAPP

SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing

March 30 - April 3, 2020

Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
205
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rosa Martins FSeixas Pereira LDuarte C(2024)QualState: Finding Website States for Accessibility EvaluationProceedings of the 21st International Web for All Conference10.1145/3677846.3677851(96-105)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677851
Sigalov DGamayunov D(2024)Dead or aliveJournal of Information Security and Applications10.1016/j.jisa.2024.10374682:COnline publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1016/j.jisa.2024.103746
Sigalov DGamayunov D(2024)Finding Server-Side Endpoints with Static Analysis of Client-Side JavaScriptComputer Security. ESORICS 2023 International Workshops10.1007/978-3-031-54129-2_26(442-458)Online publication date: 12-Mar-2024
https://doi.org/10.1007/978-3-031-54129-2_26
Hassanshahi BLee HKrishnan P(2022)Gelato: Feedback-driven and Guided Security Analysis of Client-side Web Applications2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00079(618-629)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00079
Leithner MSimos D(2021)CHIEvACM SIGAPP Applied Computing Review10.1145/3477133.347713421:1(5-23)Online publication date: 20-Jul-2021
https://dl.acm.org/doi/10.1145/3477133.3477134
Corazza ADi Martino SPeron AStarace LLanubile F(2021)Web Application TestingProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484187(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484187

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten