skip to main content
research-article

Fuzzy Clustering of Crowdsourced Test Reports for Apps

Published: 02 February 2018 Publication History

Abstract

DevOps is a new approach to drive a seamless Application (App) cycle from development to delivery. As a critical part to promote the successful implementation of DevOps, testing can significantly improve team productivity and reliably deliver user experience. However, it is difficult to use traditional testing to cover diverse mobile phones, network environments, operating systems, and so on. Hence, many large companies crowdsource their App testing tasks to workers from open platforms. In crowdsourced testing, test reports submitted by workers may be highly redundant, and their quality may vary sharply. Meanwhile, multi-bug test reports may be submitted, and their root causes are hard to diagnose. Hence, it is a time-consuming and tedious task for developers to manually inspect these test reports. To help developers address the above challenges, we issue the new problem of Fuzzy Clustering Test Reports (FULTER). Aiming to resolve FULTER, a series of barriers need to be overcome. In this study, we propose a new framework named Test Report Fuzzy Clustering Framework (TERFUR) by aggregating redundant and multi-bug test reports into clusters to reduce the number of inspected test reports. First, we construct a filter to remove invalid test reports to break through the invalid barrier. Then, a preprocessor is built to enhance the descriptions of short test reports to break through the uneven barrier. Last, a two-phase merging algorithm is proposed to partition redundant and multi-bug test reports into clusters that can break through the multi-bug barrier. Experimental results over 1,728 test reports from five industrial Apps show that TERFUR can cluster test reports by up to 78.15% in terms of AverageP, 78.41% in terms of AverageR, and 75.82% in terms of AverageF1 and outperform comparative methods by up to 31.69%, 33.06%, and 24.55%, respectively. In addition, the effectiveness of TERFUR is validated in prioritizing test reports for manual inspection.

References

[1]
Armin Balalaie, Abbas Heydarnoori, and Pooyan Jamshidi. 2015. Migrating to cloud-native architectures using microservices: An experience report. In Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC’15). Springer, 201--215.
[2]
Mauro Barni, Vito Cappellini, and Alessandro Mecocci. 1996. Comments on “A possibilistic approach to clusterin.” IEEE Trans. Fuzzy Syst. 4, 3 (1996), 393--396.
[3]
Kevin Bartz, Jack W. Stokes, John C. Platt, Ryan Kivett, David Grant, Silviu Calinoiu, and Gretchen Loihle. 2008. Finding similar failures using callstack similarity. In Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML’08). USENIX Association, San Diego, CA.
[4]
Pavel Berkhin. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data—Recent Advances in Clustering. 25--71.
[5]
James C. Bezdek. 2013. Pattern Recognition with Fuzzy Objective Function Algorithms. Springer Science 8 Business Media.
[6]
Gloria Bordogna and Gabriella Pasi. 2009. Hierarchical-hyperspherical divisive fuzzy C-means (H2D-FCM) clustering for information retrieval. In Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence (WI’09). IEEE Computer Society, 614--621.
[7]
Yguaratã Cerqueira Cavalcanti, Eduardo Santana de Almeida, Carlos Eduardo Albuquerque da Cunha, Daniel Lucrédio, and Silvio Romero de Lemos Meira. 2010. An initial study on the bug report duplication problem. In Proceedings of the 14th European Conference on Software Maintenance and Reengineering (CSMR’10). IEEE Computer Society, 264--267.
[8]
Zhenyu Chen and Bin Luo. 2014. Quasi-crowdsourcing testing for educational projects. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, 272--275.
[9]
I-Jen Chiang, Charles Chih-Ho Liu, Yi-Hsin Tsai, and Ajit Kumar. 2015. Discovering latent semantics in web documents using fuzzy clustering. IEEE Trans. Fuzzy Syst. 23, 6 (2015), 2122--2134.
[10]
Hung Chim and Xiaotie Deng. 2008. Efficient phrase-based document similarity for clustering. IEEE Trans. Knowl. Data Eng. 20, 9 (2008), 1217--1229.
[11]
Yingnong Dang, Rongxin Wu, Hongyu Zhang, Dongmei Zhang, and Peter Nobel. 2012. ReBucket: A method for clustering duplicate crash reports based on call stack similarity. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 1084--1093.
[12]
Eelco Dolstra, Raynor Vliegendhart, and Johan A. Pouwelse. 2013. Crowdsourcing GUI tests. In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 332--341.
[13]
Yang Feng, Zhenyu Chen, James A. Jones, Chunrong Fang, and Baowen Xu. 2015. Test report prioritization to assist crowdsourced testing. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’15). ACM, 225--236.
[14]
Yang Feng, James A. Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 202--213.
[15]
Fabio Guaiani and Henry Muccini. 2015. Crowd and laboratory testing, can they co-exist? An exploratory study. In Proceedings of the 2nd IEEE/ACM International Workshop on CrowdSourcing in Software Engineering (CSI-SE’15). IEEE Computer Society, 32--37.
[16]
Khaled M. Hammouda and Mohamed S. Kamel. 2004. Efficient phrase-based document indexing for web document clustering. IEEE Trans. Knowl. Data Eng. 16, 10 (2004), 1279--1296.
[17]
Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.
[18]
Michael Httermann. 2012. DevOps for Developers. Apress.
[19]
A. K. Jain, M. N. Murty, and P. J. Flynn. 1999. Data clustering: A review. ACM Comput. Surv. 31, 3 (1999), 264--323.
[20]
Jung-Yi Jiang, Ren-Jia Liou, and Shie-Jue Lee. 2011. A fuzzy self-constructing feature clustering algorithm for text classification. IEEE Trans. Knowl. Data Eng. 23, 3 (2011), 335--349.
[21]
Sunghun Kim, Thomas Zimmermann, and Nachiappan Nagappan. 2011. Crash graphs: An aggregated view of multiple crashes to improve crash triage. In Proceedings of the 2011 IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’11). IEEE Compute Society, 486--493.
[22]
B. Kirubakaran and Dr. V. Karthikeyani. 2013. Mobile application testing - Challenges and solution approach through automation. In Proceedings of the 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME’13). IEEE Computer Society.
[23]
Ravikumar Kondadadi and Robert Kozma. 2002. A modified fuzzy art for soft document clustering. In Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN’02). IEEE Computer Society, 2545--2549.
[24]
Raghu Krishnapuram, Anupam Joshi, and Liyu Yi. 1999. A fuzzy relative of the K-medoids algorithm with application to web document and snippet clustering. In Proceedings of the IEEE InternationalFuzzy Systems Conference Proceedings (FUZZ-IEEE’99). IEEE Computer Society, 1281--1286.
[25]
Yves Ledru, Alexandre Petrenko, and Sergiy Boroday. 2009. Using string distances for test case prioritisation. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE’09). IEEE Computer Society, 510--514.
[26]
Yanjun Li, Soon M. Chung, and John D. Holt. 2008. Text document clustering based on frequent word meaning sequences. Data Knowl. Eng. 64, 1 (2008), 381--404.
[27]
Yanjun Li, Congnan Luo, and Soon M. Chung. 2008. Text clustering with feature selection by using statistical data. IEEE Trans. Knowl. Data Eng. 20, 5 (2008), 641--652.
[28]
King-Ip Lin and Ravikumar Kondadadi. 2001. A similarity-based soft clustering algorithm for documents. In Proceedings of the 7th International Conference on Database Systems for Advanced Applications (DASFAA’01). IEEE Computer Society, 40--47.
[29]
Di Liu, Randolph G. Bias, Matthew Lease, and Rebecca Kuipers. 2012. Crowdsourcing for usability testing. Proc. Am. Soc. Inf. Sci. Technol. 49, 1 (2012), 1--10.
[30]
Christopher D. Manning and Hinrich Schütze. 2001. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge.
[31]
Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2017. A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software 126 (2017), 57--84.
[32]
Louis Massey. 2005. Evaluating and comparing text clustering results. In Proceedings of the International Conference on Computational Intelligence. IASTED/ACTA Press, Alberta, Canada, 85--90.
[33]
M. E. S. Mendes and Lionel Sacks. 2004. Dynamic knowledge representation for E-learning applications. In Enhancing the Power of the Internet. Springer, 259--282.
[34]
Natwar Modani, Rajeev Gupta, Guy M. Lohman, Tanveer Fathima Syeda-Mahmood, and Laurent Mignet. 2007. Automatically identifying known software problems. In Proceedings of the 23rd International Conference on Data Engineering Workshops (ICDE’07). IEEE Computer Society, 433--441.
[35]
Michael Nebeling, Maximilian Speicher, Michael Grossniklaus, and Moira C. Norrie. 2012. Crowdsourced web site evaluation with crowdstudy. In Proceedings of 12th International Conference on Web Engineering (ICWE’12). Springer, 494--497.
[36]
Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N. Nguyen, David Lo, and Chengnian Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’12). ACM, 70--79.
[37]
Rahul R. Papalkar and G. Chandel. 2013. Fuzzy clustering in web text mining and its application in IEEE abstract classification. Int. J. Comput. Sci. Manage. Rese. 2, 2 (2013), 1529--1533.
[38]
Fabrizio Pastore, Leonardo Mariani, and Gordon Fraser. 2013. CrowdOracles: Can the crowd solve the oracle problem? In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 342--351.
[39]
Andy Podgurski, David Leon, Patrick Francis, Wes Masri, Melinda Minch, Jiayang Sun, and Bin Wang. 2003. Automated support for classifying software failure reports. In Proceedings of the 25th International Conference on Software Engineering (ICSE’03). IEEE Compute Society, 465--477.
[40]
M. E. S. Mendes Rodrigues and L. Sacks. 2004. A scalable hierarchical fuzzy clustering algorithm for text mining. In Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC’04). IEEE Computer Society, 269--274.
[41]
Guoping Rong, He Zhang, and Dong Shao. 2016. CMMI guided process improvement for devops projects: An exploratory case study. In Proceedings of the International Conference on Software and Systems Process (ICSSP’16). ACM, 76--85.
[42]
Dmitri Roussinov and Hsinchun Chen. 1999. Document clustering for electronic meetings: An experimental comparison of two techniques. Decis. Support Syst. 27, 1--2 (1999), 67--79.
[43]
Per Runeson, Magnus Alexandersson, and Oskar Nyholm. 2007. Detection of duplicate defect reports using natural language processing. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 499--510.
[44]
Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo Juzgado. 2015. Are students representatives of professionals in software engineering experiments? In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE’15). IEEE Computer Society, 666--676.
[45]
Andrew Skabar and Khaled Abdalgader. 2013. Clustering sentence-level text using a novel fuzzy relational clustering algorithm. IEEE Trans. Knowl. Data Eng. 25, 1 (2013), 62--75.
[46]
Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11). IEEE Computer Society, 253--262.
[47]
Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, Volume 1 (ICSE’10). ACM, 45--54.
[48]
Amy J. C. Trappey, Charles V. Trappey, Fu-Chiang Hsu, and David W. Hsiao. 2009. A fuzzy ontological knowledge document clustering methodology. IEEE Trans. Syst. Man Cybernet. B 39, 3 (2009), 806--814.
[49]
Junjie Wang, Qiang Cui, Qing Wang, and Song Wang. 2016. Towards effectively test report classification to assist crowdsourced testing. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’16). 6:1--6:10.
[50]
Junjie Wang, Song Wang, Qiang Cui, and Qing Wang. 2016. Local-based active classification of test report to assist crowdsourced testing. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 190--201.
[51]
Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. 2008. An approach to detecting duplicate bug reports using natural language and execution information. In Proceedings of the 30th International Conference on Software Engineering (ICSE’08). ACM, 461--470.
[52]
Johannes Wettinger, Vasilios Andrikopoulos, and Frank Leymann. 2015. Automated capturing and systematic usage of devops knowledge for cloud applications. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’15). IEEE Computer Society, 60--65.
[53]
Thaung Thaung Win and LinMon. 2010. Document clustering by fuzzy C-mean algorithm. In Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC’10). IEEE Computer Society, 239--242.
[54]
Jinglin Xu, Junwei Han, Kai Xiong, and Feiping Nie. 2016. Robust and sparse fuzzy K-means clustering. In 25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’2016). IJCAI/AAAI Press, New York, 2224--2230.
[55]
Jifeng Xuan, He Jiang, Yan Hu, Zhilei Ren, Weiqin Zou, Zhongxuan Luo, and Xindong Wu. 2015. Towards effective bug triage with software data reduction techniques. IEEE Trans. Knowl. Data Eng. 27, 1 (2015), 264--280.
[56]
Jifeng Xuan, He Jiang, Zhilei Ren, Jun Yan, and Zhongxuan Luo. 2010. Automatic bug triage using semi-supervised text classification. In Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE’10). Knowledge Systems Institute Graduate School, Zurich, Switzerland, 209--214.
[57]
Jifeng Xuan, He Jiang, Zhilei Ren, and Weiqin Zou. 2012. Developer prioritization in bug repositories. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 25--35.
[58]
Rongrong Zhang, Qingtian Zeng, and Sen Feng. 2010. Data query using short domain question in natural language. In Proceedings of the 2010 IEEE 2nd Symposium on Web Society (SWS’10). IEEE Computer Society, 351--354.
[59]
Wen Zhang, Taketoshi Yoshida, Xijin Tang, and Qing Wang. 2010. Text clustering using frequent itemsets. Knowl.-Based Syst. 23, 5 (2010), 379--388.

Cited By

View all
  • (2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
  • (2025)Redefining crowdsourced test report prioritization: An innovative approach with large language modelInformation and Software Technology10.1016/j.infsof.2024.107629179(107629)Online publication date: Mar-2025
  • (2025)Collaboration failure analysis in cyber-physical system-of-systems using context fuzzy clusteringEmpirical Software Engineering10.1007/s10664-024-10572-330:2Online publication date: 1-Mar-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology
ACM Transactions on Internet Technology  Volume 18, Issue 2
Special Issue on Internetware and Devops and Regular Papers
May 2018
294 pages
ISSN:1533-5399
EISSN:1557-6051
DOI:10.1145/3182619
  • Editor:
  • Munindar P. Singh
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 February 2018
Accepted: 01 June 2017
Revised: 01 February 2017
Received: 01 September 2016
Published in TOIT Volume 18, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Crowdsourced testing
  2. duplicate detection
  3. fuzzy clustering
  4. test report
  5. unsupervised method

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)2
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
  • (2025)Redefining crowdsourced test report prioritization: An innovative approach with large language modelInformation and Software Technology10.1016/j.infsof.2024.107629179(107629)Online publication date: Mar-2025
  • (2025)Collaboration failure analysis in cyber-physical system-of-systems using context fuzzy clusteringEmpirical Software Engineering10.1007/s10664-024-10572-330:2Online publication date: 1-Mar-2025
  • (2024)Clustering and Prioritization of Web Crowdsourced Test Reports Based on Text ClassificationInternational Journal of Web Services Research10.4018/IJWSR.35799921:1(1-19)Online publication date: 7-Nov-2024
  • (2024)Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text ConversionSymmetry10.3390/sym1601008016:1(80)Online publication date: 8-Jan-2024
  • (2024)Design and implementation of a vulnerability mining system for power internet of things terminal based on fuzzy mathematical model simulation platformIntelligent Decision Technologies10.3233/IDT-23024118:4(2873-2883)Online publication date: 1-Jan-2024
  • (2024)UEFI-based Research on the Inner Operation Mechanism and Characteristics of Firmware Vulnerabilities in Key Devices of Electric Power Monitoring SystemsApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-01369:1Online publication date: 31-Jan-2024
  • (2024)Research Review of Crowdsourcing TestingSoftware Engineering and Applications10.12677/sea.2024.13302913:03(295-301)Online publication date: 2024
  • (2024)Semi-supervised Crowdsourced Test Report Clustering via Screenshot-Text Binding RulesProceedings of the ACM on Software Engineering10.1145/36607761:FSE(1540-1563)Online publication date: 12-Jul-2024
  • (2023)A Survey on Bug Deduplication and Triage Methods from Multiple Points of ViewApplied Sciences10.3390/app1315878813:15(8788)Online publication date: 29-Jul-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media