Siva Natarajan
ABSTRACT
Developing mobile applications is typically a labor-intensive process in which software engineers manually re-implement in code screen designs, inter-screen transitions, and in-screen animations developed by user interface and user experience experts. Other engineering domains have used computer vision techniques to automate human perception and manual data entry tasks. The P2A tool adopts computer vision techniques for developing animated mobile applications. P2A infers from mobile application screen designs the user interface portion of an application's source code and other assets that are ready to be compiled and executed on a mobile phone. Among others, inferred mobile applications contain inter-screen transitions and in-screen animations. In our experiments on screenshots of 30 highly-ranked third-party Android applications, the P2A-generated application user interfaces exhibited high pixel-to-pixel similarity with their input screenshots. P2A took an average of 26 seconds to infer in-screen animations.
- Apple Inc. 2013. View programming guide for iOS. https://developer.apple.com/library/content/documentation/WindowsViews/Conceptual/ViewPG<sub>i</sub>PhoneOS/Introduction/Introduction.html. (Oct. 2013). Accessed March 2018.Google Scholar
- Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of Android apps. In Proc. ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA). ACM, 641--660. Google ScholarDigital Library
- Patti Bao, Jeffrey S. Pierce, Stephen Whittaker, and Shumin Zhai. 2011. Smart phone use by non-mobile business users. In Proc. 13th Conference on Human-Computer Interaction with Mobile Devices and Services (Mobile HCI). ACM, 445--454. Google ScholarDigital Library
- Gary Bradski and Adrian Kaehler. 2008. Learning OpenCV: Computer Vision with the OpenCV Library (first ed.). O'Reilly.Google Scholar
- Anabela Caetano, Neri Goulart, Manuel Fonseca, and Joaquim Jorge. 2002. JavaS-ketchIt: Issues in sketching the look of user interfaces. In Proc. AAAI Spring Symposium on Sketch Understanding. AAAI, 9--14.Google Scholar
- Pedro F. Campos and Nuno Jardim Nunes. 2007. Practitioner tools and workstyles for user-interface design. IEEE Software 24, 1 (Jan. 2007), 73--80. Google ScholarDigital Library
- John Canny. 1986. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 6 (Nov. 1986), 679--698. Google ScholarDigital Library
- Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2011. Associating the visual representation of user interfaces with their internal structures and metadata. In Proc. 24th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 245--256. Google ScholarDigital Library
- Stéphane Chatty, Stéphane Sire, Jean-Luc Vinot, Patrick Lecoanet, Alexandre Lemort, and Christophe P. Mertz. 2004. Revisiting visual interface programming: creating GUI tools for designers and programmers. In Proc. 17th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 267--276. Google ScholarDigital Library
- Adrien Coyette, Suzanne Kieffer, and Jean Vanderdonckt. 2007. Multi-fidelity prototyping of user interfaces. In Proc. 11th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT). Springer, 150--164. Google ScholarDigital Library
- Tiago Silva da Silva, Angela Martin, Frank Maurer, and Milene Selbach Silveira. 2011. User-centered design and agile methods: A systematic review. In Proc. Agile Conference (AGILE). IEEE, 77--86. Google ScholarDigital Library
- Marco de Sà, Luís Carriço, Luís Duarte, and Tiago Reis. 2008. A mixed-fidelity prototyping tool for mobile devices. In Proc. Working Conference on Advanced Visual Interfaces (AVI). ACM, 225--232. Google ScholarDigital Library
- Morgan Dixon and James Fogarty. 2010. Prefab: Implementing advanced behaviors using pixel-based reverse engineering of interface structure. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 1525--1534. Google ScholarDigital Library
- Morgan Dixon, Daniel Leventhal, and James Fogarty. 2011. Content and hierarchy in pixel-based methods for reverse engineering interface structure. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 969--978. Google ScholarDigital Library
- Leopoldina Fortunati and Sakari Taipale. 2014. The advanced use of mobile phones in five European countries. The British Journal of Sociology 65, 2 (June 2014), 317--337.Google ScholarCross Ref
- Marko Gargenta and Masumi Nakamura. 2014. Learning Android: Develop Mobile Apps Using Java and Eclipse (second ed.). O'Reilly. Google ScholarDigital Library
- Zahid Hussain, Martin Lechner, Harald Milchrahm, Sara Shahzad, Wolfgang Slany, Martin Umgeher, Thomas Vlk, and Peter Wolkerstorfer. 2008. User interface design for a mobile multimedia application: An iterative approach. In Proc. 1st International Conference on Advances in Computer-Human Interaction (ACHI). IEEE, 189--194. Google ScholarDigital Library
- Flinto Inc. 2017. https://www.flinto.com/. (2017). Accessed March 2018.Google Scholar
- Google Inc. 2017. Material design for Android. https://developer.android.com/design/material/index.html. (2017). Accessed March 2018.Google Scholar
- Google Inc. 2017. Top free in Android apps. https://play.google.com/store/apps/collection/topselling<sub>f</sub>ree?hl=en. (2017). Accessed March 2018.Google Scholar
- Invision Inc. 2017. https://www.invisionapp.com/. (2017). Accessed March 2018.Google Scholar
- Kony Inc. 2014. Bridging the gap: Mobile app design and development. http://forms.kony.com/rs/konysolutions/images/Bridging<sub>G</sub>ap<sub>B</sub>rochure<sub>d</sub>ec<sub>1</sub>0<sub>1</sub>4.pdf. (Dec. 2014). Accessed March 2018.Google Scholar
- Motif Tools BV Inc. 2017. https://framer.com/. (2017). Accessed March 2018.Google Scholar
- Zeplin Inc. 2017. https://www.Zeplin.io/. (2017). Accessed March 2018.Google Scholar
- Amy K. Karlson, Brian Meyers, Andy Jacobs, Paul Johns, and Shaun K. Kane. 2009. Working overtime: Patterns of smartphone and PC usage in the day of an information worker. In Proc. 7th International Conference on Pervasive Computing (Pervasive). Springer, 398--405. Google ScholarDigital Library
- Kati Kuusinen and Tommi Mikkonen. 2013. Designing user experience for mobile apps: Long-term product owner perspective. In Proc. 20th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 535--540. Google ScholarDigital Library
- James A. Landay and Brad A. Myers. 1995. Interactive sketching for the early stages of user interface design. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 43--50. Google ScholarDigital Library
- James A. Landay and Brad A. Myers. 2001. Sketching interfaces: Toward more human interface design. IEEE Computer 34, 3 (March 2001), 56--64. Google ScholarDigital Library
- Xiaojun Meng, Shengdong Zhao, Yongfeng Huang, Zhongyuan Zhang, James Eagan, and Ramanathan Subramanian. 2014. WADE: simplified GUI add-on development for third-party software. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, 2221--2230. Google ScholarDigital Library
- Mehmet Kivanç Mihçak and Ramarathnam Venkatesan. 2001. New iterative geometric methods for robust perceptual image hashing. In Proc. 1st ACM Digital Rights Management Workshop (DRM). Springer, 13--21. Google ScholarDigital Library
- Vishal Monga and Brian L. Evans. 2006. Perceptual image hashing via feature points: Performance evaluation and tradeoffs. IEEE Transactions on Image Processing 15, 11 (Nov. 2006), 3452--3465. Google ScholarDigital Library
- Brad A. Myers. 1994. Challenges of HCI design and implementation. Interactions 1, 1 (Jan. 1994), 73--83. Google ScholarDigital Library
- Brad A. Myers. 2012. Graphical user interface programming. In Computer Science Handbook (second ed.), Allen B. Tucker (Ed.). CRC Press.Google Scholar
- Brad A. Myers, Scott E. Hudson, and Randy F. Pausch. 2000. Past, present, and future of user interface software tools. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 1 (March 2000), 3--28. Google ScholarDigital Library
- Vandad Nahavandipoor. 2013. iOS 7 Programming Cookbook (first ed.). O'Reilly. Google ScholarDigital Library
- Mark W. Newman and James A. Landay. 1999. Sitemaps, storyboards, and specifications: A sketch of web site design practice as manifested through artifacts. Technical Report UCB/CSD-99-1062. EECS Department, University of California, Berkeley. Google ScholarDigital Library
- Tuan A. Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with REMAUI. In Proc. 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 248--259.Google Scholar
- Greg Nudelman. 2013. Android Design Patterns: Interaction Design Solutions for Developers. Wiley. Google ScholarDigital Library
- R. Plamondon and S.N. Srihari. 2000. Online and off-line handwriting recognition: A comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1 (Jan. 2000), 63--84. Google ScholarDigital Library
- James R. Rudd, Kenneth R. Stern, and Scott Isensee. 1996. Low vs. high-fidelity prototyping debate. Interactions 3, 1 (1996), 76--85. Google ScholarDigital Library
- Seyyed Ehsan Salamati Taba, Iman Keivanloo, Ying Zou, Joanna Ng, and Tinny Ng. 2014. An Exploratory Study on the Relation between User Interface Complexity and the Perceived Quality of Android Applications. In Proc. 14th International Conference on Web Engineering (ICWE). Springer.Google Scholar
- Julian Seifert, Bastian Pfleging, Elba del Carmen Valderrama Bahamóndez, Martin Hermes, Enrico Rukzio, and Albrecht Schmidt. 2011. Mobidev: A tool for creating apps on mobile phones. In Proc. 13th Conference on Human-Computer Interaction with Mobile Devices and Services (Mobile HCI). ACM, 109--112. Google ScholarDigital Library
- Eeshan Shah and Eli Tilevich. 2011. Reverse-engineering user interfaces to facilitate porting to and across mobile devices and platforms. In Proc. Conference on Systems, Programming, and Applications: Software for Humanity (SPLASH) Workshops. ACM, 255--260. Google ScholarDigital Library
- Alireza Sahami Shirazi, Niels Henze, Albrecht Schmidt, Robin Goldberg, Benjamin Schmidt, and Hansjörg Schmauder. 2013. Insights into layout patterns of mobile user interfaces by an automatic analysis of Android apps. In Proc. ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS). ACM, 275--284. Google ScholarDigital Library
- Supernova Studio. 2018. https://supernova.studio/. (2018). Accessed March 2018.Google Scholar
- Ashwin Swaminathan, Yinian Mao, and Min Wu. 2006. Robust and secure image hashing. IEEE Transactions on Information Forensics and Security 1, 2 (June 2006), 215--230. Google ScholarDigital Library
- Richard Szeliski. 2010. Computer Vision: Algorithms and Applications. Springer. Google Scholar
- The Nielsen Company. 2013. The Mobile Consumer: A Global Snapshot. http://www.nielsen.com/us/en/insights/reports/2013/mobile-consumer-report-february-2013.html. (Feb. 2013). Accessed March 2018.Google Scholar
- Øivind Due Trier, Anil K. Jain, and Torfinn Taxt. 1996. Feature extraction methods for character recognition---A survey. Pattern Recognition 29, 4 (April 1996), 641--662.Google Scholar
- Zhou Wang and Alan C. Bovik. 2009. Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE Signal Processing Magazine 26 (Jan. 2009), 98--117. Issue 1.Google Scholar
- Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (April 2004), 600--612. Google ScholarDigital Library
- Yin Yin Wong. 1992. Rough and ready prototypes: Lessons from graphic design. In Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), Posters and Short Talks. ACM, 83--84. Google ScholarDigital Library
- Shengqian Yang, Dacong Yan, Haowei Wu, Yan Wang, and Atanas Rountev. 2015. Static control-flow analysis of user-driven callbacks in Android applications. In Proc. 37th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 89--99. Google ScholarDigital Library
- Shengqian Yang, Hailong Zhang, Haowei Wu, Yan Wang, Dacong Yan, and Atanas Rountev. 2015. Static window transition graphs for Android. In Proc. 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 658--668.Google ScholarDigital Library
- Christoph Zauner. 2010. Implementation and benchmarking of perceptual image hash functions. Master's thesis. Upper Austria University of Applied Sciences, Hagenberg Campus.Google Scholar
- Clemens Zeidler, Christof Lutteroth, Wolfgang Stürzlinger, and Gerald Weber. 2013. The Auckland layout editor: An improved GUI layout specification process. In Proc. 26th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 343--352. Google ScholarDigital Library
- Clemens Zeidler, Christof Lutteroth, Wolfgang Stürzlinger, and Gerald Weber. 2013. Evaluating direct manipulation operations for constraint-based layout. In Proc. 14th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT). Springer, 513--529.Google ScholarCross Ref
Index Terms
- P2A: a tool for converting pixels to animated mobile application user interfaces
Recommendations
A comparative study of smartphone and smartwatch apps
SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied ComputingDespite that our community has spent numerous efforts on analyzing mobile apps, there is no study proposed for characterizing the relationship between smartphone and smartwatch apps. To fill this gap, we present to the community a comparative study of ...
The evolution of mobile apps: an exploratory study
DeMobile 2013: Proceedings of the 2013 International Workshop on Software Development Lifecycle for MobileAs mobile apps continue to grow in popularity, it is important to study their evolution. Lehman's laws of software evolution have been proposed and used to study the evolution of traditional, large software systems (also known as desktop apps). However,...
An empirical history of permission requests and mistakes in open source Android apps
MSR '19: Proceedings of the 16th International Conference on Mining Software RepositoriesAndroid applications (apps) rely upon proper permission usage to ensure that the user's privacy and security are adequately protected. Unfortunately, developers frequently misuse app permissions in a variety of ways ranging from using too many ...
Comments