skip to main content
10.1145/3290605.3300920acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

A Badge, Not a Barrier: Designing for-and Throughout-Digital Badge Implementation

Published: 02 May 2019 Publication History

Abstract

We synthesize insights from a multi-year project involving the design and implementation of a digital badge system with youth co-designers at a science center. Using stakeholder interviews and surveys, participatory design session data, and user analytics, we identify the sociotechnical, sociocultural, and technical challenges of long-term badge implementation and propose several recommendations for the design and implementation of future badge systems. By identifying these challenges and providing recommendations that foreground stakeholder values and participation, we show how to support implementation throughout the entire design-to-implementation cycle.

Supplementary Material

MP4 File (paper690.mp4)

References

[1]
Samuel Abramovich, Christian Schunn, and Ross Mitsuo Higashi. 2013. Are badges useful in education?: it depends upon the type of badge and expertise of learner. Educational Technology Research and Development 61, 2 (March 2013), 217--232.
[2]
June Ahn, Anthony Pellicone, and Brian S. Butler. 2014. Open badges for education: what are the implications at the intersection of open systems and badging? Research in Learning Technology 22, 0 (Aug. 2014).
[3]
Raewyn Bassett, Brenda L. Beagan, Svetlana Ristovski-Slijepcevic, and Gwen E. Chapman. 2008. Tough Teens: The Methodological Challenges of Interviewing Teenagers as Research Participants. Journal of Adolescent Research 23, 2 (March 2008), 119--131.
[4]
Gordon Baxter and Ian Sommerville. 2011. Socio-technical systems: From design methods to systems engineering. Interacting with Computers 23, 1 (Jan. 2011), 4--17.
[5]
Adam Bell and Katie Davis. 2016. Learning Through Participatory Design: Designing Digital Badges for and with Teens. In Proceedings of the The 15th International Conference on Interaction Design and Children (IDC '16). ACM, New York, NY, USA, 218--229.
[6]
Philip Bell. 2004. On the Theoretical Breadth of Design-Based Research in Education. Educational Psychologist 39, 4 (Dec. 2004), 243--253.
[7]
Hugh R. Beyer and Karen Holtzblatt. 1995. Apprenticing with the Customer. Commun. ACM 38, 5 (May 1995), 45--52.
[8]
Erling Björgvinsson, Pelle Ehn, and Per-Anders Hillgren. 2010. Participatory Design and "Democratizing Innovation". In Proceedings of the 11th Biennial Participatory Design Conference (PDC '10). ACM, New York, NY, USA, 41--50.
[9]
Goldie Blumenstyk. 2018. With Employers in the Mix, Can Badges Become More Than a Fad? The Chronicle of Higher Education (Aug. 2018). https://www.chronicle.com/article/ With-Employers-in-the-Mix-Can/244322
[10]
Ivica Boticki, Jelena Baksa, Peter Seow, and Chee-Kit Looi. 2015. Usage of a mobile social learning platform with virtual badges in a primary school. Computers & Education 86 (Aug. 2015), 120--136.
[11]
Ann L. Brown. 1992. Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences 2, 2 (April 1992), 141--178.
[12]
Carla Casilli and Daniel Hickey. 2016. Transcending conventional credentialing and assessment paradigms with information-rich digital badges. The Information Society 32, 2 (March 2016), 117--129.
[13]
Paul Cobb, Jere Confrey, Andrea diSessa, Richard Lehrer, and Leona Schauble. 2003. Design Experiments in Educational Research. Educational Researcher 32, 1 (Jan. 2003), 9--13.
[14]
Cynthia E. Coburn, William R. Penuel, and Kimberly E. Geil. 2013. Research-practice partnerships at the district level: A new strategy for leveraging research for educational improvement. Technical Report. William T. Grant Foundation, New York. https://wtgrantfoundation.org/library/uploads/2015/10/ Research-Practice-Partnerships-at-the-District-Level.pdf
[15]
Cynthia E. Coburn and Mary Kay Stein (Eds.). 2010. Research and Practice in Education: Building Alliances, Bridging the Divide. Rowman & Littlefeld Publishers.
[16]
Michael Cole. 2006. The Fifth Dimension: An after-school program built on diversity. Russell Sage Foundation.
[17]
Juliet Corbin and Anselm Strauss. 2015. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (4 ed.). SAGE.
[18]
Alexei Czeskis, Ivayla Dermendjieva, Hussein Yapit, Alan Borning, Batya Friedman, Brian Gill, and Tadayoshi Kohno. 2010. Parenting from the Pocket: Value Tensions and Technical Directions for Secure and Private Parent-teen Mobile Safety. In Proceedings of the Sixth Symposium on Usable Privacy and Security (SOUPS '10). ACM, New York, NY, USA, 15:1--15:15.
[19]
Elizabeth A. Davis, Annemarie Sullivan Palincsar, P. Sean Smith, Anna Maria Arias, and Sylvie M. Kademian. 2017. Educative Curriculum Materials: Uptake, Impact, and Implications for Research and Design. Educational Researcher 46, 6 (Aug. 2017), 293--304.
[20]
Katie Davis and Eve Klein. 2015. Investigating High School Students' Perceptions of Digital Badges in Afterschool Learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 4043--4046.
[21]
Katie Davis, Caroline Pitt, Adam Bell, and Ada Kim. 2018. Using digital badges to promote student agency and identity in science learning. In Proceedings of the Connected Learning Summit (CLS '18).
[22]
Katie Davis and Simrat Singh. 2015. Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Computers & Education 88 (Oct. 2015), 72--83.
[23]
Christian Dindler, Ole Sejer Iversen, Rachel Smith, and Rune Veerasawmy. 2010. Participatory Design at the Museum: Inquiring into Children's Everyday Engagement in Cultural Heritage. In Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction (OZCHI '10). ACM, New York, NY, USA, 72--79.
[24]
Allison Druin. 1999. Cooperative Inquiry: Developing New Technologies for Children with Children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). 592--599.
[25]
Allison Druin. 2002. The role of children in the design of new technology. Behaviour & Information Technology 21, 1 (Jan. 2002), 1--25.
[26]
Angela Elkordy. 2016. Development and Implementation of Digital Badges for Learning Science, Technologly, Engineering and Math (STEM) Practices in Secondary Contexts: A Pedagogical Approach with Empirical Evidence. In Foundation of Digital Badges and MicroCredentials: Demonstrating and Recognizing Knowledge and Competencies, Dirk Ifenthaler, Nicole Bellin-Mularski, and Dana-Kristin Mah (Eds.). Springer International Publishing, Cham, 483--508.
[27]
Angela Elkordy. 2016. Digital Badges for STEM Learning in Secondary Contexts: A Mixed Methods Study. Ph.D. Eastern Michigan University, United States -- Michigan. https://commons.emich.edu/theses/809/
[28]
Yrjö Engeström. 2008. From Design Experiments to Formative Interventions. In Proceedings of the 8th International Conference on International Conference for the Learning Sciences - Volume 1 (ICLS'08). International Society of the Learning Sciences, Utrecht, The Netherlands, 3--24. http://dl.acm.org/citation.cfm?id=1599812.1599814
[29]
Barry Fishman, Stephanie Teasley, and Steven Cederquist. 2018. MicroCredentials as Evidence for College Readiness: Report of an NSF Workshop. Technical Report. 27 pages. https://deepblue.lib.umich.edu/handle/ 2027.42/143851
[30]
Barry J. Fishman, William R. Penuel, Anna-Ruth Allen, Britte Haugan Cheng, and Nora Sabelli. 2013. Design-based implementation research: An emerging model for transforming the relationship of research and practice. National Society for the Study of Education 112, 2 (2013), 136--156.
[31]
Daniel Fitton, Janet C. Read, and Matthew Horton. 2013. The Challenge of Working with Teens As Participants in Interaction Design. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 205--210.
[32]
Geraldine Fitzpatrick and Gunnar Ellingsen. 2013. A Review of 25 Years of CSCW Research in Healthcare: Contributions, Challenges and Future Agendas. Computer Supported Cooperative Work (CSCW) 22, 4 (Aug. 2013), 609--665.
[33]
Mary Flanagan, Daniel C. Howe, and Helen Nissenbaum. 2005. Values at Play: Design Tradeofs in Socially-oriented Game Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 751--760.
[34]
Christine Floyd, Wolf-Michael Mehl, Fanny-Michaela Resin, Gerhard Schmidt, and Gregor Wolf. 1989. Out of Scandinavia: Alternative Approaches to Software Design and System Development. Human--Computer Interaction 4, 4 (Dec. 1989), 253--350.
[35]
Emily Ford, Betty Izumi, Jost Lottes, and Dawn Richardson. 2015. Badge it!: A collaborative learning outcomes based approach to integrating information literacy badges within disciplinary curriculum. Reference Services Review 43, 1 (Feb. 2015), 31--44.
[36]
Christopher Frauenberger, Judith Good, and Wendy Keay-Bright. 2011. Designing technology for children with special needs: bridging perspectives through participatory design. CoDesign 7, 1 (March 2011), 1--28.
[37]
Linda Frederiksen. 2013. Digital Badges. Public Services Quarterly 9, 4 (Oct. 2013), 321--325.
[38]
Batya Friedman and David Hendry. {n. d.}. Value Sensitive Design: The Shape of Being with Technology. MIT Press, Cambridge, MA.
[39]
Batya Friedman, Lisa P. Nathan, and Daisy Yoo. 2016. Multi-Lifespan Information System Design in Support of Transitional Justice: Evolving Situated Design Principles for the Long (er) Term. Interacting with Computers (2016), 80--96.
[40]
Batya Friedman and Daisy Yoo. 2017. Pause: A Multi-lifespan Design Mechanism. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 460--464.
[41]
Sheryl Grant. 2014. Badges: Show What You Know. Young Adult Library Services 12, 2 (2014), 28--32.
[42]
Mona Leigh Guha, Allison Druin, and Jerry Alan Fails. 2013. Cooperative Inquiry revisited: Refections of the past and guidelines for the future of intergenerational co-design. International Journal of Child-Computer Interaction 1, 1 (Jan. 2013), 14--23.
[43]
Kris D. Gutiérrez and Shirin Vossoughi. 2010. Lifting Of the Ground to Return Anew: Mediated Praxis, Transformative Learning, and Social Design Experiments. Journal of Teacher Education 61, 1--2 (Jan. 2010), 100--117.
[44]
Michael D. Hanus and Jesse Fox. 2015. Assessing the efects of gamifcation in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, efort, and academic performance. Computers & Education 80 (Jan. 2015), 152--161.
[45]
Daniel Hickey. 2017. How Open E-Credentials Will Transform Higher Education. The Chronicle of Higher Education (April 2017). https: //www.chronicle.com/article/How-Open-E-Credentials-Will/239709
[46]
Daniel Hickey and Katerina Schenke. 2019. Digital badges and reward structures. In The Cambridge Handbook of Motivation. Cambridge University Press, New York, NY, USA.
[47]
Daniel Hickey, Suraj L. Uttamchandani, and Grant T. Chartrand. 2019. Competencies in Context: New Approaches to Capturing, Recognizing, and Endorsing Learning. In Handbook of Research in Educational Communications and Technology. Springer, New York, NY, USA.
[48]
Daniel T Hickey and James E. Willis. 2017. Where Open Badges Appear to Work Better: Findings from the Design Principles Documentation Project. Technical Report. Indiana University, Center for Research on Learning and Technology. 87 pages. https://www.researchgate.net/ profle/Daniel_Hickey2/publication/312740974_Design_Principles_ for_Digital_Badge_Systems/links/5a818eb10f7e9be137ca69af/ Design-Principles-for-Digital-Badge-Systems.pdf
[49]
Clara E. Hill, Barbara J. Thompson, and Elizabeth Nutt Williams. 1997. A Guide to Conducting Consensual Qualitative Research. The Counseling Psychologist 25, 4 (Oct. 1997), 517--572.
[50]
Heather C. Hill and Charalambos Y. Charalambous. 2012. Teacher knowledge, curriculum materials, and quality of instruction: Lessons learned and open issues. Journal of Curriculum Studies 44, 4 (Aug. 2012), 559--576.
[51]
Ole Sejer Iversen and Rachel Charlotte Smith. 2012. Scandinavian Participatory Design: Dialogic Curation with Teenagers. In Proceedings of the 11th International Conference on Interaction Design and Children (IDC '12). ACM, New York, NY, USA, 106--115.
[52]
Henry Jenkins. 2012. How to Earn Your Skeptic "Badge". http: //henryjenkins.org/2012/03/how_to_earn_your_skeptic_badge.html
[53]
A. Susan Jurow and Molly Shea. 2015. Learning in Equity-Oriented Scale-Making Projects. Journal of the Learning Sciences 24, 2 (April 2015), 286--307.
[54]
Karl M. Kapp. 2013. The Gamifcation of Learning and Instruction Fieldbook: Ideas into Practice. John Wiley & Sons.
[55]
John Laubersheimer, Dorothy Ryan, and John Champaign. 2016. InfoSkills2Go: Using Badges and Gamifcation to Teach Information Literacy Skills and Concepts to College-Bound High School Students. Journal of Library Administration 56, 8 (Nov. 2016), 924--938.
[56]
Joseph A. Maxwell. 2012. Qualitative Research Design: An Interactive Approach. SAGE.
[57]
Sharan B. Merriam. 2009. Qualitative Research: A Guide to Design and Implementation. John Wiley & Sons.
[58]
Matthew B. Miles, A. Michael Huberman, and Johnny Saldaña. 2014. Qualitative Data Analysis: A methods sourcebook. SAGE.
[59]
Lin Y. Muilenburg and Zane L. Berge. 2016. Digital Badges in Education: Trends, Issues, and Cases. Routledge.
[60]
Lisa P. Nathan, Predrag V. Klasnja, and Batya Friedman. 2007. Value Scenarios: A Technique for Envisioning Systemic Efects of New Technologies. In CHI '07 Extended Abstracts on Human Factors in Computing Systems (CHI EA '07). ACM, New York, NY, USA, 2585--2590.
[61]
Tobias Nyström and Moyen Mohammad Mustaquim. 2014. Sustainable Information System Design and the Role of Sustainable HCI. In Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services (AcademicMindTrek '14). ACM, New York, NY, USA, 66--73.
[62]
Michael Olneck. 2015. Whom Will Digital Badges Empower? Sociological Perspectives on Digital Badges. 5--11. http://ceur-ws.org/ Vol-1358/paper1.pdf
[63]
Luís Pedro, Carlos Santos, Mónica Aresta, and Sara Almeida. 2015. Peer-supported badge attribution in a collaborative learning platform: The SAPO Campus case. Computers in Human Behavior 51, Part B (Oct. 2015), 562--567.
[64]
William R Penuel. 2015. Infrastructuring as a practice for promoting transformation and equity in design-based implementation research. https://tinyurl.com/Infrastructuring-DBIR
[65]
William R. Penuel, Anna-Ruth Allen, Cynthia E. Coburn, and Caitlin Farrell. 2015. Conceptualizing Research--Practice Partnerships as Joint Work at Boundaries. Journal of Education for Students Placed at Risk (JESPAR) 20, 1--2 (April 2015), 182--197.
[66]
Thomas M. Philip, Megan Bang, and Kara Jackson. 2018. Articulating the "How," the "For What," the "For Whom," and the "With Whom" in Concert: A Call to Broaden the Benchmarks of our Scholarship. Cognition and Instruction 36, 2 (April 2018), 83--88.
[67]
Caroline Pitt and Katie Davis. 2017. Designing Together?: Group Dynamics in Participatory Digital Badge Design with Teens. In Proceedings of the 2017 Conference on Interaction Design and Children (IDC '17). ACM, New York, NY, USA, 322--327.
[68]
Janet C. Read, Daniel Fitton, and Matthew Horton. 2014. Giving Ideas an Equal Chance: Inclusion and Representation in Participatory Design with Children. In Proceedings of the 2014 Conference on Interaction Design and Children (IDC '14). ACM, New York, NY, USA, 105--114.
[69]
Mitchel Resnick. 2012. Still a Badge Skeptic. https://www.hastac.org/ blogs/mres/2012/02/27/still-badge-skeptic
[70]
Michelle M. Riconscente, Amy Kamarainen, and Margaret Honey. 2013. STEM badges: Current terrain and the road ahead. Technical Report. https://badgesnysci.fles.wordpress.com/2013/08/nsf_ stembadges_fnal_report.pdf
[71]
Nancy R. Romance and Michael R. Vitale. 2001. Implementing an in-depth expanded science model in elementary schools: Multi-year fndings, research issues, and policy implications. International Journal of Science Education 23, 4 (April 2001), 373--404.
[72]
William Sandoval. 2014. Conjecture Mapping: An Approach to Systematic Educational Design Research. Journal of the Learning Sciences 23, 1 (Jan. 2014), 18--36.
[73]
Donald A. Schön. 1987. Educating the refective practitioner: Toward a new design for teaching and learning in the professions. Jossey-Bass.
[74]
Peter Smagorinsky. 2008. The Method Section as Conceptual Epicenter in Constructing Social Science Research Reports. Written Communication 25, 3 (July 2008), 389--411.
[75]
Susan Leigh Star and Karen Ruhleder. 1996. Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces. Information Systems Research 7, 1 (March 1996), 111--134.
[76]
Cathy Tran, Katerina Schenke, and Daniel Hickey. 2014. Design principles for motivating learning with digital badges: Consideration of contextual factors of recognition and assessment. Proceedings of International Conference of the Learning Sciences, ICLS 2 (Jan. 2014), 1027--1031.
[77]
Greg Walsh, Elizabeth Foss, Jason Yip, and Allison Druin. 2013. FACIT PD: A Framework for Analysis and Creation of Intergenerational Techniques for Participatory Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2893--2902.
[78]
Jason C. Yip, Tamara Clegg, June Ahn, Judith Odili Uchidiuno, Elizabeth Bonsignore, Austin Beck, Daniel Pauw, and Kelly Mills. 2016. The Evolution of Engagements and Social Bonds During Child-Parent Co-design. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3607--3619.
[79]
Jason C. Yip, Kiley Sobel, Caroline Pitt, Kung Jin Lee, Sijin Chen, Kari Nasu, and Laura R. Pina. 2017. Examining Adult-Child Interactions in Intergenerational Participatory Design. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5742--5754.
[80]
Panayiotis Zaphiris, Sri Kurniawan, and Mariya Ghiawadwala Bulsara. 2006. A systematic approach to the development of research-based web design guidelines for older people. Universal Access in the Information Society 6 (May 2006).

Cited By

View all
  • (2023)Understanding accounting students’ intentions to use digital badges to showcase employability skillsAccounting Education10.1080/09639284.2023.227620033:6(906-934)Online publication date: 11-Nov-2023
  • (2022)Digital badges affect need satisfaction but not frustration in males in higher educationComputers & Education10.1016/j.compedu.2022.104484182:COnline publication date: 1-Jun-2022
  • (2021)Connected Learning, Collapsed ContextsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445635(1-14)Online publication date: 6-May-2021
  • Show More Cited By

Index Terms

  1. A Badge, Not a Barrier: Designing for-and Throughout-Digital Badge Implementation

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
      May 2019
      9077 pages
      ISBN:9781450359702
      DOI:10.1145/3290605
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 May 2019

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. design methods
      2. design recommendations
      3. digital badges
      4. implementation
      5. informal education
      6. participatory design
      7. sociotechnical system

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      CHI '19
      Sponsor:

      Acceptance Rates

      CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)290
      • Downloads (Last 6 weeks)33
      Reflects downloads up to 08 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2023)Understanding accounting students’ intentions to use digital badges to showcase employability skillsAccounting Education10.1080/09639284.2023.227620033:6(906-934)Online publication date: 11-Nov-2023
      • (2022)Digital badges affect need satisfaction but not frustration in males in higher educationComputers & Education10.1016/j.compedu.2022.104484182:COnline publication date: 1-Jun-2022
      • (2021)Connected Learning, Collapsed ContextsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445635(1-14)Online publication date: 6-May-2021
      • (2020)Do Badges Affect Intrinsic Motivation in Introductory Programming Students?Simulation and Gaming10.1177/104687811988499651:1(33-54)Online publication date: 1-Feb-2020

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media