skip to main content
10.1145/3531146.3533227acmotherconferencesArticle/Chapter ViewAbstractPublication PagesfacctConference Proceedingsconference-collections
research-article

Testing Concerns about Technology's Behavioral Impacts with N-of-one Trials

Published: 20 June 2022 Publication History

Abstract

As public trust in technology companies has declined, people are questioning the effects of digital technologies in their lives. In this context, many evidence-free claims from corporations and tech critics are widely circulated. How can members of the public make evidence-based decisions about digital technology in their lives? In clinical fields, N -of-one trials enable participant-investigators to make personalized causal discoveries about managing health, improving fitness, and improving their education. Similar methods could help community scientists understand and manage how they use digital technologies. In this paper, we introduce Conjecture, a system for coordinating N -of-one trials that can guide personal decisions about technology use and contribute to science. We describe N -of-one trials as a design challenge and present the design of the Conjecture system. We evaluate the system with a field experiment that tests folk theories about the influence of colorful screens on alleged phone addiction. We present findings on the design of N -of-one-trial systems based on submitted data, interviews, and surveys with 14 participants. Taken together, this paper introduces N -of-one trials as a fruitful direction for computer scientists designing industry-independent systems for evidence-based technology governance and accountability.

References

[1]
2018. Why We're Changing From "Citizen Science" to "Community Science" | Audubon Center at Debs Park. https://debspark.audubon.org/news/why- were-changing-citizen-science-community-science
[2]
2020. The Citizen Browser Project—Auditing the Algorithms of Disinformation – The Markup. https://themarkup.org/citizen-browser
[3]
Eytan Bakshy, Dean Eckles, and Michael S. Bernstein. 2014. Designing and Deploying Online Field Experiments. In Proceedings of the 23rd International Conference on World Wide Web (WWW ’14). ACM, New York, NY, USA, 283–292. https://doi.org/10.1145/2566486.2567967
[4]
Ben Bambery, Michael Selgelid, Charles Weijer, Julian Savulescu, and Andrew J. Pollard. 2016. Ethical Criteria for Human Challenge Studies in Infectious Diseases. Public Health Ethics 9, 1 (April 2016), 92–103. https://doi.org/10.1093/phe/phv026
[5]
Jamie Barker, Paul McCarthy, Marc Jones, and Aidan Moran. 2011. Single-case research methods in sport and exercise psychology. Routledge.
[6]
Jennifer Birch. 1997. Efficiency of the Ishihara test for identifying red-green colour deficiency. Ophthalmic and Physiological Optics 17, 5 (1997), 403–408. Publisher: Elsevier.
[7]
Deborah Blum. 2019. The Poison Squad: One Chemist's Single-Minded Crusade for Food Safety at the Turn of the Twentieth Century (illustrated edition ed.). Penguin Books, New York, New York.
[8]
Rick Bonney and Janis L. Dickinson (Eds.). 2012. Citizen science: Public participation in environmental research. Cornell University Press.
[9]
Nellie Bowles. 2018. Is the Answer to Phone Addiction a Worse Phone? The New York Times (Jan. 2018). https://www.nytimes.com/2018/01/12/technology/grayscale-phone.html
[10]
Sam Burnett and Nick Feamster. 2015. Encore: Lightweight measurement of web censorship with cross-origin requests. In Proceedings of the 2015 ACM conference on special interest group on data communication. 653–667.
[11]
Benjamin R. Cohen. 2011. Analysis as border patrol: Chemists along the boundary between pure food and real adulteration. Endeavour 35, 2-3 (2011), 66–73. Publisher: Elsevier.
[12]
Noel Coley. 2005. The fight against food adulteration. Education in Chemistry 42, 2 (2005), 46–49. Publisher: Royal Society of Chemistry.
[13]
Thomas D. Cook, Donald Thomas Campbell, and William Shadish. 2002. Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Boston, MA.
[14]
Caren B. Cooper, Janis Dickinson, Tina Phillips, and Rick Bonney. 2007. Citizen science as a tool for conservation in residential ecosystems. Ecology and society 12, 2 (2007). Publisher: JSTOR.
[15]
Nediyana Daskalova, Karthik Desingh, Alexandra Papoutsaki, Diane Schulze, Han Sha, and Jeff Huang. 2017. Lessons learned from two cohorts of personal informatics self-experiments. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1, 3 (2017), 1–22. Publisher: ACM New York, NY, USA.
[16]
Nediyana Daskalova, Eindra Kyi, Kevin Ouyang, Arthur Borem, Sally Chen, Sung Hyun Park, Nicole Nugent, and Jeff Huang. 2021. Self-E: Smartphone-Supported Guidance for Customizable Self-Experimentation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.
[17]
Thomas Dietz, Elinor Ostrom, and Paul C. Stern. 2003. The struggle to govern the commons. science 302, 5652 (2003), 1907–1912. http://www.sciencemag.org/content/302/5652/1907.short
[18]
Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2021. Fairness and Discrimination in Information Access Systems. arXiv preprint arXiv:2105.05779 (2021).
[19]
Deborah Estrin and Ida Sim. 2010. Open mHealth architecture: an engine for health care innovation. Science 330, 6005 (2010), 759–760. Publisher: American Association for the Advancement of Science.
[20]
Aleksander Fabijan, Pavel Dmitriev, Helena Holmström Olsson, and Jan Bosch. 2017. The Evolution of Continuous Experimentation in Software Product Development. In International Conference on Software Engineering (ICSE). http://exp-platform.com/Documents/2017-05%20ICSE2017_ EvolutionOfExP.pdf
[21]
Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. 2018. Uncertainty displays using quantile dotplots or cdfs improve transit decision-making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
[22]
Arturo Filasto and Jacob Appelbaum. 2012. OONI: Open Observatory of Network Interference. In FOCI.
[23]
Alan S. Gerber and Donald P. Green. 2012. Field experiments: Design, analysis, and interpretation. WW Norton.
[24]
Lorine Swainston Goodwin. 2006. The pure food, drink, and drug crusaders, 1879-1914. McFarland.
[25]
Gordon Guyatt, David Sackett, D. Wayne Taylor, John Ghong, Robin Roberts, and Stewart Pugsley. 1986. Determining optimal therapy—randomized trials in individual patients. New England Journal of Medicine 314, 14 (1986), 889–892. Publisher: Mass Medical Soc.
[26]
Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences 33, 2-3 (2010), 61–83. Publisher: Cambridge University Press.
[27]
Lancelot Hogben and M. Sim. 1953. The Self-Controlled and Self-Recorded Clinical Trial for Low-Grade Morbidity. Journal of Epidemiology & Community Health 7, 4 (Oct. 1953), 163–179. https://doi.org/10.1136/jech.7.4.163 Publisher: BMJ Publishing Group Ltd Section: Articles.
[28]
Lilly Irani and M. Silberman. 2014. From critical design to critical infrastructure: Lessons from Turkopticon. interactions 21, 4 (2014), 32–35. http://dl.acm.org/citation.cfm?id=2627392
[29]
Lilly C. Irani and M. Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611–620. http://dl.acm.org/citation.cfm?id=2470742
[30]
Niklas Johannes, Philipp K. Masur, Matti Vuorre, and Andrew K. Przybylski. 2021. How should we investigate variation in the relation between social media and well-being? (2021). Publisher: PsyArXiv.
[31]
Linda K Kaye, Amy Orben, David A Ellis, Simon C Hunter, and Stephen Houghton. 2020. The conceptual and methodological mayhem of “screen time”. International Journal of Environmental Research and Public Health 17, 10 (2020), 3661. Publisher: Multidisciplinary Digital Publishing Institute.
[32]
Rob Kitchin. 2017. Thinking critically about and researching algorithms. Information, Communication & Society 20, 1 (2017), 14–29.
[33]
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online Controlled Experiments at Large Scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’13). ACM, New York, NY, USA, 1168–1176. https://doi.org/10.1145/2487575.2488217
[34]
Geza Kovacs. 2019. HabitLab: In-the-wild Behavior Change Experiments at Scale. Ph. D. Dissertation. Stanford University.
[35]
R. L. Kravitz, N. Duan, I. Eslick, N. B. Gabler, H. C. Kaplan, and E. B. Larson. 2014. Design and implementation of N-of-1 trials: a user's guide. Agency for healthcare research and quality, US Department of Health and Human Services (2014). https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/n- 1-trials_research-2014-5.pdf
[36]
Richard L. Kravitz, Christopher H. Schmid, Maria Marois, Barth Wilsey, Deborah Ward, Ron D. Hays, Naihua Duan, Youdan Wang, Scott MacDonald, Anthony Jerant, Joseph L. Servadio, David Haddad, and Ida Sim. 2018. Effect of Mobile Device–Supported Single-Patient Multi-crossover Trials on Treatment of Chronic Musculoskeletal Pain: A Randomized Clinical Trial. JAMA Internal Medicine 178, 10 (Oct. 2018), 1368–1377. https://doi.org/10.1001/jamainternmed.2018.3981
[37]
Victor Le Pochat, Laura Edelson, Tom Van Goethem, Wouter Joosen, Damon McCoy, and Tobias Lauinger. 2022. An Audit of Facebook's Political Ad Policy Enforcement. In Proceedings of the 31st USENIX Security Symposium. USENIX Association.
[38]
Max Liboiron. 2021. Pollution is colonialism. Duke University Press.
[39]
Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI?. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[40]
J. Matias, Amy Johnson, Whitney Erin Boesel, Brian Keegan, Jaclyn Friedman, and Charlie DeTar. 2015. Reporting, reviewing, and responding to harassment on Twitter. Available at SSRN 2602018 (2015).
[41]
J. Nathan Matias. 2016. Going dark: Social factors in collective action against platform operators in the Reddit blackout. In Proceedings of the 2016 CHI conference on human factors in computing systems. 1138–1151.
[42]
J. Nathan Matias, Austin Hounsel, and Nick Feamster. 2021. Software-Supported Audits of Decision-Making Systems: Testing Google and Facebook's Political Advertising Policies. arXiv preprint arXiv:2103.00064 (2021).
[43]
J. Nathan Matias and Merry Mou. 2018. CivilServant: Community-led experiments in platform governance. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
[44]
E. Scott Maynes. 1984. Test and Protest: The Influence of Consumers Union. JSTOR.
[45]
Jeremy B. Merrill. 2018. What We Learned From Collecting 100,000 Targeted Facebook Ads. ProPublica (Dec. 2018). https://www.propublica.org/article/facebook-political-ad-collector-targeted-ads-what-we-learned?token=1fnMWV3DOiq0OUA8y5HDwG7SC_9TWm2G
[46]
Betsy Mikel. 2018. Former Google Designer Says 1 Simple Trick Can Curb Your Smartphone Addiction. Inc. (2018). https://www.inc.com/betsy-mikel/former-google-designer-says-1-simple-trick-can-curb-your-smartphone-addiction.html
[47]
R. D. Mirza, S. Punja, S. Vohra, and G. Guyatt. 2017. The history and development of N-of-1 trials. Journal of the Royal Society of Medicine 110, 8 (2017), 330–340. Publisher: SAGE Publications Sage UK: London, England.
[48]
Jane Nikles, Patrick Onghena, Johan WS Vlaeyen, Rikard K. Wicksell, Laura E. Simons, James M. McGree, and Suzanne McDonald. 2021. Establishment of an International Collaborative Network for N-of-1 Trials and Single-Case Designs. Contemporary clinical trials communications 23 (2021), 100826. Publisher: Elsevier.
[49]
Candice Odgers. 2018. Smartphones are bad for some teens, not all. Nature 554, 7693 (Feb. 2018), 432–434. https://doi.org/10.1038/d41586-018-02109-8 Bandiera_abtest: a Cg_type: Comment Number: 7693 Publisher: Nature Publishing Group Subject_term: Technology, Depression, Society.
[50]
Amy Orben. 2020. The Sisyphean cycle of technology panics. Perspectives on Psychological Science 15, 5 (2020), 1143–1157. Publisher: SAGE Publications Sage CA: Los Angeles, CA.
[51]
Amy Orben. 2020. Teenagers, screens and social media: a narrative review of reviews and key studies. Social psychiatry and psychiatric epidemiology 55, 4 (2020), 407–414. Publisher: Springer.
[52]
Amy Orben and Andrew K. Przybylski. 2019. The association between adolescent well-being and digital technology use. Nature Human Behaviour 3, 2 (2019), 173–182. Publisher: Nature Publishing Group.
[53]
Vineet Pandey, Amnon Amir, Justine Debelius, Embriette R. Hyde, Tomasz Kosciolek, Rob Knight, and Scott Klemmer. 2017. Gut instinct: Creating scientific theories with online learners. In Proceedings of the 2017 CHI conference on human factors in computing systems. 6825–6836.
[54]
Andrew K. Przybylski, Amy Orben, and Netta Weinstein. 2020. How much is too much? Examining the relationship between digital screen engagement and psychosocial functioning in a confirmatory cohort study. Journal of the American Academy of Child & Adolescent Psychiatry 59, 9 (2020), 1080–1088. Publisher: Elsevier.
[55]
Pablo Rey-Mazón, Hagit Keysar, Shannon Dosemagen, Catherine D'Ignazio, and Don Blair. 2018. Public lab: Community-based approaches to urban and environmental health and justice. Science and Engineering Ethics 24, 3 (2018), 971–997. Publisher: Springer.
[56]
Steven Shapin and Simon Schaffer. 2011. Leviathan and the air-pump. Princeton University Press.
[57]
Jennifer L. Shirk, Heidi L. Ballard, Candie C. Wilderman, Tina Phillips, Andrea Wiggins, Rebecca Jordan, Ellen McCallie, Matthew Minarchek, Bruce V. Lewenstein, and Marianne E. Krasny. 2012. Public participation in scientific research: a framework for deliberate design. Ecology and society 17, 2 (2012). Publisher: JSTOR.
[58]
Aaron Smith. 2018. How Americans View Tech Companies. Technical Report. Pew Research Center. https://www.pewresearch.org/internet/2018/06/ 28/public-attitudes-toward-technology-companies/
[59]
Justin D. Smith. 2012. Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods 17, 4 (2012), 510–550. https://doi.org/10.1037/a0029312 Place: US Publisher: American Psychological Association.
[60]
James Tabery. 2011. Commentary: Hogben vs the Tyranny of Averages. International Journal of Epidemiology 40, 6 (Dec. 2011), 1454–1458. https://doi.org/10.1093/ije/dyr027
[61]
Kaveh Waddell. 2021. California's New Privacy Rights Are Tough to Use, Consumer Reports Study Finds. Consumer Reports (March 2021). https://www.consumerreports.org/privacy/californias-new-privacy-rights-are-tough-to-use-a1497188573/
[62]
Meredith Whittaker. 2021. The steep cost of capture. Interactions 28, 6 (2021), 50–55. Publisher: ACM New York, NY, USA.

Index Terms

  1. Testing Concerns about Technology's Behavioral Impacts with N-of-one Trials
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Other conferences
          FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency
          June 2022
          2351 pages
          ISBN:9781450393522
          DOI:10.1145/3531146
          This work is licensed under a Creative Commons Attribution International 4.0 License.

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 20 June 2022

          Check for updates

          Author Tags

          1. community science
          2. consumer protection
          3. n-of-one trials
          4. phone addiction
          5. research software

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          FAccT '22
          Sponsor:

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • 0
            Total Citations
          • 174
            Total Downloads
          • Downloads (Last 12 months)39
          • Downloads (Last 6 weeks)5
          Reflects downloads up to 11 Feb 2025

          Other Metrics

          Citations

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media