Abstract
The basic goal of quantitative ethnography (QE) is to use statistics to warrant theoretical saturation. In QE studies that use epistemic network analysis (ENA), statistical comparisons between two or more samples are often used as warrants. However, no standard quantitative techniques have been developed to provide warrants absent of differences between samples—e.g., for the data overall or for particular individuals in the data. In this paper, I introduce the expected value test (EVT), a technique for finding statistical warrants for single samples in the context of ENA-based QE analyses. Building on the concepts of agreement due to chance and randomization, the EVT generates a distribution of networks whose edge weights approximate the expected value of the edge weights due to chance. Using these distributions affords tests of statistical significance been observed networks and chance-based networks. These tests provide warrants that connections observed qualitatively and measured by ENA are theoretically saturated.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Crucially, though less important to the arguments in this paper, researchers also close the interpretive loop—that is, they judge whether their qualitative interpretations align with the quantitative results.
- 2.
Another way in which Cohen’s kappa is meaningful is the extent to which it generalizes, which is addressed by Shaffer’s rho.
- 3.
In the coded data examined by Csanadi and colleagues, each line could be coded for only one code, therefore this issue did not apply to their data. However, in other contexts, multiple codes may be assigned to the same line.
- 4.
Typically, ENA also performs a dimensional reduction on the collection of normalized and centered networks via singular value decomposition. However, the analysis presented here focused on the complete networks rather than their projections in a low dimensional space.
- 5.
In this sense, the EVT is essentially an empirical version of Hoetelling’s T test. However, for that test, the squared Mahalanobis distance is used in place of the Euclidean distance and a normal distribution is assumed.
- 6.
Because a dimensional reduction was not used to project the networks in a lowdimensional space, I have positioned the nodes of these networks in a circle to simplify their presentation.
References
Bressler, D.M., Bodzin, A.M., Eagan, B., Tabatabai, S.: Using epistemic networkanalysis to examine discourse and scientific practice during a collaborative game. J. Sci. Educ. Technol. 28(5), 553–566 (2019)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)
Csanadi, A., Eagan, B., Kollar, I., Shaffer, D.W., Fischer, F.: When coding-andcounting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. Int. J. Comput.-Support. Collab. Learn. 13(4), 419–438 (2018)
Glaser, B., Strauss, A.: The Discovery of Grounded Theory. Aldine, Chicago (1967)
Hod, Y., Katz, S., Eagan, B.: Refining qualitative ethnographies using epistemic network analysis: a study of socioemotional learning dimensions in a humanistic knowledge building community. Comput. Educ. 156, 103943 (2020)
Johnston, J.H., Poirier, J., Smith-Jentsch, K.A.: Decision making under stress: creating a research methodology. In: Cannon-Bowers, J.A., Salas, E. (eds.) Making Decisions Under Stress: Implications for Individual and Team Training, pp. 39–59. American Psychological Association, Washington D.C (1998)
Marquart, C., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., Shaffer, D.W.: rENA: Epistemic Network Analysis (Version 0.2.1.2) (2018). https://cran.rstudio.com/web/packages/rENA/index.html
Shaffer, D.W.: Quantitative Ethnography. Cathcart Press, Madison (2017)
Shaffer, D.W., Collier, W., Ruis, A.R.: A tutorial on epistemic network analysis: analyzing the structure of connections in cognitive, social, and interaction data. J. Learn. Anal. 3(3), 9–45 (2016)
Ruis, A.R., Rosser, A.A., Quandt-Walle, C., Nathwani, J.N., Shaffer, D.W., Pugh, C.M.: The hands and head of a surgeon: modeling operative competency with multimodal epistemic network analysis. Am. J. Surg. 216(5), 835–840 (2018)
Swiecki, Z., Ruis, A.R., Farrell, C., Shaffer, D.W.: Assessing individual contributions to collaborative problem solving: a network analysis approach. Comput. Hum. Behav. 104, 105876 (2020)
Wu, B., Hu, Y., Ruis, A.R., Wang, M.: Analysing computational thinking in collaborative programming: a quantitative ethnography approach. J. Comput. Assist. Learn. 35, 421–434 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Swiecki, Z. (2022). The Expected Value Test: A New Statistical Warrant for Theoretical Saturation. In: Wasson, B., Zörgő, S. (eds) Advances in Quantitative Ethnography. ICQE 2021. Communications in Computer and Information Science, vol 1522. Springer, Cham. https://doi.org/10.1007/978-3-030-93859-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-93859-8_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93858-1
Online ISBN: 978-3-030-93859-8
eBook Packages: Computer ScienceComputer Science (R0)