Skip to main content
SpringerLink
Log in

Comparison of human and machine-based educational standard assignment networks

  • Published:
International Journal on Digital Libraries Aims and scope Submit manuscript

Abstract

Increasing availability of digital libraries of K-12 educational resources, coupled with an increased emphasis on standard-based teaching necessitates assignment of the standards to those resources. Since manual assignment is a laborious and ongoing task, machine-based standard assignment tools have been under development for some time. Unfortunately, data on the performance of these machine-based classifiers are mostly lacking. In this article, we explore network modeling and layout to gain insight into the differences between assignments made by catalogers and those by the well-known Content Assignment Tool (CAT) machine-based classifier. To build the standard assignment networks, we define standards to be linked if they are jointly assigned to a learning resource. Comparative analysis of the topology and layout of the networks shows that whereas the cataloger-based network reflects the underlying curriculum, i.e., clusters of standards separate along lines of lesson content and pedagogical principles, the machine-based network lacks these relationships. This shortcoming is partially traced back to the machine classifier’s difficulties in recognizing standards that express ways and means of conducting science.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson, N.: Governors, State School Superintendents Propose Common Academic Standards. Education Brief. The Washington Post, March 10, 2010. http://www.socialstudies.org/governors_state_school_superintendents_propose_common_academic_standards

  2. Baeza-Yates R., Ribeiro-Neto B.: Modern Information Retrieval. ACM Press, Harlow (1999)

    Google Scholar 

  3. Bar-Ilan J., Keenoy K., Yaari E., Levene M.: User rankings of search engine results. J. Am. Soc. Inf. Sci. Technol. 58, 1254–1266 (2007)

    Article  Google Scholar 

  4. Blumenthal, D.: Evaluation Approaches for a K-12 Digital Library Collection. WGBH Educational Foundation, Boston, MA (2003). http://eduimpact.comm.nsdl.org/evalworkshop/blumenthal.doc

  5. Bologna Secretariat.: Welcome to the website of the European Higher Education Area: The Official Bologna Process Website 2007–2010. Benelux Bologna Secretariat, Brussels (2010). http://www.ond.vlaanderen.be/hogeronderwijs/bologna/

  6. Breen, C.: Review: NetTrekker. MacWorld.com (2008). http://www.macworld.com/article/134787/2008/08/nettrekker.html

  7. BrightPlanet.com.: Brief on the Deep Web—What is Google Missing? (2009). http://www.brightplanet.com/images/uploads/12550157301-Brief%20on%20The%20Deep%20Web%20-%20What%20is%20Google%20Missing.pdf

  8. Cunningham, C.M., Hester, K.: Engineering is elementary: an engineering and technology curriculum for children. In: Proceedings of the ASEE Annual Conference, Honolulu, HI. ASEE, Washington DC (2007). http://www.starwars.mos.org/eie/pdf/research/asee_2007_dev_paper.pdf

  9. De Nooy W., Mrvar A., Batagelj V.: Exploratory social network analysis with Pajek. Cambridge University Press, Cambridge (2005)

    Google Scholar 

  10. Devaul H., Diekema A.R., Ostwald J.: Computer-assisted assignment of educational standards using natural language processing. J. Am. Inform. Sci. Technol. 62, 395–405 (2011)

    Article  Google Scholar 

  11. Diekema A.R.: Implications and challenges of educational standards metadata. J Libr Metadata 9, 239–251 (2009)

    Article  Google Scholar 

  12. Diekema, A.R., Chen, J.: Experimenting with the automatic assignment of educational standards to digital library content. In: Proceedings of the 5th ASM/IEEE Joint Conference on Digital Libraries. ACM, New York, NY (2005)

  13. Diekema, A.R., Yilmazel, O., Bailey, J., Harwell, S.C., Liddy E.D.: Standards alignment for metadata assignment. In: Proceedings of the Joint Conference of Digital Libraries (JCDL’07), Vancouver, BC. IEEE-ACM. New York, NY (2007)

  14. Felix, K.: In the Spotlight. NetTrekker. (2004). http://www.infotoday.com/MMSchools/jan04/felix.shtml

  15. Fruchterman T.M.J., Reingold E.M.: Graph drawing by force-directed placement. Software, Practice and Experience 21, 1129–1164 (1991)

    Article  Google Scholar 

  16. Gateway NSDL:ASN Achievement Standards Network. (2007). http://www.thegateway.org/asn

  17. Hillman D.I.: Metadata quality: from evaluation to augmentation. Cataloging and Classification Quarterly 46, 65–80 (2008)

    Article  Google Scholar 

  18. Jay, M., Longdon, D.: Death, Taxes and Correlations: A Primer on the State of Correlation in the K-12 Education. Upgrade SIIA. pp. 20–21 (2003)

  19. Johnson A.: Summative Evaluation Teachers’ Domain. Report submitted to WGBH Educational Productions. WGBH, Boston, MA (2003)

    Google Scholar 

  20. Kamada T., Kawai S.: An algorithm for drawing general undirected graphs. Inform. Process. Lett. 31, 7–15 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  21. Kendall J.S., Marzano R.J.: Content Knowledge. A Compendium of Standards and Benchmarks for K-12 Education, 2nd edn. MCREL, Aurora, CO (1997)

    Google Scholar 

  22. Kim G.: Relationship between index term specificity and relevance judgment. Inform. Process. Manage. 42, 1218–1229 (2006)

    Article  Google Scholar 

  23. Kulm G., Grier L.: Mathematics Curriculum Materials Analysis Reliability Study AAAS—Project 2061. AAAS, Washington, DC (1998)

    Google Scholar 

  24. Lachapelle, C.P., Cunningham, C.M.: Engineering is elementary: children’s changing understandings of science and engineering. In: Proceedings of the ASEE Annual Conference, Honolulu, HI. ASEE, Washington, DC (2007). http://www.museumofscience.org/eie/pdf/research/asee_2007_students_understandings.pdf

  25. Lyman, P., Varian, H.R., Swearingen, K., et al.: How Much Information. University of California, Berkeley (2003). http://www2.sims.berkeley.edu/research/projects/how-much-info-2003

  26. Marshall B., Reitsma R., Cyr, M. Standards or semantics for curriculum search? In: Proceedings of the 7th Joint ACM/IEEE Conference on Digital Libraries (JCDL 2007), Vancouver, BC (2007)

  27. Marshall, B., Reitsma, R.: World vs. Method: Educational Standard Formulation Impacts Document Retrieval. Proceedings of the 11th Joint ACM/IEEE Conference on Digital Libraries (JCDL 2011). Ottawa, ON (2011)

  28. McAnear, A.: What Does Globalization Mean for Education? Learning and Leading with Technology, vol. 5 (2007).http://www.iste.org/Content/NavigationMenu/Publications/LL/LLIssues/Volume_34_2006_2007_/February_5_/34505m.pdf

  29. National Governors Association Center for Best Practices, Council of Chief State School Officers: Common Core State Standards Initiative (2010). http://www.corestandards.org

  30. Porter, A., Polikoff, M., Smithson, J.: Is There a De Facto National Curriculum: Evidence from State Content Standards? Workshop on Assessing the Role of K-12 Academic Standards in States. National Research Council, Center for Education, Washington, DC. http://www7.nationalacademies.org/cfe/Porter_Smithson%20State%20Standards%20Paper_Tables.pdf. Accessed 18 Jan 2008

  31. Reitsma, R., Marshall, B., Dalton, M., Cyr, M.: Exploring educational standard alignment. In search of ’relevance.’ In: Proceedings of the Joint Conference on Digital Libraries (JCDL’08). IEEE-ACM, Pittsburgh, PA (2008)

  32. Reitsma R., Marshall B., Zarske M.: Aspects of ’relevance’ in the alignment of curriculum with educational standards. Inform. Process. Manage. 46, 362–376 (2010)

    Article  Google Scholar 

  33. Roseman J.E., Stern L., Koppal M.: A method for analyzing the coherence of high school biology textbooks. J. Res. Sci. Teach. 47, 47–70 (2009)

    Article  Google Scholar 

  34. Salton G., Yang C.S., Wong A.: A vector space model for automatic indexing. Commun. ACM 18, 613–620 (1975)

    Article  MATH  Google Scholar 

  35. Saracevic T.: Relevance: a review of the literature and a framework for thinking on the notion in information science. J. Am. Soc. Inf. Sci. 26, 321–343 (1975)

    Article  Google Scholar 

  36. Saracevic T.: Relevance: a review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance. J. Am. Soc. Inf. Sci. Technol. 58, 1915–1933 (2007)

    Article  Google Scholar 

  37. Saracevic T.: Relevance: a review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. J. Am. Soc. Inf. Sci. Technol. 58, 2126–2144 (2007)

    Article  Google Scholar 

  38. Saylor J.M., Minton-Morris C.: The national science digital library: an update on systems, services and collection development. Sci. Technol. Libr. 26, 61–78 (2006)

    Article  Google Scholar 

  39. Sullivan, J.F., Cyr, M.N., Mooney, M.A., Reitsma, R.F., Shaw, N.C., Zarske, M.S., Klenk, P.A.: The TeachEngineering Digital Library: engineering comes alive for K-12 youth. In: Proceedings of the ASEE Annual Conference, Portland, OR. ASEE, Washington, DC (2005). http://www.teachengineering.org/documents/ASEE%202005-851_TE_Final.pdf

  40. Sumner T., Ahmad F., Bhushan S., Gu Q., Molina F., Stedman W., Wright M., Davis L., Janée G.: Linking learning goals and educational resources through interactive concept map visualizations. Intern. J. Digit. Libr. 5, 18–24 (2005)

    Google Scholar 

  41. Sutton, S.A., Golder, D.: Achievement Standards Network (ASN): an application profile for mapping K-12 educational resources to achievement standards. In: Proceedings of the 2008 International Conference on Dublin Core and Metadata Applications, Humboldt University, Berlin, Germany, pp. 69–79 (2008)

  42. USGPO (United States Printing Office): Public Law 107. An Act to Close the Achievement Gap with Accountability, Flexibility, and Choice, so that No Child Is Left Behind (2001). http://www.gpo.gov/fdsys/pkg/PLAW-107publ110/content-detail.html

  43. Wallis, C., Steptoe, S.: How to Bring Our Schools Out of the 20th Century. Times, December 10 (2006). http://lutheransouth-hs.org/ourpages/publications/Article%20Time%20-%20How%20to%20Bring%20Schools%20out%20of%2020th%20century.pdf

  44. Voorhees, E.M.: Variations in relevance judgments and the measurement of retrieval effectiveness. In: Croft, W.B., Moffat, A., van Rijsbergen, C.J. (eds.) Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York, NY, pp. 315–324 (1998)

  45. Widdows: Geometry and Meaning. CSLI Publication, Stanford, CA (2004)

  46. Yilmazel, O., Balasubramanian, N., Harwell, S.C., Bailey, J., Diekema, A.R., Liddy, E.D.: Text categorization for aligning educational standards. In: Proceedings of the 40th Hawaii International Conference of Systems Sciences. IEEE. New York, NY (2007)

  47. Yilmazel, O., Ingersoll, G., Liddy, E.D.: Finding questions to your answers. In: IEEE 23rd International Conference on Data Engineering (ICDE 2007) Istanbul, Turkey. CNLP, Syracuse, NY (2007)

  48. Zia, L.: The NSF National Science, Technology, Engineering and Mathematics Education Digital Library (NSDL) Program. New Projects in Fiscal Year 2002. D-Lib Magazine, vol. 8 (2002). http://www.dlib.org/dlib/november02/zia/11zia.html

Download references

Author information

Authors and Affiliations

  1. College of Business, Oregon State University, Corvallis, OR, USA

    René F. Reitsma

  2. Instructional Technology and Learning Sciences, Utah State University, Logan, UT, USA

    Anne R. Diekema

Authors
  1. René F. Reitsma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anne R. Diekema
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to René F. Reitsma.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reitsma, R.F., Diekema, A.R. Comparison of human and machine-based educational standard assignment networks. Int J Digit Libr 11, 209–223 (2010). https://doi.org/10.1007/s00799-011-0074-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-011-0074-8

Keywords

Search

Navigation