Abstract
With the aim of assessing the extent of the differences in the context of Italian educational system, the paper applies multilevel modeling to a new administrative dataset, containing detailed information for more than 500,000 students at grade 6 in the year 2011/2012, provided by the Italian Institute for the Evaluation of Educational System. Data are grouped by classes, schools and geographical areas. Different models for each area are fitted, in order to properly address the heteroscedasticity of the phenomenon. The results show that it is possible to estimate statistically significant “school effects”, i.e., the positive/negative association of attending a specific school and the student’s test score, after a case-mix adjustment. Therefore, the paper’s most important message is that school effects are different in terms of magnitude and types in the three geographical macro areas (Northern, Central and Southern Italy) and are dependent on specific students’ and schools’ characteristics.
Similar content being viewed by others
Notes
Indeed, we are aware that the quality of schools is characterized by a wider set of dimensions (such as non-cognitive achievements, school climate, cognitive test scores in additional subjects, teachers’ satisfaction, etc.). Thus, while the formulation “schools with higher tests” is somehow less intuitive, it is definitely more adequate for describing the real aspect that we are monitoring in this research. However to ease notation we use the word quality in the text.
A recent paper (Agasisti and Falzetti 2013) also showed that schools in the South practice a within-school segmentation (e.g., between classes) stronger than their counterparts in the North. In this paper, we explore between-schools differences, but we are aware that similar mechanisms (that is to say, differential effects on achievement between classes of the same school) are also operating within schools.
An interesting example of study of longitudinal data at student level in Italy is presented in Bartolucci et al. (2011).
While the distribution of \(b_j\) is checked later, its characteristics of being exogenous (in the sense of Steele et al. 2007) is not empirically verifiable in this setting.
Let us remind that the number of schools is much lower in Central Italy because the administrative classification of North, Centre and South used does not separate the country in three equal parts; instead, Central Italy only includes four regions out of twenty (Tuscany, Lazio, Marche, Umbria).
References
Agasisti T, Vittadini G (2012) Regional economic disparities as determinants of students’ achievement in Italy. Res Appl Econ 4(1):33–53
Agasisti T, Falzetti P (2013) Between-classes sorting within schools and test scores: an empirical analysis of the Italian junior secondary schools. INVALSI working paper no. 20/2013
Bartolucci F, Pennoni F, Vittadini G (2011) Assessment of school performance through a multilevel latent Markov Rasch model. J Educ Behav Stat 36:491–522
Bertoni M, Brunello G, Rocco L (2013) When the cat is near, the mice won’t play: the effect of external examiners in the Italian schools. J Public Econ 104(1):65–77
Bowers AJ, Sprott R (2012) Examining the multiple trajectories associated with dropping out of high school: a growth mixture model analysis. J Educ Res 105(3):176–195
Bratti M, Checchi D, Filippin D (2007) Geographical differences in Italian students’ mathematical competencies: evidence from PISA 2003. G Econ Ann Econ 66(3):299–333
Brunello G, Checchi D (2005) School quality and family background in Italy. Econ Educ Rev 24(5):563–577
Cipollone P, Montanaro P, Sestito P (2010) Value-added measures in Italian high schools: problems and findings. G Econ Ann Econ 69(2):81–114
Conti E, Duranti S, Maitino ML, Rampichini C, Sciclone N (2013) The future has early roots. Learning outcomes and school’s effectiveness in Tuscany’s primary education system. Workshop youth and their future: work, education and health, October 17th–18th, 2013, University of Salerno
De Simone G, Gavosto A (2013) Patterns of value-added creation in the transition from primary to lower secondary education in Italy. Paper presented at the XXVIII national conference of labour economics, Rome, September 2013
Di Liberto A (2008) Education and Italian regional development. Econ Educ Rev 27(1):94–107
Goldstein H (2011) Multilevel statistical models, 4th edn. Arnold, London
Goldstein H, Sammons P (1997) The influence of secondary and junior schools on sixteen year examination performance: a cross-classified multilevel analysis. Sch Eff Sch Improv 8:219–230
Goldstein H, Rasbash J, Yang M, Woodhouse G, Pan H, Nuttall D, Thomas S (1993) A multilevel analysis of school examination results. Oxf Rev Educ 19:425–433
Goldstein H, Browne W, Rasbash J (2002) Partitioning variation in multilevel models. Underst Stat 1(4):223–231
Goldstein H, Carpenter JR, Browne WJ (2014) Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms. J R Stat Soc Ser A 177:553–564
Gorard S, Hordosy R, Siddiqui N (2012) How unstable are ’school effects’ assessed by a value-added technique? Int Educ Stud 6(1):1
Grieco N, Ieva F, Paganoni AM (2012) Performance assessment using mixed effects models: a case study on coronary patient care. IMA J Manag Math 23(2):117–131
Haveman R, Wolfe B (1995) The determinants of children’s attainments: a review of methods and findings. J Econ Lit 33(4):1829–1878
Heckman JJ, Kautz T (2012) Hard evidence on soft skills. Labour Econ 19(4):451–464
Ieva F, Paganoni AM (2015) Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag Sci 18:166–172
Leckie G, Goldstein H (2009) The limitations of using school league tables to inform school choice. J R Stat Soc Ser A 172(4):835–851
Little R, Rubin D (2002) Statistical analysis with missing data. Wiley, London
Masci C, Ieva F, Agasisti T, Paganoni AM (2016) Does class matter more than school? Evidence from a multilevel statistical analysis on Italian junior secondary school students. Socio-Econ Plan Sci 54:47–57
Mohammed MA, Deeks JJ (2008) In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann Thorac Surg 86:348
OECD (2010) PISA 2009 results: what students know and can do. OECD, Paris
OECD (2013) PISA 2012 results in focus: what 15-year-olds know and what they can do with what they know. OECD, Paris
Paccagnella M, Sestito P (2014) School cheating and social capital. Educ Econ 22(4):367–388
Perry L, McConney A (2010) Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. Teach Coll Rec 112(4):7–8
Pigott T (2001) A review of methods for missing data. Educ Res Eval 7(4):353–383
Pinheiro J, Bates D, DebRoy S, Sarkar D, The R Development Core Team (2013) Nlme: linear and nonlinear mixed effects models. R package version 3.1-111
Plewis I (2011) Contextual variations in ethnic group differences in educational attainments. J R Stat Soc Ser A 174(2):419–437
R Development Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. [Online] http://www.R-project.org
Rasbash J, Leckie G, Pillinger R, Jenkins J (2011) Children’s educational progress: partitioning family, school and area effects. J R Stat Soc Ser A 173(3):419–437
Snijders T, Bosker R (2012) Multilevel analysis: an introduction to basic and advanced multilevel modeling, 2nd edn. Sage, London
Spiegelhalter DJ (2002) Funnel plots for institutional comparisons (letter). Qual Saf Health Care 11:390–391
Spiegelhalter DJ (2005) Funnel plots for comparing institutional performance. Stat Med 24:1185–1202
Steele F, Vignoles A, Jenkins A (2007) The effect of school resources on pupil attainment: a multilevel simultaneous equation modelling approach. J R Stat Soc Ser A 170(3):801–824
Struyf A, Hubert M, Rousseeuw PJ (1996) Clustering in an object-oriented environment. J Stat Softw 1(4):1–30
Willms JD, Raudenbush SW (1989) A longitudinal hierarchical linear model for estimating school effects and their stability. J Educ Meas 26(3):209–232
Acknowledgments
This work is within FARB—Public Management Research: Health and Education Systems Assessment, funded by Politecnico di Milano. The authors are grateful to Invalsi for having provided the original dataset, and P. Falzetti for the statistical assistance in building the specific database used in this paper.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
As suggested by an anonymous referee, we report the estimates of model (1), fitted discarding the CMS5, for each geographical areas, in two cases: the entire database and the reduced one. We report only the estimates of fixed effects to ease comparison between the two cases (see Table 7). This enforces us in relying the listwise deletion approach, despite the discrepancies highlighted in Sect. 2.2.
There are no significant differences in the two cases, so we can claim that the large proportion of missing data is not a big problem.
Rights and permissions
About this article
Cite this article
Agasisti, T., Ieva, F. & Paganoni, A.M. Heterogeneity, school-effects and the North/South achievement gap in Italian secondary education: evidence from a three-level mixed model. Stat Methods Appl 26, 157–180 (2017). https://doi.org/10.1007/s10260-016-0363-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10260-016-0363-x