Value-added in Higher Education
a new methodology for undergraduate programs in Brazil
Palavras-chave:
Value-added model, Learning outcomes, Quality measuresResumo
Among the quality indicators released by the Brazilian Higher Education Assessment System (Sinaes), the Indicator “Difference between Observed and Expected Performance” (IDD) has the purpose of measuring the contribution of the course to student achievement during undergraduate programs. The research presented here offers a new methodology for calculating the IDD (Model IDD-VDCF), examining the philosophical and statistical underpinnings of quality measures, focusing on those that capture the value-added as a student achievement growth. The survey included a sample of 30,668 students, from 911 accounting undergraduate programs in Brazil. The insertion of control variables (at the student and at the institution level) reduced the bias of the IDD estimate associated with the student's selection in specific Accounting Sciences courses. The results call attention to the need to consider the students' learning context when one wants to compare the performance between institutions based on standardized tests. The major contribution of this work is the development of a measure that disentangles more fully what the contribution of program is to student learning, and what merely is a reflection of the capacity that a student brought to the program.
Downloads
Referências
AKAIKE, H. A new look at the statistical model identification, IEEE Transactions on Automatic Control, New Jersey, v. 19, n. 6, p. 716-723, 1974. Available at: https://ieeexplore.ieee.org/document/1100705. Accessed on: 3 Aug. 2019.
ALEXANDER, Nicola; JANG, Sung Tae; KANKANE, Shipi. The performance cycle: the association between student achievement and state policies tying together teacher performance, student achievement, and accountability. American Journal of Education, Chicago, v. 123, n. 3, p. 413-446, 2017. Available at: https://doi.org/10.1086/691229. Accessed on: 9 Oct. 2019.
ALEXANDER, Nicola; JANG, Sung Tae. Expenditures on the professional development of teachers: The case of Minnesota. Journal of Education Finance, Baltimore, v. 44, n. 4, p. 385-404, 2019. Available at: https://www.muse.jhu.edu/article/738161. Accessed on: 9 Oct. 2019.
BAILEY, Thomas; XU, Di. Input-adjusted graduation rates and college accountability: what is known from twenty years of research? Context for Success, Austin, 2012. Available at: http://www.hcmstrategists.com/contextforsuccess/papers/LIT_REVIEW.pdf. Accessed on: 10 Sep. 2020.
BOGOYA, José Daniel; BOGOYA, Johan Manuel. An academic value-added mathematical model for higher education in Colombia. Ingeniería E Investigación, Bogotá, v. 33, n. 2, p. 76-81, 2013. Available at: https://revistas.unal.edu.co/index.php/ingeinv/article/view/39521/42363. Accessed on: 19 Aug. 2019.
BRASIL. Lei nº 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências. Brasília, 2004. Available at: http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm. Accessed on: 10 Sep. 2020.
CANZIANI, Alex et al. (org.). Financiamento da educação superior no Brasil: impasses e perspectivas. Brasília: Câmara dos Deputados, 2018. Available at: https://www2.camara.leg.br/a-camara/estruturaadm/altosestudos/pdf/financiamento-da-educacao-superior-no-brasil-impasses-e-perspectivas. Accessed on: 10 Sep. 2020.
CUNHA, Jesse; MILLER, Trey. Measuring value-added in higher education: possibilities and limitations in the use of administrative data. Economics of Education Review, Amsterdam, v. 42, p. 64-77, 2014. Available at: https://doi.org/10.1016/j.econedurev.2014.06.001. Accessed on: 10 Sep. 2019.
FERNANDES, V. D. C. et al. ENADE: Uma análise sobre os determinantes do desempenho acadêmico dos estudantes de Ciências Contábeis desde a sua primeira edição. In: ENCONTRO NACIONAL DA ANPAD, 42, 2018, Curitiba. Anais [...]. Curitiba: ANPED, 2018. p. 1-16.
FERNANDES, V. D. C.; MIRANDA, G. J.; ALEXANDER, N. Value-added measures in higher education: a historical contextualization of Brazilian experiences. Revista Brasileira de Estudos Pedagógicos, Brasília, v. 101, n. 259, p. 691-720, 2020. Available at: https://doi.org/10.24109/2176-6681.rbep.101i259.4469. Accessed on: 10 Feb. 2021.
FRENETTE, M.; FRANK, Kristyn; DENG, Zechuan. School closures and the online preparedness of children during COVID-19 Pandemic. Economics Insight: Statistics Canada, Ottawa, n. 103, 2020. Available at: https://files.eric.ed.gov/fulltext/ED605398.pdf Accessed on: 18 Jun. 2020.
HAECK, C.; LEFEBVRE, P. Pandemic school closures may increase inequality in test scores. Canadian Public Policy, Toronto, v. 26, n.1, p. 82-87, jul. 2020. Available at: https://doi.org/10.3138/cpp.2020-055. Accessed on: 18 Nov. 2020.
HAIR, Joseph et al. Multivariate data analysis. 7. ed. New Jersey: Prentice Hall, 2010.
INEP - INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA. NOTA TÉCNICA Nº 33/2017/CGCQES/DAES. Brasília: Inep, 2017. Available at: http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_tecnica_n33_2017_cgcqes_daes_calculo_idd.pdf. Accessed on: 10 Aug. 2020.
INEP - INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA. ENADE. Brasília: Inep, 2018. Available at: http://portal.inep.gov.br/web/guest/enade. Accessed on: 10 ago. 2020.
INEP - INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA. Censo da educação superior: notas estatísticas 2018. Brasília: Inep, 2019. Available at: http://download.inep.gov.br/educacao_superior/censo_superior/documentos/2019/censo_da_educacao_superior_2018-notas_estatisticas.pdf. Accessed on: 10 ago. 2020.
JOHNES, Jill. University rankings: what do they really show? Scientometrics, Budapest, v. 115, n. 1, p. 585-606, 2018. Available at: https://doi.org/10.1007/s11192-018-2666-1. Accessed on: 19 Sep. 2019.
KIM, Hoonho; LALANCETTE, Diane. Literature review on the value-added measurement in higher education. Paris: OECD. 2013. Available at: https://www.oecd.org/education/skills-beyond-school/Litterature%20Review%20VAM.pdf. Accessed on: 19 Apr. 2020.
LAROS, Jacob Arie; MARCIANO, João Luiza Pereira. Análise multinível aplicada aos dados do NELS:88. Estudos em Avaliação Educacional, São Paulo, v. 19, n. 40, p. 263-278, 2008. Available at: https://www.fcc.org.br/pesquisa/publicacoes/eae/arquivos/1440/1440.pdf Accessed on: 24 Nov. 2019.
LIU, Lydia. Measuring value-added in higher education: conditions and caveats - results from using the Measure of Academic Proficiency and Progress (MAPPTM). Assessment and Evaluation in Higher Education, London, v.36, n. 1, p. 81-94, 2011a. Available at: https://doi.org/10.1080/02602930903197917. Accessed on: 12 May 2019.
LIU, Lydia. Value-added assessment in higher education: a comparison of two methods. Higher Education, Berlim, v. 61, p. 445-461, 2011b. Available at: http://dx.doi.org/10.1007/s10734-010-9340-8. Accessed on: 12 May 2019.
MELGUIZO, Tatiana et al. The methodological challenges of measuring student learning, degree attainment, and early labor market outcomes in higher education. Journal of Research on Educational Effectiveness, London, v. 27, n.2, p. 424-448, 2017. Available at: https://doi.org/10.1080/19345747.2016.1238985. Accessed on: 18 Jul. 2019.
MERTON, R. K. The Matthew Effect in Science: the reward and communication
systems of science are considered. Science, Washington, v. 159, n. 3810, p. 56-63, 1968. Available at: https://doi.org/10.1126/science.159.3810.56. Accessed on: 12 Nov. 2019.
MILLA, Joniada; SAN MARTÍN, Ernesto; BELLEGEM, Sébastien. Higher education value added using multiple outcomes. Journal of Educational Measurement, Boston, v. 53, n.3, p. 368-400, 2016. Available at: https://doi.org/10.1111/jedm.12114. Accessed on: 11 Jun. 2019.
MITCHELL, Douglas; MITCHELL, Ross. The political economy of education policy: the case of class size reduction. Peabody Journal of Education, London, v. 78, n.4, p. 120-152, 2003. Available at: https://doi.org/10.1207/S15327930PJE7804_07. Accessed on: 2 Aug. 2019.
NATIONAL RESEARCH COUNCIL. Getting Value out of Value-Added: report of a Workshop. Washington, DC: The National Academies Press, 2010.
ONYEMA, E. M. et al. Impact of Coronavirus Pandemic on education. Journal of Education and Practice, USA, v. 11, n. 13, p. 108-121, 2020. Available at: https://iiste.org/Journals/index.php/JEP/article/view/52821/54575. Accessed on: 18 Dec. 2020.
PIKE, G. Considerations when using value-added models in higher education assessment. Assessment Update, London, v. 28, n. 5, p. 8-10, 2016. Available at: https://doi.org/10.1002/au.30073. Accessed on: 2 Sep. 2019.
RUDENSTINE, S. et al. Depression and Anxiety During the COVID-19 Pandemic in an Urban, Low-Income Public University Sample. Journal of Traumatic Stress, Boston, v. 34, p. 12-22, 2021. Available at: https://doi.org/10.1002/jts.22600. Accessed on: 10 Jan. 2021.
SCHWARZ, G. Estimating the Dimension of a Model. Ann. Statist., Waite Hill, v. 6, n. 2, p. 461-464, 1978. Available at: https://doi.org/10.1214/aos/1176344136. Accessed on: 3 Aug. 2019.
SHAVELSON, Richard et al. On the practices and challenges of measuring higher education value-added: the case of Colombia. Assessment and Evaluation in Higher Education, London, v. 41, n. 5, p. 695-720, 2016. Available at: https://doi.org/10.1080/02602938.2016.1168772. Accessed on: 5 Jul. 2019.
STEEDLE, J. T. Selecting value-added models for post-secondary institutional assessment. Assessment and Evaluation in Higher Education, London, v. 37, n. 6, p. 637-652, 2012. Available at: https://doi.org/10.1080/02602938.2011.560720. Accessed on: 5 Jul. 2019.
USHER, Alex; MEDOW, Jon. A global survey of university rankings and league tables. In: KEHM, B. M.; STENSAKER, B. University rankings, diversity, and the new landscape of higher education. Rotterdam: Sense Publishers, 2009. p. 3-18.
YEH, Stuart. Understanding and addressing the achievement gap through individualized instruction and formative assessment. Assessment in Education: Principles, Policy and Practice, London, v.17, n.2, p. 169-182, 2010. Available at: https://doi.org/10.1080/09695941003694466. Accessed on: 5 Jul. 2019.
YEH, Stuart. Educational accountability, value-added modeling, and the origin of the achievement gap. Education and Urban Society, Los Angeles, v. 52, n. 8, p. 1181-1203, 2020. Available at: https://doi.org/10.1177/0013124519896823. Accessed on: 10 Feb. 2020.
Downloads
Publicado
Como Citar
Edição
Seção
Licença
Copyright (c) 2021 Avaliação: Revista da Avaliação da Educação Superior
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial 4.0 International License.
A remessa de originais implica a autorização para publicação e disponibilização on-line sem o pagamento de direitos atorais.
A reprodução de artigos publicados na Avaliação só é permitida com indicação da fonte. Utilização de material publicado para fins comerciais é proibida.
