Research Evaluation
Mapping the Field Structure
DOI:
https://doi.org/10.1590/S1414-40772020000300003Keywords:
Avaliação da Pesquisa, Estrutura intelectual, Análise bibliométrica, Visualização de dadosAbstract
O tema Avaliação da Pesquisa tem sido objeto de intenso estudo tanto pela sua importância para múltiplos propósitos, quanto pelo desenvolvimento teórico do modelo sistema de avaliação numa perspetiva de procura de melhoria contínua da qualidade de desempenho da pesquisa e do seu impacto societal e impacto na inovação. O objetivo deste estudo exploratório é recolher e mapear a estrutura intelectual da Avaliação de Pesquisa. Usando citações de redes, capturamos os blocos de construção latentes e as linhas condutoras deste tema. A visualização e análise de dados foram úteis para fazer uma exploração dos antecedentes da literatura; observando as redes de citação e seus agrupamentos, identificamos artigos-chave (artigos seminais e artigos relevantes) que suportam as diferentes áreas de Avaliação de Pesquisa. O principal resultado é o mapa de citações organizado em seis grupos (clusters), fornece uma visão geral rápida desses territórios interligados, dando uma compreensão dessas diferentes áreas de conhecimento. Esta é uma imagem clara deste tópico, útil para acadêmicos, pesquisadores juniores, bem como dos diversos interessados práticos com interesse na política de avaliação de pesquisa ou na avaliação da qualidade da pesquisa e sua implementação. Identificamos publicações seminais e publicações relevantes com elevados índices de citação. Este estudo é um sólido ponto de partida para uma futura e profunda revisão da literatura sobre “avaliação de pesquisas”.
Downloads
References
AKSNES, D. W. Characteristics of highly cited papers. Research Evaluation, Oxford, UK , v. 12, n. 3, p. 159-170, 2003.
AKSNES, D. W.; RIP, A. Researchers’ perceptions of citations. Research Policy, Brighton, UK, v. 38, n. 6, p. 895-905, 2009.
ANDERSON, G. Fundamentals of educational research. London: Routledge, 1998.
APA. Diagnostic and Statistical Manual of Mental Disorders (DSM-IV). 4. Arlington: American Psychiatric Association, 1994.
BAIRD, L. M.; OPPENHEIM, C. Do citations matter? Journal of Information Science, Aberystwyth, UK, v. 20, n. 1, p. 2-15, 1994.
BARTOL, T. et al. Assessment of research fields in Scopus and Web of Science in the view of national research evaluation in Slovenia. Scientometrics, Belgium, v. 98, n. 2, p. 1491-1504, Feb 2014.
BECK, A. T. et al. An inventory for measuring depression. Archives of General Psychiatry, USA, v. 4, p. 561-571, 1961.
BORDONS, M.; FERNÁNDEZ, M.; GÓMEZ, I. Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, Belgium, v. 53, n. 2, p. 195-206, 2002.
BROUWERS, M. C. et al. AGREE II: advancing guideline development, reporting and evaluation in health care. Canadian Medical Association Journal, Canada, v. 182, n. 18, p. E839-E842, dec. 2010.
BUXTON, M.; HANNEY, S. How Can Payback from Health Services Research Be Assessed? Journal of Health Services Research, UK, v. 1, n. 1, p. 35-43, 1996.
CALLAHAM, M.; WEARS, R. L.; WEBER, E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA, USA, v. 287, n. 21, p. 2847-2850, 2002.
CECI, S. J.; PETERS, D. P. Peer Review: A Study of Reliability. Change: The Magazine of Higher Learning, Canada, v. 14, n. 6, p. 44-48, set. 1982.
CHARNES, A.; COOPER, W. W.; RHODES, E. Measuring the efficiency of decision making units. European Journal of Operational Research, Europe, v. 2, n. 6, p. 429-444, 1978.
CLUZEAU, F. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality & Safety in Health Care, UK, v. 12, n. 1, p. 18-23, 2003.
COATS, A. W. The role of scholarly journals in the history of economics: An essay. Journal of Economic Literature, USA, v. 9, n. 1, p. 29-44, 1971.
COHEN, J. Statistical Power Analysis for the Behavioral Sciences. 2. ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.
CRONBACH, L. J.; MEEHL, P. E. Construct validity in psychological tests. Psychological Bulletin, USA, v. 52, n. 4, p. 281-302, 1955.
DE SOLLA PRICE, D. Editorial statements. Scientometrics, Belgium, v. 1, n. 1, p. 3-8, sep. 1978.
DIETZ, J. S.; BOZEMAN, B. Academic careers, patents, and productivity: industry experience as scientific and technical human capital. Research Policy, Brighton, UK,v. 34, n. 3, p. 349-367, 2005.
DIMAGGIO, P. J.; POWELL, W. W. The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields. American Sociological Review, USA, v. 48, n. 2, p. 147-160, 1983.
DONOVAN, C. The qualitative future of research evaluation. Science and Public Policy, Oxford, UK, v. 34, n. 8, p. 585-597, 2007.
FREEMAN, L. C. Centrality in Social Networks Conceptual Clarification. Social Networks, UK, v. 1, n. 3, p. 215-239, 1979.
GARFIELD, E. Citation indexes for science: a new dimension in documentation through association of ideas. Science, UK, v. 122, p. 108-111, 1955.
GARFIELD, E. Citation Analysis As a Tool in Journal Evaluation - Journals Can be Ranked by Frequency and Impact of Citations for Science Policy Studies. Science, UK, v. 178, p. 471-479, 1972.
GARFIELD, E. Is citation analysis a legitimate evaluation tool? Scientometrics, Belgium, v. 1, n. 4, p. 359-375, May 1979.
GARFIELD, E. Current Contents. Mapping cholera research and the impact of Sambhu Nath De of Calcutta. Essays of an Information Scientist, USA, v. 9, n. 15, p. 112, 1986.
GARFIELD, E. Citation indexes for retrieval and research evaluation. Consensus Conference on the Theory and Practice of Research Assessment. Capri, Italy 1996.
GARFIELD, E. The evolution of the science citation index. International microbiology, UK, v. 10, n. 1, p. 65, 2007.
GARNER, J. G. et al. Assessing research network and disciplinary engagement changes induced by an NSF program. Research Evaluation, Oxford, UK, v. 21, n. 2, p. 89-104, June 2012.
GEARY, J.; MARRIOTT, L.; ROWLINSON, M. Journal Rankings in Business and Management and the 2001. Research Assessment Exercise in the UK. British Journal of Management, UK, v. 15, n. 2, p. 95-141, 2004.
GIBBONS, M. et al. The New Production of Knowledge: The dynamics of science and research in contemporary societies. London: SAGE Publications, 1994.
GLÄNZEL, W.; SCHOEPFLIN, U. Little scientometrics, big scientometrics ... and beyond? Scientometrics, Belgium, v. 30, n. 2, p. 375-384, 1994.
GLANZEL, W.; THIJS, B.; DEBACKERE, K. The application of citation-based performance classes to the disciplinary and multidisciplinary assessment in national comparison and institutional research assessment. Scientometrics, Belgium, v. 101, n. 2, p. 939-952, nov. 2014.
GLASER, B. G.; STRAUSS, A. L. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine, 1967.
GUTHRIE, S. et al. Measuring research: a guide to research evaluation frameworks and tools. Santa Monica: Rand, 2013.
HICKS, D. Evolving regimes of multi-university research evaluation. Higher Education, Oxford,UK, v. 57, n. 4, p. 393-404, Apr. 2009.
HICKS, D. Performance-based university research funding systems. Research Policy, Brighton, UK, v. 41, n. 2, p. 251-261, 2012.
HICKS, D. What year? Difficulties in identifying the effect of policy on university output. Journal of Informetrics, Belgium, v. 11, n. 3, p. 933-936, 2017.
HIRSCH, J. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, USA, v. 102, n. 46, p. 16569–16572, 2005.
HOFSTEDE, G. Culture’s consequences: International differences in work-related value. Beverly Hills, CA: Sage, 1980.
HOFSTEDE, G. Culture's consequences: comparing values, behaviors, institutions and organizations across nations. 2. ed. Thousand Oaks: SAGE Publications, 2001.
HOUSE, R. J. et al. Culture, leadership, and organizations: the GLOBE study of 62 societies. Thousand Oaks, CA: Sage, 2004.
JUBB, M. Prometheus assessed? Research measurement, peer review and citation analysis. Learned Publishing, Cmbridge,UK, v. 26, n. 3, p. 228-229, 2013.
KUHN, T. S. The structure of scientific revolutions. Chicago, US: University of Chicago Press, 1962.
LANDIS, J. R.; KOCH, G. G. The measurement of observer agreement for categorical data. Biometrics, Washington, USA, v. 33, n. 1, p. 159-174, 1977.
LEE, S.; BOZEMAN, B. The impact of research collaboration on scientific productivity. Social Studies of Science, Canada, v. 35, n. 5, p. 673-702, 2005.
LEVIN, S. G.; STEPHAN, P. E. Research productivity over the life cycle: evidence for academic scientists. The American economic review, Nashville, USA, p. 114-132, 1991.
LEYDESDORFF, L. Theories of citation? Scientometrics, Belgium, v. 43, n. 1, p. 5-25, 1998.
LEYDESDORFF, L.; MILOJEVIĆ, S. Scientometrics In: INTERNATIONAL Encyclopedia of the Social & Behavioral Sciences (Second Edition). Oxford: Elsevier, 2015. p. 322-327.
LIEBOWITZ, S. J.; PALMER, J. P. Assessing the relative impacts of economics journals. Journal of Economic Literature, USA, v. 22, n. 1, p. 77-88, 1984.
LINCOLN, Y. S.; GUBA, E. G. Naturalistic Inquiry. Beverly Hills: SAGE Publications, 1985.
LOTKA, A. J. The frequency distribution of scientific productivity. Journal of the Washington academy of sciences, USA,v. 16, n. 12, p. 317-323, 1926.
MACDONALD, S.; KAM, J. Ring a ring o' roses: quality journals and gamesmanship in management studies. Journal of Management Studies, UK, v. 44, n. 4, p. 640-655, June 2007.
MACROBERTS, M. H.; MACROBERTS, B. R. Problems of citation analysis: A critical review. Journal of the American Society for information Science, USA, v. 40, n. 5, p. 342, 1989.
MAHONEY, M. J. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, Boston, USA, v. 1, n. 2, p. 161-175, 1977.
MARTIN, B. R. The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster? Research Evaluation, Oxford, UK, v. 20, n. 3, p. 247-254, Sept. 2011.
MEHO, L. I.; YANG, K. Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, USA, v. 58, n. 13, p. 2105-2125, 2007.
MERTON, R. On theoretical sociology. Five essays, old and new. New York: The Free Press, 1967.
MILES, M. B.; HUBERMAN, A. M.; SALDAÑA, J. Qualitative Data Analysis. Thousand Oaks: SAGE Publications, 2013.
MOED, H. F. Citation analysis in research evaluation. Dordrecht: Springer Verlag, 2005.
NALIMOV, V.; MULCJENKO, B. Measurement of science. Study of the development of science as an information process. Washington , DC: Foreign Technology Division, 1971.
NEDERHOF, A. J. et al. Research performance indicators for university departments: A study of an agricultural university. Scientometrics, Belgium,v. 27, n. 2, p. 157-178, 1993.
NEWMAN, M. E. The structure of scientific collaboration networks. Proceedings of the national academy of sciences, USA, v. 98, n. 2, p. 404-409, 2001.
NOWOTNY, H.; SCOTT, P.; GIBBONS, M. Re-thinking science: knowledge and the public in an age of uncertainty. Cambridge: Polity Press, 2001.
OECD. The measurement of scientific and technical activities: proposed standard practice for surveys of research and development. Paris: OECD, 1962.
OECD. Frascati manual. 6. ed. Paris, France: OECD, 2002.
OSWALD, A. J. An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers. Economica, USA,v. 74, n. 293, p. 21-31, 2007.
PAGE, W. Research evaluation: welcome. Research Evaluation, Oxford, UK, v. 1, n. 1, p. 2-2, 1991.
PATSOPOULOS, N. A.; ANALATOS, A. A.; IOANNIDIS, J. P. Relative citation impact of various study designs in the health sciences. JAMA - Journal of the American Medical Association, USA, v. 293, p. 2362-2366, 2005.
PINHO, I .; ROSA, M. J. Towards a new research units' evaluation model in an open science context- a proposal. Research Day Proceedings, Aveiro, Portugal, 2016.
POLANYI, M. The republic of science: Its political and economic theory. Minerva, Netherlands, v. 1, p. 54-73, 1962.
PORTER, A. L. et al. Measuring researcher interdisciplinarity. Scientometrics, Belgium, v. 72, n. 1, p. 117-147, 2007.
STARBUCK, W. H. How much better are the most-prestigious journals? The statistics of academic publication. Organization Science, USA, v. 16, n. 2, p. 180-200, 2005.
ŠUBELJ, L.; VAN ECK, N. J.; WALTMAN, L. Clustering scientific publications based on citation relations: a systematic comparison of different methods. PLoS ONE, California, USA, v. 11, n. 4, p. e0154404, 2016.
VAN ECK, N. J.; WALTMAN, L. CitNetExplorer: a new software tool for analyzing and visualizing citation networks. Journal of Informetrics, UK, v. 8, n. 4, p. 802-823, Oct. 2014.
VAN ECK, N. J.; WALTMAN, L. VOSviewer manual. Leiden: University of Leiden, 2015.
VAN RAAN, A. F. J. Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, Belgium, v. 62, n. 1, p. 133-143, 2005.
VINKLER, P. Relations of relative scientometric indicators. Scientometrics, Belgium, v. 58, n. 3, p. 687-694, Nov. 2003.
VOOS, H.; DAGAEV, K. S. Are All Citations Equal? Or, Did We Op. Cit. Your Idem? Journal of Academic Librarianship, UK, v. 1, n. 6, p. 19-21, 1976.
WALTMAN, L.; VAN ECK, N. J. A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, USA, v. 63, n. 12, p. 2378-2392, 2012.
WEISS, C. H. The many meanings of research utilization. Public administration review, USA, v. 39, n. 5, p. 426-431, 1979.
WEISS, C. Evaluation: methods for studying programs and policies. 2. ed. Englewood Cliffs: Prentice Hall, 1998.
WOOLF, S. H. et al. Clinical guidelines: potential benefits limitations and harms of clinical guidelines. BMJ, USA, v. 318, p. 527-530, 1999.
YOUNG, N. S.; IOANNIDIS, J. P. A.; AL-UBAYDLI, O. Why current publication practices may distort science. Plos Medicine, UK, v. 5, n. 10, p. e201, 2008.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Avaliação: Revista da Avaliação da Educação Superior
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
A remessa de originais implica a autorização para publicação e disponibilização on-line sem o pagamento de direitos atorais.
A reprodução de artigos publicados na Avaliação só é permitida com indicação da fonte. Utilização de material publicado para fins comerciais é proibida.
