Scientometrics Research Journal

Scientometrics Research Journal

Mapping the Thematic Network of Humanities Research Output Evaluation: A Mixed Method

Document Type : Research Paper

Authors
Elahe Ebrahimi Dorcheh 1
Ali Mansouri 2
1 PhD in Information Science and Knowledge Studies, Department of Information Science and Knowledge Studies, University of Isfahan, Isfahan, Iran
2 Associate Professor, Department of Information Science and Knowledge Studies, University of Isfahan, Isfahan, Iran
10.22070/rsci.2025.20793.1837
Abstract
Purpose: Evaluating the outputs of the humanities has always been a fundamental challenge in scientific systems and research policy-making. Unlike empirical sciences, which often rely on quantitative and tangible criteria and indicators to assess their outputs, the humanities—due to their qualitative, interdisciplinary, and context-dependent nature—require a different evaluation model. Therefore, it is important and necessary to manage the evaluation of scientific outputs in the humanities in a manner that aligns with their unique characteristics. Although research has highlighted the distinct features of the humanities and the need for appropriate evaluation methods, as well as offered suggestions in this regard, the planning of such evaluations has received comparatively less attention. This study aims to develop a framework to enhance understanding of the evaluation process and its application in policymaking by systematically analyzing the interactions among the components involved in evaluating humanities outputs.
Methodology: This research employs a mixed-methods approach, integrating both quantitative and qualitative techniques concurrently. Initially, a scoping review was conducted to identify and screen sources related to the evaluation of scientific outputs in the humanities. To locate potentially relevant documents, the Scopus and Web of Science databases were searched without any time restrictions. The selected sources were then analyzed using thematic analysis. To develop the themes, theoretical, inductive, and descriptive coding methods were applied to create a hierarchical structure comprising categories, domains, and taxonomies. Concurrently, we developed the conceptual structure of the data using bibliometric methods. The conceptual structure feature in Bibliometrix software (an R programming language package) is a tool for generating word co-occurrence networks (clustering), thematic maps (matrices), and thematic evolution over time (time course). As a quantitative method, it complements qualitative approaches. Finally, by integrating the results from these two stages, we created a network of themes related to the subject at hand. The thematic network, as an illustrative tool, summarizes the main themes of a text and reveals its underlying structure. It is presented graphically to make hierarchical concepts tangible, to provide fluidity among the themes, and to emphasize the interrelationships throughout the network.
Findings: In the first stage, three taxonomies were identified: evaluation approaches, evaluation methods, and evaluation tools, each comprising different domains and categories. Evaluation approaches encompass five domains: evaluation fairness, conceptual consensus, impact citation, formative evaluation, and evaluation policy. Evaluation methods include three domains beyond the article level: evaluation criteria and peer review processes. Additionally, two domains—strong infrastructure and technological evaluation—are classified under evaluation tools. In the second stage, motor, niche, basic, and emerging themes, along with their changes from 2001 to 2025, were identified. Topics related to the evaluation of scientific outputs in the humanities were among the motor themes between 2001 and 2018; these themes were well-developed and highly interconnected. However, over time, these topics have tended to shift toward basic and niche categories. This shift suggests the embeddedness of these topics within research in this area and their broader recognition in the field of humanities evaluation. Between 2019 and 2025, the thematic clusters changed significantly, with the largest shift occurring from driving, fundamental-driving, and specialized-driving themes to other quadrants. In the third stage, bibliometric results were used to establish relationships between the themes identified in the first stage and the network of themes related to evaluating scientific outputs in the humanities. Across all three taxonomies—approaches, methods, and evaluation tools—there are internal relationships among the domains. Within the taxonomy of approaches, the domain of evaluation policy exhibits the most connections with other domains, highlighting its foundational role. In the taxonomy of methods, the domain of evaluation criteria is linked to the other two domains, reflecting the critical importance of how evaluation is conducted. Overall, these findings demonstrate that humanities evaluation is shaped by a complex network of interconnected domains rather than by isolated criteria, tools, or approaches.
Conclusion: In policymaking, evaluation should function like a puzzle, meaning that approaches, methods, and evaluation tools must evolve simultaneously and proportionately. It is essential to analyze the strengths, weaknesses, opportunities, and threats of the current situation and to plan accordingly for the desired outcome. This shift in approach can lead to self-sufficiency in humanities evaluation at the policymaking level and reduce evaluation anxiety among researchers. These findings suggest that a more coordinated and context-sensitive evaluation framework can strengthen the legitimacy of humanities assessment and better align it with research policy goals.
Keywords
Humanities
Research evaluation
Scientific output
Research output
Thematic analysis
Bibliometrics
Thematic network
Subjects
ابراهیمی درچه، ا.، منصوری، ع.، پشوتنی زاده، م.، میرباقری فرد، ع.، و شعبانی، ا. (الف-1402). آسیب‌ها و راهکارهای ارزیابی برون‌دادهای پژوهشی علوم‌انسانی: مطالعه موردی رشته‌های زبان و ادبیات. پژوهش‌نامه علم‌سنجی، 9(2)، 75-96.  https://doi.org/10.22070/rsci.2022.16133.1583
ابراهیمی درچه، ا.، منصوری، ع.، پشوتنی زاده، م.، میرباقری فرد، ع.، و شعبانی, ا. (ب-1402). سیاست‌ها و معیارهای ارزیابی بروندادهای علمی علوم‌انسانی از دیدگاه متخصصان: مطالعه موردی رشته‌های زبان و ادبیات. پردازش و مدیریت اطلاعات، 39(1)، 63-100.  https://doi.org/10.22034/jipm.2023.705519
ابراهیمی، ا.، و میرشاه‌ولایتی، ف. (1402). الگوی سنجش سطح آمادگی کاربردپذیری پژوهش‌های علوم ‌انسانی و اجتماعی. سیاست علم و فناوری، 16(1)، 1-15. https://doi.org/10.22034/jstp.2023.11192.1593
جنوی، ا.، و مردانی، ا. (1403). مطالعۀ تطبیقی معیارهای ارزیابی اعضای هیئت ‌علمی دانشگاه‌ها در حوزۀ علوم ‌انسانی در ایران و جهان. مطالعات کتابداری و سازماندهی اطلاعات، 35(2)، 97-142.
https://doi.org/10.30484/nastinfo.2024.3546.2263
رفاهی کمساری، ع.، اصنافی، ا.، و حاجی زین العابدینی، م. (1401). معیارهای ارزیابی و رتبه‌بندی محتوایی کتاب‌های تخصصی در حوزه‌ علوم ‌انسانی از نظر متخصصان.  کتابداری و اطلاع‌رسانی، 25(3)، 143-165.
https://doi.org/10.30481/lis.2020.199112.1617
صفرپور، ا.، عباس پور، ج.، و سلیمی، ق. (1401). کاوش تجربیات و دیدگاه‌های اعضای هیأت علمی حوزه علوم‌ انسانی و هنر دانشگاه شیراز در مورد آیین‌نامه ارتقا. مطالعات دانش‌پژوهی، 1(1)، 29-54.
https://doi.org/10.22034/jkrs.2021.47805.1006
عباس پور، ع.، مهرابی، گ.، دلاور، ع.، و خورسندی طاسکوه، ع. (1404). طراحی و اعتباریابی الگوی کیفیت پژوهش دانشگاهی در دانشگاه‌های علوم‌انسانی و اجتماعی. اندازه‌گیری تربیتی، 16(60)، 7-36.
https://jem.atu.ac.ir/article_19007.html?lang=fa
عصاره، ف.، عفیفیان، ف.، و نورمحمدی، ح. (1398). طراحی مدل مفهومی و عملیاتی ارزیابی و رتبه‌بندی علمی گروه‌های آموزشی حوزه علوم‌انسانی دانشگاه‌ها و مؤسسات آموزش عالی ایران. پژوهش‌نامه علم‌سنجی، 5(۱)، 23-46. https://doi.org/10.22070/rsci.2019.4192.1273
علی‌نژاد چمازکتی، ف.، و میرحق جو لنگرودی، س. (1402). تحلیل علم‌سنجی و الگوهای هم نویسندگی و استناد مقالات منتشرشده در نشریات وب آو ساینس کشورهای ایران و ترکیه. پژوهش‌نامه علم‌سنجی، 9(2)، 413-444. https://doi.org/10.22070/rsci.2022.14765.1513
فاضلی، ن. (1396). علوم ‌انسانی و اجتماعی در ایران: چالش‌ها، تحولات و راهبردها. پژوهشگاه علوم انسانی و مطالعات فرهنگی. https://books.ihcs.ac.ir/book_636.html
کبیری خوزانی، ن.، منصوری، ع.، پشوتنی‌زاده، م.، و ابراهیمی درچه، ا. (1404). تحلیل کتاب‌سنجی مفاهیم ارزیابی بروندادهای پژوهشی علوم‌ انسانی. پژوهش‌نامه علم‌سنجی، 11(1)، 25-48.
https://doi.org/10.22070/rsci.2024.18702.1710
مرزبان، ا.، رضایان، ا.، و رضایان، ع. (1402). آینده آیین‌نامه ارتقای مرتبه علمی اعضای هیئت علمی در ایران: یک رویکرد سناریویی. پژوهش و برنامه‌ریزی در آموزش عالی، 29(1)، 207-233.
https://journal.irphe.ac.ir/article_707546.html
نجاتی، ف.، نوروزی چاکلی، ع.، و نورمحمدی، ح. (1402). شناسایی و تبیین شاخص‌های نخبگی و ارائة چارچوبی برای کشف نخبگان علمی در حوزة علوم ‌انسانی. پژوهش‌نامه علم‌سنجی، 9(2)، 617-654.
https://doi.org/10.22070/rsci.2022.16029.1577
نوروزی چاکلی، ع. (۱۳۹۰). آشنایی با علم‌سنجی (مبانی، مفاهیم، روابط و ریشه‌ها). سازمان مطالعه و تدوین کتب دانشگاهی برای علوم اسلامی و انسانی (سمت). https://samt.ac.ir/fa/book/99
Abbaspour, A., Mehrabi, G., Delavar, A., & Khorsandi Taskooh, A. (2025). Designing and validation of a model for academic research quality in social sciences and humanities universities. Quarterly of Educational Measurement, 16(60), 7-36.
https://jem.atu.ac.ir/article_19007.html?lang=en [In Persian].
Alinezhad Chamazkoti, F., & Mirhaghjoo Langerudi, S. (2023). Scientometric analysis and co-authoring patterns and citations of articles in Iranian and Turkish journals indexed in WoS. Scientometrics Research Journal, 9(2), 413-444.
https://doi.org/ 10.22070/rsci.2022.14765.1513 [In Persian].
Antonakis, J., & Hooijberg, R. (2008). Cascading a new vision: Three steps for real commitment. Perspectives for Managers, 157, 1. Retrieved June 19, 2024, from 
https://B2n.ir/hz4054
Aria, M., Cuccurullo, C., D’Aniello, L., Misuraca, M., & Spano, M. (2022). Thematic analysis as a new culturomic tool: The social media coverage on COVID-19 pandemic in Italy. Sustainability, 14(6), 3643. https://doi.org/10.3390/su14063643
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32.
https://doi.org/10.1080/1364557032000119616
Attride-Stirling, J. (2001). Thematic networks: An analytic tool for qualitative research. Qualitative Research, 1(3), 385–405. https://doi.org/10.1177/146879410100100307
Bonaccorsi, A., Daraio, C., Fantoni, S., Folli, V., Leonetti, M., & Ruocco, G. (2017). Do social sciences and humanities behave like life and hard sciences? Scientometrics, 112(1), 607–653. https://doi.org/10.1007/s11192-017-2384-0  
Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation, 64(1), 45‑80.
https://doi.org/10.1108/00220410810844150
Braun, V., & Clarke, V. (2022). Conceptual and design thinking for thematic analysis. Qualitative Psychology, 9(1), 3–26. https://psycnet.apa.org/doi/10.1037/qup0000196
Clarke, V., & Braun, V. (2013). Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. The Psychologist, 26(2), 120–123.
https://www.semanticscholar.org/paper/Teaching-thematic-analysis%3A-Overcoming-challenges-Clarke-Braun/2901ca4762196c5410082d235cda8dd3e1315a08
Daraio, C. (2017). A framework for the assessment of research and its impacts. Journal of Data and Information Science, 2(4), 7–42. https://doi.org/10.1515/jdis-2017-0018
De Jong, S., & Spaapen, J. (2023). Formative evaluation in the Netherlands: An opportunity for the arts, humanities and social sciences. In M. Ochsner, S. Researcher, & S. Centre (Eds.), Accountability in Academic Life (pp. 172–185). Edward Elgar Publishing.
https://doi.org/10.4337/9781800885738.00021
De Rijcke, S., Cosentino, C., Crewe, R., D’Ippoliti, C., Motala-Timol, S., Binti A Rahman, N., Rovelli, L., Vaux, D., & Yupeng, Y. (2023). The Future of Research Evaluation: A Synthesis of Current Debates and Developments [Discussion paper]. Science Health Policy; Global Young Academy; Center for Science Futures; International Sciene Council.
https://doi.org/10.24948/2023.06
Donovan, C. (2007). The qualitative future of research evaluation. Science and Public Policy, 34(8), 585–597. https://doi.org/10.3152/030234207X256538
Ebrahimi Dorcheh, E., Mansouri, A., Pashootanizadeh, M., Mirbagherifard, A. A., & Shabani, A. (2023-a). Harms and solutions for evaluating humanities research outputs: A case study of language and literature. Scientometrics Research Journal, 9(2), 75-96.
https://doi.org/10.22070/rsci.2022.16133.1583 [In Persian].
Ebrahimi Dorcheh, E., Mansouri, A., Pashootanizadeh, M., Mirbagheri fard, A. A., & Shabani, A. (2023-b). Evaluation policies and criteria of humanities scientific outputs from experts' point of view: A case study of language and literature fields. Iranian Journal of Information Processing and Management, 39(1), 63-100. https://doi.org/10.22034/jipm.2023.705519
[In Persian].
Ebrahimi Dorcheh, E., Mansouri, A., Pashootanizadeh, M., Mirbagherifard, A.-A., & Shabani, A. (2024). Determining and prioritizing the evaluation criteria of humanities scientific outputs: A case study of language and literature fields. Journal of Scientometric Research, 13(1), 148–160. https://doi.org/10.5530/jscires.13.1.13
Ebrahimi, E., & Mirshahvelayati, F. (2023). A model to measure the applicability readiness level of humanities and social sciences research. Journal of Science and Technology Policy, 16(1), 1-15. https://doi.org/10.22034/jstp.2023.11192.1593 [In Persian].
Faggiolani, C., & Solimine, G. (2018). Mapping the role of the book in evaluation at the individual and department level in Italian SSH. A multisource analysis. In A. Bonaccorsi (Ed.), The Evaluation of Research in Social Sciences and Humanities: Lessons from the Italian Experience (pp. 33–53). Springer. https://doi.org/10.1007/978-3-319-68554-0_2
Fazeli, N. (2017). Humanities and Social Sciences in Iran: Challenges, Transformations, and Strategies. Institute for Humanities and Cultural Studies. 
https://books.ihcs.ac.ir/book_636.html [In Persian].
Greenhalgh, T., Raftery, J., Hanney, S., & Glover, M. (2016). Research impact: A narrative review. BMC Medicine, 14(1), 78. https://doi.org/10.1186/s12916-016-0620-8
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Ràfols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429‑431.
https://doi.org/10.1038/520429a
Hug, S. E., Ochsner, M., & Daniel, H.-D. (2013). Criteria for assessing research quality in the humanities: A Delphi study among scholars of English literature, German literature and art history. Research Evaluation, 22(5), 369–383. https://doi.org/10.1093/reseval/rvt008
Janavi, E., & Mardani, A. (2024). Comparative study of evaluation criteria for humanities faculty members of universities in Iran and the world. Librarianship and Information Organization Studies, 35(2), 97-142. https://doi.org/10.30484/nastinfo.2024.3546.2263
      [In Persian].
Kabiri Khozani, N., Mansouri, A., Pashootanizadeh, M., & Ebrahimi Dorcheh, E. (2025). Bibliometric analysis of the evaluation concepts of humanities research outputs. Scientometrics Research Journal, 11(1), 25-48.
      https://doi.org/ 10.22070/rsci.2024.18702.1710 [In Persian].
Kulczycki, E., Guns, R., Pölönen, J., Engels, T. C. E., Rozkosz, E. A., Zuccala, A. A., Bruun, K., Eskola, O., Starčič, A. I., Petr, M., & Sivertsen, G. (2020). Multilingual publishing in the social sciences and humanities: A seven-country European study. Journal of the Association for Information Science and Technology, 71(11), 1371–1385.
https://doi.org/10.1002/asi.24336
Kyvik, S. (2003). Changing trends in publishing behaviour among university faculty, 1980-2000. Scientometrics, 58(1), 35–48. https://doi.org/10.1023/A:1025475423482
Lauronen, J. P. (2022). Tension in interpretations of the social impact of the social sciences: Walking a tightrope between divergent conceptualizations of research utilization. SAGE Open, 12(2), 1-16. https://doi.org/10.1177/2158244022108996
Law, J. (1992). Notes on the theory of the actor-network: Ordering, strategy, and heterogeneity. Systems Practice, 5(4), 379–393. https://doi.org/10.1007/BF01059830
Marzban, E., Rezayan, A., & Rezayan, A. H. (2023). The futures of the faculty promotion regulations in Iran: A scenario approach.  Qurarterly Journal of Research and Planning in Higher Education, 29(1), 207-233. https://journal.irphe.ac.ir/article_707546.html
[In Persian].
Moliní, F. (2021). The Evaluation of Innovation Within a Faculty of Arts and Humanities. Journal of Higher Education Theory and Practice, 21(6), 159-174.
https://doi.org/10.33423/jhetp.v21i6.4384
Muhonen, R., Himanen, L., & Pölönen, J. (2023). Evaluation of societal impact of the Social Sciences and Humanities in Finland. In M. Ochsner, S. Researcher, & S. Centre (Eds.), Accountability in Academic Life (pp. 95–112). Edward Elgar Publishing.
https://doi.org/10.4337/9781800885738.00016
Müller, K., Salö, L., & Sörlin, S. (2024). Quality from within: Entry points to research quality in the humanities. Research Evaluation, 33, rvae029. https://doi.org/10.1093/reseval/rvae029
Nejati, F., Noroozi Chakoli, A., & Nourmohammadi, H. (2024). Identifying and explaining elite indicators and providing a framework for discovering scientific elites in the Humanities Field. Scientometrics Research Journal, 9(2), 617-654.
https://doi.org/10.22070/rsci.2022.16029.1577 [In Persian].
Noroozi Chakoli, A. (2012). Introduction to Scientometrics (Foundations, Concepts, Relations & Origins). The Organization or Researching and Composing University Textbooks in the Islamic Sciences and th Humanities (SAMT). https://samt.ac.ir/fa/book/99 [In Persian].
Olmos-Peñuela, J., Benneworth, P., & Castro-Martinez, E. (2014). Are ‘STEM from Mars and SSH from Venus’?: Challenging disciplinary stereotypes of research’s social value. Science and Public Policy, 41(3), 384–400. https://doi.org/10.1093/scipol/sct071
Osareh, F., Afifian, F., & Nourmohammadi, H. A. (2019). Designing a conceptual and operational model for scientific evaluation and ranking the of educational departments of humanities in universities and institutes of higher education in Iran. Scientometrics Research Journal, 5(1), 23-46. https://doi.org/ 10.22070/rsci.2019.4192.1273 [In Persian].
Pedersen, D. B., Grønvad, J. F., & Hvidtfeldt, R. (2020). Methods for mapping the impact of social sciences and humanities—A literature review. Research Evaluation, 29(1), 4–21.
https://doi.org/10.1093/reseval/rvz033
Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015). Guidance for conducting systematic scoping reviews. International Journal of Evidence-Based Healthcare, 13(3), 141–146. https://pubmed.ncbi.nlm.nih.gov/26134548
Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm, P., Larkin, C., Lepori, B., Mosoni-Fried, J., Oliver, E., Primeri, E., Puigvert, L., Scharnhorst, A., Schubert, A., Soler, M., Soòs, S., Sordé, T., Travis, C., & Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298–308. https://doi.org/10.1093/reseval/rvx025
Refahi Komsari, A., Asnafi, A. R., & Haji Zeinlobaedini, M. (2022). Criteria for evaluating and ranking the content of specialized books in the field of humanities according to specialists' viewpoint. Library and Information Sciences, 25(3), 143-165.
https://doi.org/10.30481/lis.2020.199112.1617 [In Persian].
Rovira-Esteva, S., & Orero, P. (2012). Evaluating quality and excellence in translation studies research: Publish or perish, the Spanish way. International Journal of Translation, 58(3), 264–288. https://www.researchgate.net/publication/263110817
Safarpour, E., Abbaspour, J., & Salimi, G. (2022). Exploring the experiences and views of Shiraz University’s Art and Humanities Faculty Members on Promotion Regulation. Journal of Knowledge-Research Studies, 1(1), 29-54.
      https://doi.org/10.22034/jkrs.2021.47805.1006 [In Persian].
Sani’ee, N., Nemati-Anaraki, L., Sedghi, S., Noroozi Chakoli, A., & Goharinezhad, S. (2022). The effective trends and driving forces in the future of research performance evaluation: A qualitative study. Medical Journal of The Islamic Republic of Iran, 36(1), 393–406.
https://pmc.ncbi.nlm.nih.gov/articles/PMC9448456
Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics, 107(2), 357–368.
https://doi.org/10.1007/s11192-016-1845-1
Toledo, E. G. (2016). Research assessment in humanities and social sciences in review. Revista Española de Documentación Científica, 41(3), 1–14.
https://doi.org/10.3989/redc.2018.3.1552
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … & Straus, S. E. (2018). PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Verleysen, F. T., & Weeren, A. (2016). Clustering by publication patterns of senior authors in the social sciences and humanities. Journal of Informetrics, 10(1), 254–272.
https://doi.org/10.1016/j.joi.2016.01.004
Vučković, D., Pekovic, S., Popović, S., & Janinovic, J. (2023). Assessing the Appraisal of Research Quality in Social Sciences and Humanities: A Case Study of the University of Montenegro. Business Systems Research, 14(1), 131–152.
https://doi.org/10.2478/bsrj-2023-0007
Wang, L., & Wang, T. (2026). Research efficiency evaluation and regional differences analysis of humanities and social sciences of colleges and universities in China. Soft Computing, 30, 2027–2038. https://doi.org/10.1007/s00500-024-09662-7
Wang, R. (2023). Research on problems of scientific research performance evaluation mechanism of humanities and social sciences. Frontiers in Educational Research, 6(18). https://doi.org/10.25236/FER.2023.061823
Wilsdon, J. (2015). The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management.  Sage.
https://sk.sagepub.com/book/mono/the-metric-tide/toc#_
Wróblewska, M. N. (2021). Research impact evaluation and academic discourse. Humanities and Social Sciences Communications, 8(1), 1–12.
https://doi.org/10.1057/s41599-021-00727-8
Yang, S., Zheng, M., Yu, Y., & Wolfram, D. (2021). Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities? Journal of Informetrics, 15(1), 1–21. https://doi.org/10.1016/j.joi.2020.101120