The Criteria and Indicators Affecting the Divergence Between the Iranian National Ranking System and Global University Ranking Systems

Document Type : Research Paper

Author

Ph.D. in Knowledge and Information Science; Assistant Professor; Department of Scientometrics; Islamic World Science and Technolo-gy Monitoring and Citation Institute (ISC

Abstract

Purpose: Alongside research and educational indicators, national ranking systems evaluate the performance of universities in terms of such indicators as social responsibilities, facilities, entrepreneurship, commercialization of technology, university infrastructure, economic impact, technological developments, and quality of education. These indicators are typically not utilized in global rianking systems because of the difficulties associated with collecting data on a global level. The Iranian National University Ranking System, which evaluates the performance of Iranian universities, differs from global university ranking systems. These differences may result from its methodology. While it incorporates some criteria and indicators from the global university ranking systems, it has added some new ones to compensate for the shortcomings of the global university ranking systems and, thereby, achieve a more realistic picture of the performance of universities. These innovations necessitate comparing the two approaches in order to understand how they affect the results of the university ranking systems. This research aimed to identify the criteria and indicators leading to the divergence of the results of the Iranian National University Ranking System­ (as a national university ranking) and several well-known global university ranking systems. The latter included Times Higher Education (THE), Shanghai Academic Ranking of World Universities (ARWU), University Ranking by Academic Performance (URAP), and Islamic World Science Citation (ISC WUR).
 
Methodology: The present study was a descriptive scientometric research based on a library method. It examined a sample of top Iranian universities commonly ranked by the Iranian system and each of the aforementioned global systems in 2020. Using semi-partial correlation analysis in SPSS 23, we analyzed the correlations between the universities' total scores reported by the Iranian and global systems through controlling the criteria and indicators (un)common between these systems.
 
Findings: The findings showed that some criteria and indicators of the Iranian system account for the divergence of its results from those of the THE system. These include education, faculty member hierarchy, employment of graduates, effectiveness of research outputs, scientific publications, international mobility of students and faculty members, university budget, university income, and facilities. In addition, the citation criterion in THE contributes to the difference between the results of the two systems. Besides, it has been found that the criterion of economic impact partially contributes to the divergence of results between the Iranian National Ranking System and the ARWU system. Moreover, graduates' employment is an indicator that contributes to the divergence between the results of the Iranian system of ranking universities and the results of the URAP system. On the other hand, none of the indicators of the Iranian National Ranking System lead to a divergence of its results from those of the ISC WUR system.
 
Conclusion: The results showed that some innovative criteria and indicators used in the Iranian National University Ranking System play a role in the divergence of its results and those of the global university ranking systems. This implies that methodological innovations have enriched this national system, enabling it to measure different dimensions of a university's performance. However, there are inconsistencies. Some innovative criteria and indicators led to the convergence of the results or showed no effect on the correlations, while some similar factors unexpectedly contributed to the divergence. The divergence in the ranking results caused by similar criteria and indicators is likely to stem from differences in the calculation method, periods, and data sources. Overall, it appears that the methodology used in the Iranian National University Ranking System leads to variations in the ranking outcomes. However, the differences cannot necessarily be interpreted as conductive to a more realistic evaluation. Consequently, there is a need for further investigations, particularly qualitative ones, to assess the conformity of the results with a gold standard benchmark.

Keywords

References

ارشدی، ه.، و اخوتی، م. (1401). تحلیل مقایسه‌ای عملکرد پژوهشی دانشگاه علوم پزشکی کرمان و دانشگاه‌های منتخب. پژوهش‌نامه علم‌سنجی، (زودآیند). https://doi.org/ 10.22070/rsci.2022.16043.1580
حسابی، س.، ستوده، ه.، و یوسفی، ز. (1401). هم‌بستگی بین رتبه‌بندی دانشگاه‌ها به دو روش کتاب‌سنجی محض و داوری غنی‌شده با کتاب‌سنجی: نمونه مورد مطالعه نظام تعالی پژوهش انگلیس و نظام‌های رتبه‌بندی معتبر جهانی. پژوهش‌نامه علم‌سنجی، 8 (16)، 75-98. https://doi.org/ 10.22070/rsci.2020.5836.1437
خانی‌زاد، ر.، و منتظر، غ. (1396). ارزیابی تطبیقی نظام‌های رتبه‌بندی دانشگاه‌های جهان. سیاست علم و فناوری، 9 (3)، 31-43. https://jstp.nrisp.ac.ir/article_12985.html
رجبعلی بگلو، ر.، و جوکار، ع. (1385). رابطه رتبه‌بندی دانشگاه‌ها جهان و میزان پیوند به آنها: بر اساس رتبه‌بندی دانشگاه شانگهای چین و آموزش عالی تایمز. اطلاع‌شناسی، 4 (1-2)، 179-190. https://www.sid.ir/paper/467253/fa
عصاره، ف.، فرج‌پهلو، ع.، رحیمی، ف.، و پارسایی محمدی، پ. (1398). مطالعه معیارها و شاخص‌‌های موجود در رتبه‌بندی‌های دانشگاهی ملّی. پژوهش‌نامه علم‌سنجی، 5 (10)، 1-22. https://doi.org/ 10.22070/rsci.2019.4344.1283
مشتاق، م.، ستوده، ه.، یقطین، م.، و جوکار، ط. (1400). هم‌بستگی نتایج سامانه‌های رتبه‌بندی نمایه نیچر و لایدن با تایمز و کیو-اس. پژوهش‌نامه علم‌سنجی، 7 (14)، 157-172. https://doi.org/10.22070/rsci.2020.5488.1384
موسوی اوندی، ن. (1396). بررسی رتبه‌بندی دانشگاه شهید چمران اهواز بر اساس نظام رتبه‌بندی پایگاه استنادی علوم جهان اسلام طی برنامه پنجم توسعه کشور 1390-1394 [پایان‌نامه کارشناسی ارشد منتشرنشده]. دانشگاه شهید چمران اهواز.
 
Aguillo, I., Bar-Ilan, J., Levene, M., & Ortega, J. (2010). Comparing university rankings. Scientometrics, 85(1), 243-256. https://doi.org/10.1007/s11192-010-0190-z
Alaşehir, O., Çakır, M. P., Acartürk, C., Baykal, N., & Akbulut, U. (2014). URAP-TR: a national ranking for Turkish universities based on academic performance. Scientometrics, 101, 178-159. https://doi.org/10.1007/s11192-014-1333-4
Anowar, F., Helal, M. A., Afroj, S., Sultana, S., Sarker, F., & Mamun, K. A. (2015). A critical review on world university ranking in terms of top four ranking systems. New trends in networking, computing, e-learning, systems sciences, and engineering, 559-566. https://doi.org/10.1007/978-3-319-06764-3_72
Arshadi, H., & Akhovati, M. (2022). Comparative analysis of research performance of Kerman University of Medical Sciences and selected universities. Scientometrics Research Journal, https://doi.org/ 10.22070/rsci.2022.16043.1580 [In Persian]
Asher, A., & Savino, M. (2007). A Global Survey of Rankings and League Tables/College and University Ranking Systems Global Perspectives and American Challenges/Institute for Higher Education Policy (editor), 23–34, Voprosy obrazovaniya/Educational Studies Moscow, (4), April, 201-216. Retrieved from https://vo.hse.ru/data/2010/12/31/1208181914/8%20Asher.pdf
Basu, A., Banshal, S. K., Singhal, K., & Singh, V. K. (2016). Designing a Composite Index for research performance evaluation at the national or regional level: ranking Central Universities in India. Scientometrics, 107, 1171-1193. https://doi.org/10.1007/s11192-016-1935-0
Baty, P. (2011). Global Rankings: Change for the better. The World University Rankings, Times Higher Education, 6, 2011-2012. Retrieved from https://www.timeshighereducation.com/world-university-rankings/2011-12/world-ranking/analysis/the-global-rankings
Çakır, M. P., Acartürk, C., Alaşehir, O., & Çilingir, C. (2015). A comparative analysis of global and national university ranking systems. Scientometrics, 103(3), 813-848. https://doi.org/10.1007/s11192-015-1586-6
Chen, K. H., & Liao, P. Y. (2012). A comparative study on world university rankings: a bibliometric survey. Scientometrics, 92(1), 89-103. https://doi.org/10.1007/s11192-012-0724-7
Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press. https://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf
Dill, D. D., & Soo, M. (2005). Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems. Higher education, 49(4), 495-533. https://doi.org/10.1007/s10734-004-1746-8
Docampo, D. (2012). Adjusted sum of institutional scores as an indicator of the presence of university systems in the ARWU ranking. Scientometrics, 90(2), 701-713. https://doi.org/10.1007/s11192-011-0490-y
Docampo, D., & Cram, L. (2015). On the effects of institutional size in university classifications: The case of the Shanghai ranking. Scientometrics, 102(2), 1325-1346. https://doi.org/10.1007/s11192-014-1488-z
Federkeil, G., Van Vught, F. A., & Westerheijden, D. F. (2012). An evaluation and critique of current rankings. In Multidimensional ranking: The design and development of U-Multirank, 39-70, Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-94-007-3005-2_4
Frenken, K., Heimeriks, G. J., & Hoekman, J. (2017). What drives university research performance? An analysis using the CWTS Leiden Ranking data. Journal of informetrics, 11(3), 859-872. https://doi.org/10.1016/j.joi.2017.06.006
Geraci, M., & Degli Esposti, M. (2011). Where do Italian universities stand? An in-depth statistical analysis of national and international rankings. Scientometrics, 87(3), 667-681. https://doi.org/10.1007/s11192-011-0350-9
                         
Hesabi, S., Sotudeh, H., & Yousefi, Z. (2022). A Correlation Study of Bibliometric-Based and Informed-Peer-Review University Rankings: The Case of UK Research Excellence Framework (REF) and the World's Prestigious University Ranking Systems. Scientometrics Research Journal, 8 (2), 75-98. https://doi.org/10.22070/rsci.2020.5836.1437 [In Persian]
Huang, M. H. (2011). A comparison of three major academic rankings for world universities: From a research evaluation perspective. Journal of Library & Information Studies, 9(1). License: CC BY-NC-ND 4.0
Hushyar Sherwani, K. (2018). Comparative Analysis of National University Ranking System in Kurdistan-Region and Other National University Rankings: An Emphasis on Criteria and Methodologies. International Journal of Social Sciences & Educational Studies, 5(1), 7-15. https://doi.org/10.23918/ijsses.v5i1p7
Johnes, J. (2018). University rankings: What do they really show?. Scientometrics, 115(1), 585-606. https://doi.org/10.1007/s11192-018-2666-1
Jöns, H., & Hoyler, M. (2013). Global geographies of higher education: The perspective of world university rankings. Geoforum, 46, 45-59. https://doi.org/10.1016/j.geoforum.2012.12.014
Khanizad, R., & Montazer, G. (2017). A Comparative Evaluation of the World University Rankings Systems. Journal of Science and Technology Policy, 9(3), 31-43. https://jstp.nrisp.ac.ir/article_12985.html [In Persian].
Khosrowjerdi, M., & Kashani, Z. S. (2013). Asian top universities in six world university ranking systems. Webology, 10(2), 1-9. https://www.webology.org/2013/v10n2/a114.pdf
Lee, D. H. (2019). Predictive power of conference-related factors on citation rates of conference papers. Scientometrics, 118(1), 281-304. https://doi.org/10.1007/s11192-018-2943-z
Loyola-González, O., Medina-Pérez, M. A., Valdez, R. A. C., & Choo, K. K. R. (2020). A contrast pattern-based scientometric study of the qs world university ranking. IEEE Access, 8, 206088-206104. https://doi.org/10.1109/ACCESS.2020.3037665
Memisevic, H., & Memisevic, M. (2022). Relationship between Webometrics University rankings and research gate scores, Scopus and web of science. International Journal of Information Science and Management (IJISM), 20(3), 1-8. https://www.researchgate.net/publication/361747686_Relationship_between_Webometrics_University_Rankings_and_Research_Gate_Scores_Scopus_and_Web_of_Science
Moshfeghi, N., & Nadi, M. A. (2018). The comparative study of ranking system of islamic countries universities and national ranking of universities in iran using the most famous ranking systems in the world. Czech-polish historical and pedagogical journal, 10(2). https://doi.org/10.5817/cphpj-2018-016
Moshtagh, M., Sotudeh, H., Yaghtin, M., & Jowkar, T. (2021). The Correlation of Nature and Leiden Index Ranking Systems with Times and QS. Scientometrics Research Journal, 7((Issue 2, Autumn & Winter)), 157-172. https://doi.org/10.22070/rsci.2020.5488.1384 [In Persian].
Mousavi Avendi, N. (2018). A study of the shahid Chamran University based on the Islamic Citation Sciences Ranking during the country's fifth development plan 2012-16 [Unpublished master dissertation]. Shahid Chamran University, Ahvaz. [In Persian].
Osareh, F., Farajpahlou, A., Rahimi, F., & ParsaeiMohammadi, P. (2019). A study of the criteria and indicators of the National University Ranking Systems. Scientometrics Research Journal, 5, Issue 2, (Autumn & Winter), 1-22. https://doi.org/ 10.22070/rsci.2019.4344.1283 [In Persian].
Rajabali Beglou, R., Jowkar, A. (2006). The relationship between the world university rankings and their degree of linkage based on the Shanghai University ranking of China and Times Higher Education. Informology, 14, 179-190. https://www.sid.ir/paper/467253/fa [In Persian].
Robinson-García, N., Torres-Salinas, D., Delgado López-Cózar, E., & Herrera, F. (2014). An insight into the importance of national university rankings in an international context: the case of the I-UGR rankings of Spanish universities. Scientometrics, 101, 1309-1324. https://doi.org/10.1007/s11192-014-1263-1
Robinson-Garcia, N., Torres-Salinas, D., Herrera-Viedma, E., & Docampo, D. (2019). Mining university rankings: Publication output and citation impact as their basis. Research Evaluation, 28(3), 232-240. https://doi.org/10.1093/reseval/rvz014
Sani'ee, N., Nemati-Anaraki, L., Sedghi, S., Noroozi Chakoli, A., & Goharinezhad, S. (2022). Comparative study of research performance and innovation-industry indicators in national and international university ranking systems. Journal of Medical Library and Information Science, 3, e30, 1-11. https://doi.org/10.22037/jmlis.v3i.38107
Selten, F., Neylon, C., Huang, C. K., & Groth, P. (2020). A longitudinal analysis of university rankings. Quantitative Science Studies, 1(3), 1109-1135. https://doi.org/10.1162/qss_a_00052
Shehatta, I., & Mahmood, K. (2016). Correlation among top 100 universities in the major six global rankings: policy implications. Scientometrics, 109(2), 1231-1254. https://doi.org/10.1007/s11192-016-2065-4
Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In University rankings, diversity, and the new landscape of higher education‌,  1-18, Brill. https://doi.org/10.1080/03797720701618831
Van Raan, A. F. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133-143. https://doi.org/10.1007/s11192-005-0008-6
Velicer, W. F. (1978). Suppressor variables and the semipartial correlation coefficient. Educational and Psychological Measurement, 38(4), 953-958. https://doi.org/10.1177/001316447803800415
Yang, H., Wong, W. H., Bradley, K. D., & Toland, M. D. (2017). Partial and semi-partial correlations for categorical variables in educational research: addressing two common misconceptions. General linear model journal, 43(1), 1-15. https://doi.org/10.31523/glmj.043001.001
Zhang, Y., Xiao, Y., Wu, J., & Lu, X. (2021). Comprehensive world university ranking based on ranking aggregation. Computational Statistics, 36, 1139-1152. https://doi.org/10.1007/s00180-020-01033-8

Files

Share

How to cite

Statistics