Identifying and Categorizing the Indicators Used to Evaluate Scientific Outputs in the Scholarly Publication Ecosystem.

Document Type : Research Paper

Authors

1 Ph.D. in Knowledge and Information Science, Iranian Research Institution for Information Science and Technology (Iran‌Doc), Tehran, Iran .

2 Ph.D. in Knowledge and Information Science; Assistant Professor; Iranian Research Institution for Information Science and Technology (Iran‌Doc), Tehran, Iran.

3 Associate professor. department of educational science, Yadegare Imam Khomeini (RAH) Shahre rey branch, Islamic Azad University, Tehran, Iran.

Abstract

Purpose: Science and technology are the most critical infrastructures for the country's progress and essential tools for competition in various fields. Evaluation is at the core of all scientific efforts, which has become increasingly important with the proliferation of scientific publications. Evaluation is not a simple and transparent process; it is considered a delicate activity. The presence of multiple evaluation indicators to assess the value of scientific outputs in texts, databases, and scientific centers or publications has prompted a study of these three sources (texts, scientific networks, and experts) to establish integrated criteria for evaluating outputs in the scholarly publication ecosystem. Some scientific outputs, such as lectures, workshops, and scientific meetings, are often overlooked due to the absence of an integrated framework in the scholarly publication ecosystem. Additionally, only a few specific quantitative aspects, such as the impact factor, the number of citations, or the number of uses, are typically considered in the evaluation of scientific works. Moreover, evaluations are usually confined to a short timeframe. Identifying and categorizing these indicators within a framework can positively impact addressing these issues and establishing a continuous evaluation process for both pre and post-publication of scientific works. Therefore, the current research aims to identify comprehensive evaluation criteria in the scholarly publication ecosystem by taking into account texts, scholarly publication networks, and the perspectives of scientific publication experts.
Methodology: Results from all three methods indicate that there are three key indicators and nine sub-indicators crucial for evaluating research outcomes within the Scientific Publishing System. These indicators are predominantly categorized based on form, type, and format. The triangulation method has introduced a conceptual framework of evaluation indicators in the scholarly publication ecosystem. Initially, a systematic review was conducted to extract evaluation criteria from 331 sources. Subsequently, to validate the extracted criteria and finalize the initial framework, the identified criteria were scrutinized across 12 scientific databases, and ultimately, it was endorsed by 30 domestic and international scholarly publication experts. The purposive sampling method was employed in all three studies.
Findings: Research shows that the ecosystem of scientific publication consists of various components, including experts, scientific centers, information media, subject areas, information, and knowledge systems, which require different indicators and methods. The data extracted from the systematic review in the field of evaluation were classified into three groups: form, type, and format. Evaluation forms include content, open, altmetric, and bibliographic evaluations (creative and source evaluation). However, some experts distinguish between bibliometric evaluation indicators and scientometric and informatics evaluation indicators, and most experts in different subject areas define all three categories as bibliographic evaluations. In the evaluation form, creators include individuals and scientific organizations such as universities. Open evaluation can refer to judging an output not just by a jury of experts but rather by a jury of anyone interested in the output. In other words, open evaluations are an ongoing post-publication process of transparent peer evaluation. Multiple paper evaluation functions freely defined by individuals or groups provide various perspectives on the scientific literature. Multiple paper evaluation functions alongside more diverse research evaluation criteria beyond traditional methods are emerging, and with these come a range of practical, ethical, and social factors to consider. Altmetric evaluations are a set of methods based on the social web that measure and monitor the reach and impact of scholarly output through online interactions. Simply, altmetrics are metrics beyond traditional citations. This evaluating form measures cite, like, view, reuse, discussion, bookmark, etc. The types of evaluations include quantitative, qualitative, and mixed evaluations. The form of evaluation also includes technical evaluation and non-technical evaluation (researcher-made evaluation, discussion-based evaluation). Technical evaluations are indicators that follow predefined procedures or repetitive processes to reach the result, while experts define non-technical evaluations according to specific situations and conditions.
Conclusion: The results of all three methods indicate that there are three key indicators and nine sub-indicators crucial for evaluating scientific output within the scholarly publication ecosystem. These indicators are predominantly grouped based on form, type, and format. The findings demonstrate alignment among the three studies (systematic review, observation of scholarly publication networks, and survey of experts). However, each study has highlighted specific evaluation indicators. According to the systematic review, observation of scholarly publication networks, and expert opinions, the primary focus in evaluating the scholarly publication ecosystem is on the form and type of evaluation. There is a greater emphasis on common and well-established formats in the evaluation process. Furthermore, individual and organizational needs and objectives significantly influence the selection of evaluation indicators. Categorizing evaluation indicators will assist stakeholders in understanding the evaluation procedures of scholarly publication ecosystems and in choosing appropriate evaluation methods.

Keywords

Main Subjects

References

     امامی، ز.، حریری، ن.، خمسه، م.، و نوشین فرد، ف. (1395). ترسیم نقشه برون‌دادهای علمی بیماری‌های تیروئید در ایران و خاورمیانه: مطالعه علم‌سنجی. مجله غدد درون‌ریز و متابولیسم ایران، ۱۸(۱‌)، ۹119-۱991. http://ijem.sbmu.ac.ir/article-1-2041-fa.html
       امین‌پور، آ.، و صدیق، گ. (1393). اصول علم تغذیه. تهران: شرکت سهامی انتشار.
       رحمان‌سرشت، ح.، طیبی عبدالحسنی، ا.، و روحانی راد، ش.(1398). تحلیل شبکه‌های همکاری علمی پژوهشگران حوزه مدیریت راهبردی در ایران. پژوهش‌های مدیریت در ایران. 23(3)، 1-27. https://mri.modares.ac.ir/article_501.htm
رسول‌آبادی، م.، حیدری، ع.، زارعی، م.، خضری، ا.، و غریبی، ف.(1394). الگوی همکاری علمی در مقالات نمایه شده دانشگاه علوم‌پزشکی کردستان در بانک استنادی اسکوپوس از سال 2010 تا 2014. مجله دانشگاه علوم پزشکی کردستان، 20، 105-112. https://doi.org/10.22102/20.3.105
     رضی، م.، هاشم‌زاده، م. عصاره، ف.، و محمدی استانی، م. (1395). تبیین الگوی رفتار استنادی و تییین میزان همکاری علمی پژوهشگران در مجله‌های علمی پژوهشی زمین‌شناسی ایران. مطالعات کتابداری و علم اطلاعات، 8(1)، 41-62. https://slis.scu.ac.ir/article_12090.html
     عبداللهی، ف. (1397). بررسی وضعیت تولیدات و الگوهای همکاری‌های علمی دانشگاه تهران در سال 2017. [‌پایان‌نامه کارشناسی‌ارشد منتشر نشده‌[. دانشگاه تهران.
     عرفان‌منش، م.، و مروتی اردکانی، م. (1395). مطالعه علم‌سنجی و تحلیل شبکه‌های همکاری علمی در فصلنامه مطالعات میان رشته‌ای در علوم‌انسانی. مطالعات میان رشته‌ای در علوم‌انسانی، 8(4)، 55-77. https://doi.org/ 10.22035/isih.2016.230    
     کرمی، ل.، پیرحقی، م.، و صبوری، ع.(1394). شاخص‌های مرسوم و جدید در علم‌سنجی. نشاء علم، 6(1)، 6-12. http://magiran.com/p1559560
     مجیدی، م. (1395). تحلیل موضوعی، نگاشت و دیداری‌سازی گرایش‌های پژوهشی زیست فناوری ایران، [پایان‌نامه کارشناسی‌ارشد منتشر نشده‌[. دانشگاه شیراز.
    محمدزاده، ف. فهیمی‌فر، س.، و حسن‌زاده، م.(1400). بررسی مقاله‌های پراستناد پژوهشگران ایرانی در پایگاه وب‌آو‌ساینس براساس الگوی همکاری‌ها در سال‌های 2007 تا 2017. پژوهش‌نامه‌ علم‌سنجی. 7(2)، 76-98. https://doi.org/10.22070/RSCI.2020.3850.1241
    مصطفوی، ا.، و آژ، م. (1400). بررسی تأثیر همکارهای علمی بین‌المللی در افزایش کیفیت بروندادهای علمی پژوهشگران ایران در پایگاه وب آو ساینس. پژوهش‌نامه علم‌سنجی، 9(1)، 19-42. https://doi.org/10.22070/RSCI.2020.3850.1241
    همتی، م.، اکبر تبار طوری، م.، شمس، م.، بهروز پور، ا.، و رضایی، ع.(1398). سنجش سواد تغذیه‌ای معلمان ابتدایی شهری یاسوج‌: مطالعه مقطعی. ارمغان دانش، 23(1)، 124-133. http://armaghanj.yums.ac.ir/article-1-1992-fa.htm
        
 
Abdi, N., Taghdisi, M. H., Zamani-Alavijeh, F., & Sadeghi, R. (2018). Nutrition literacy promotion, effective approach for society health promotion. Journal of Health Literacy, 3(3), 49-52. https://doi.org/10.22038/JHL.2018.35136.1015
Abdullahi, F. (2017). Examining the status of productions and scientific cooperation patterns of Tehran University in 2017 [Unpublished master's dissertation]. University of Tehran. [In Persian].
Al Tell, M., Natour, N., Alshawish, E., & Badrasawi, M.(2023). The relationship between nutrition literacy and nutrition information seeking attitudes and healthy eating patterns among a group of palestinians. BMC Public Health, 23(‌165). https://doi.org/10.1186/s12889-023-15121-z
Aminpour, A., & Siddique, G. (2013). Principles of nutrition science. Tehran: Entsha Co. [In Persian].
Aytac, S.(2010). International Scholarly Collaboration in science, technology, medicine and social science of Turkish scientists. The International Information & Library Review, 42, 227-241.         https://doi.org/10.1016/j.iilr.2010.10.003
Bilik, O., Turhan Damar, H., Ozdagoglu, G., Askin Ozdagoglu, A., & Damar, M. (2020). Identifying trends, patterns, and collaborations in nursing career research. Collegian, 27, 40–4. https://doi.org/10.1016/j.colegn.2019.04.005.
Blitstein, J. L., & Evans, W.D. (2006). Use of Nutrition Facts Panels among Adults Who Make Household Food Purchasing Decisions. Journal of Nutrition Education and Behavior, 38(6), 360–364. https://doi.org/10.1016/j.jneb.2006.02.009
Chang, Y.W., Huang, M.H., & Lin, C.W. (2015). Evolution of research subjects in library and information science based on keyword, bibliographical coupling, and co-citation analyses. Scientometrics, 105(3), 2071-87. https://doi.org/10.1007/s11192-015-1762-8
Chen, S. Ren, S., Zheng, L., Yang, H., Du, W., & Cao, X. (2022). A comparison study of educational scientific collaboration in China and the USA. Physica A: Statistical Mechanics and its Applications, 585, 126-330. https://doi.org/10.1016/j.physa.2021.126330
Chien, T. W., Wang, H. Y., Chang, Y., & Kan, W. C. (2019). Using Google Maps to display the pattern of coauthor collaborations on the topic of schizophrenia: A systematic review between 1937 and 2017. Schizophrenia research, 204, 206-213. https://doi.org/ 10.1016/j.schres.2018.09.015
De Stefano, D., Fuccella, V., Vitale, M. P., & Zaccarin, S. (2013). The use of different data sources in the analysis of co-authorship networks and science performance. Social Networks, 35(3), 370-381. https://doi.org/10.1016/j.socnet.2013.04.004
Emami, Z., Hariri, N., Khamseh, M. E., & Nooshinfard, F. (2016). Mapping scientific output of thyroid disease publications in Iran and the middleast: a scientometric study. Iranian Journal of Endocrinology and Metabolism, 18(1), 1-9. http://ijem.sbmu.ac.ir/article-1-2041-en.html [In Persian].
Erfanmanesh, M., & Morovati Ardakani, M. (2016). Scientometric study and analysis of scientific cooperation networks in the quarterly journal of interdisciplinary studies in humanities. Interdisciplinary Studies in Humanities, 8(4), 55-77. https://doi.org/ 10.22035/isih.2016.230     [In Persian].
Hemati, M., Akbar Tabar Tori, M., Shams, M., Behroz pour, A., & Rezaei, A. (2017). Assessment of nutritional literacy of primary teachers in Yasouj city: a cross-sectional study. Armaghan Danesh, 23(1), 124-133. http://armaghanj.yums.ac.ir/article-1-1992-fa.htm [In Persian].
Kalkan, I. (2019). The impact of nutrition literacy on the food habits among young adults in Turke. Nutrition Research and Practice, 13(4), 352–7. https://doi.org/10.4162/nrp.2019.13.4.352
Karami, L., Pirhaghi, M., & Saboury, A. A. (2015). Current and Modern indicators in Scientometric. Science Cultivation, 6(1), 55-77. http://magiran.com/p1559560 [In Persian].
Kolasa, K. M., Peery, A., Harris, N.G., & Shovelin, K. (2001) Food Literacy Partners Program: A Strategy To Increase Community Food Literacy. Topics in Clinical Nutrition, 16(4), 1–10. https://doi.org/ 10.1097/00008486-200116040-00002
Li, J., Goerlandt, F., Reniers, G., Feng, C., & Liu, L. (2022). Chinese international process safety research: Collaborations, research trends, and intellectual basis. Journal of Loss Prevention in the Process Industries, 74, 104-657. https://doi.org/10.1016/j.jlp.2021.104657
Lu, W., Ren, Y., Huang, Y. Bu, Y., & Zhang, Y. (2012). Scientific collaboration and career stages: An ego-centric perspective. Journal of Informetric, 15(4). https://doi.org/10.1016/j.joi.2021.101207
Majidi, M. (2015). Thematic analysis, mapping and visualization of Iranian biotechnology research trends [Unpublished master dissertation]. Shiraz University. [In Persian].
Mohammadzadeh, F., Fahimifar, S., & Hasanzadeh, M. (2020). Investigating and Visualization of Iranian highly cited papers in order to discover the most effective at the international level in the period of ten years 2007-2017. Scientometrics Research Journal, 7(2), 77-98. https://doi.org/10.22070/RSCI.2020.3850.1241 [In Persian].
Mostafavi, I., & Azh, M. (2023). Investigating the relationship between international scientific collaboration and citations of highly cited and hot papers of the Iranian researchers in the Web of Science Database. Scientometrics Research Journal, 9(1), 19-42. https://doi.org/10.22070/rsci.2021.13871.1477 [In Persian].
 Nardi, P., Di Matteo, G., Palahi, M., & Mugnozza, G. S. (2016). Structure and Evolution                                          Mediterranean Forest Research: A Science Mapping Approach. PloS one, 11(5), https://doi.org/10.1371/journal.pone.0155016
Newman, M. E. (2004). Coauthorship networks and patterns of scientific collaboration. Proceedings of the National Academy of Sciences, 101(suppl 1), 5200-5205. https://doi.org/10.1073/pnas.0307545100
Nutbeam, D. (2000). Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promotion Intervention, 15(3), 259–67. https://doi.org/10.1093/heapro/15.3.259
Nutbeam, D. (2008). The evolving concept of health literacy. Social Science & Medicine, 67(12), 2072-8. https://doi.org/ 10.1016/j.socscimed.2008.09.050
OECD. (2023). Health at a Glance 2023: OECD Indicators. Paris: OECD Publishing. https://doi.org/10.1787/7a7afb35-en.
Pourkarimi Daranjani, Z., Galyani-Moghaddam, G., & Jalali Dizaji, A. (2017). Analyzing Levels of Scientific Collaboration of Iranian Researchers in the Web of Science: A Case Study of Social Sciences. Knowledge Retrieval and Semantic Systems, 3(9), 21-40. https://doi.org/10.22054/jks.2017.15975.1109 [In Persian].  
Price, D. (1963). Little science, Big science. New York: Columbia University Press. https://doi.org/10.7312/pric91844
Rahmanseresht, H., Tayibi Abolhasani, A. & Rohani Rad, Sh. (2018). Analysis of scientific cooperation networks of researchers in the field of strategic management in Iran. Management research in Iran, 23(3), 1-29. https://mri.modares.ac.ir/article 501.html file:///C:/Users/Akbar/Downloads/27713980301%20(1).pdf [In Persian].
Rasulabadi, M. Haidari, A., Zarea, M., Khezri, A. & Gharibi, F. (2014). The pattern of scientific collaboration in the indexed articles of Kurdistan University of Medical Sciences in Scopus citation bank from 2010 to 2014. Journal of Kurdistan University of Medical Sciences, 20(3), 105-112. https://doi.org/10.22102/20.3.105 [In Persian].
Razi, M., Hashemzadeh, M., Osareh, F. & Mohammadi Ostani, M. (2015). Citation behavior pattern analysis and analyzing the amount of scientific collaborations in scientific research journals of geology in Iran. Studies in Library and Information Science, 23(1), 61-82. [In Persian]. https://slis.scu.ac.ir/article_12090.html
Velmurugan, C., & Radhakrishnan, N. (2016). Malaysian journal of library and information science: A scientometric profile. Journal of Scientometric Research, 5(1), 62-70. https://doi.org/10.5530/jscires.5.1.9
Vettori, V., Lorini, C, Milani, C., & Bonaccorsi G. (2019). Towards the implementation of a conceptual framework of food and nutrition literacy: providing healthy eating for the population. International Journal of Environmental Research and Public Health, 16(24), 5041. https://doi.org/10.3390/ijerph16245041
Wang, Y., Wu, Y., Pan, Y., Ma, Z., & Rousseau, R. (2005). Scientific collaboration in China as reflected in co-authorship. Scientometrics, 62(2),183-198. https://doi.org/10.1007/s11192-005-0013-9
Wasserman, S., & Faust, K. (1994). Social Network Analysis: Methods and Applications (Structural Analysis in the Social Sciences). Cambridge: Cambridge University Press.
https://doi.org/10.1017/CBO9780511815478
Zoellner, J., Connell, C., Bounds, W., Crook, L., & Yadrick, K. (2009). Nutrition Literacy Status and Preferred Nutrition Communication Channels Among Adults in the Lower Mississippi Delta. Preventing Chronic Disease, 6(4), 1-11. https://www.researchgate.net/publication/26812852
 

Files

Share

How to cite

Statistics