Impact evaluation of scientific journals based on citation content analysis data and citation indicators

doi:10.11946/cjstp.202203230197

Abstract

Abstract:

[Purposes] Combining the citation content analysis data deprived from Scite.ai platform and the traditional citation indicators, this paper investigates the method of evaluating the academic influence of scientific journals. [Methods] Taking 42 international otorhinolaryngology journals as cases, combining 8 traditional citation indicators and 5 citation content analysis indicators, we used the method of multi-indicator fusion to construct a comprehensive evaluation model of journal impact. Through normalization processing, correlation analysis, reliability analysis, validity analysis, and factor analysis, we evaluated the macroscopical impact (F_macro) and microscopical impact (F_micro) of otorhinolaryngology journals respectively. [Findings] It is found that 8 traditional citation indicators present strong positive correlations, and the indicators are highly consistent overall and internally. There are high positive correlations among 4 citation indicators based on Scite.ai platform, and the SI (Scite Index) does not have correlation with the other 4 citation indicators. The correlation coefficient between F_macro and F_micro is 0.492, with a moderate positive correlation. It indicates that the evaluation of otolaryngology journals' microscopical impact based on Scite.ai indicators is a useful supplement to the evaluation of academic impact based on traditional citation. [Conclusions] The indicator SI is not appropriate for evaluating journal impact solely due to its week correlations with other Scite.ai indicators. It makes sense to combine the traditional citation indicators and citation content analysis data to evaluate academic impact.

Key words: Scite.ai, Content analysis of citation, Academic influence, Otolaryngology

摘要：

【目的】参考二维度融合的期刊影响力评价模型,探索利用人工智能引文内容分析平台——Scite.ai所提供的数据,结合传统引文指标开展科技期刊的影响力评价。【方法】以Web of Science数据库中耳鼻喉科学分类下收录的国际期刊为研究对象,对期刊的8项传统引文指标和5项引文内容分析指标,采用归一化处理、相关性分析、信度分析、效度分析和因素分析方法,分析期刊通过传统引文指标表现出来的宏观影响力和通过引文内容分析指标表现出来的微观影响力,最后融合两类影响力得到二维评价结果。【结果】8项传统引文指标之间的正相关性极显著,指标整体的一致性也较高;5项引文内容分析指标中,4项指标之间的正相关性极显著,引文支持型占比指标(Scite Index,SI)与其余4项指标不具有相关性,不适合单独用于期刊影响力评价;两类影响力间的相关性系数为0.492,呈中度正相关,引文内容分析指标可以作为传统引文指标评价体系的有益补充。【结论】由于不具有相关性,Scite.ai所提供的SI指标不适合单独用于期刊影响力评价;利用传统引文指标进行期刊影响力评价时有必要结合Scite.ai平台的其他引文内容分析指标以全面反映期刊的学术质量。

关键词: Scite.ai, 引文内容分析, 学术影响力, 耳鼻喉科学

LI Wei, HUANG Li. Impact evaluation of scientific journals based on citation content analysis data and citation indicators[J]. Chinese Journal of Scientific and Technical Periodicals, 2022, 33(8): 1119-1125.

李伟, 黄历. 结合引文内容分析数据的科技期刊影响力评价研究[J]. 中国科技期刊研究, 2022, 33(8): 1119-1125.

/ / Recommend / Download Citations

URL: https://www.cjstp.cn/EN/10.11946/cjstp.202203230197

https://www.cjstp.cn/EN/Y2022/V33/I8/1119

Figures/Tables 11

References 18

[1]	Garfield E. Citation indexes for science. A new dimension in documentation through association of ideas[J]. Science, 1955, 122(3159):108-111. doi: 10.1126/science.122.3159.108 URL
[2]	Garfield E. The history and meaning of the journal impact factor[J]. Journal of the American Medical Association, 2006, 295(1):90-93.
[3]	Waltman L. A review of the literature on citation impact indicators[J]. Journal of Informetrics, 2016, 10(2):365-391. doi: 10.1016/j.joi.2016.02.007 URL
[4]	Walters W H. Citation-based journal rankings:Key questions,metrics,and data sources[J]. IEEE Access, 2017, 5:22036-22053. doi: 10.1109/ACCESS.2017.2761400 URL
[5]	Kilicoglu H, Peng Z S, Tafreshi S, et al. Confirm or refute? A comparative study on citation sentiment classification in clinical research publications[J]. Journal of Biomedical Informatics, 2019, 91:103123. doi: 10.1016/j.jbi.2019.103123 URL
[6]	Shotton D. CiTO,the citation typing ontology[J]. Journal of Biomedical Semantics, 2010, 1(Suppl 1):S6. doi: 10.1186/2041-1480-1-S1-S6 URL
[7]	Xu J, Zhang Y Y, Wu Y H, et al. Citation sentiment analysis in clinical trial papers[J]. AMIA Annual Symposium Proceedings, 2015, 2015:1334-1341.
[8]	刘盛博, 丁堃, 张春博. 引文分析的新阶段:从引文著录分析到引用内容分析[J]. 图书情报知识, 2015(3):25-34.
[9]	王景周, 唐寅杉. 人工智能背景下学术论文的引用内容分析及其应用:以Scite.ai为例[J]. 中国编辑, 2021(10):62-67.
[10]	Rife S C, Rosati D, Nicholson J M. Scite:The next generation of citations[J]. Learned Publishing, 2021, 34:454-456. doi: 10.1002/leap.1379 URL
[11]	Nicholson J M, Mordaunt M, Lopez P, et al. Scite:A smart citation index that displays the context of citations and classifies their intent using deep learning[J]. Quantitative Science Studies, 2021, 2(3):882-898. doi: 10.1162/qss_a_00146 URL
[12]	Khamsi R. Coronavirus in context:Scite.ai tracks positive and negative citations for COVID-19 literature[J/OL]. (2020-05-01)[2021-05-06]. https://www.nature.com/articles/d41586-020-01324-6 URL
[13]	Bordignon F. Self-correction of science:A comparative study of negative citations and post-publication peer review[J]. Scientometrics, 2020, 124(2):1225-1239. doi: 10.1007/s11192-020-03536-z URL
[14]	Nicholson J M, Uppala A, Sieber M, et al. Measuring the quality of scientific references in Wikipedia:An analysis of more than 115M citations to over 800 000 scientific articles[J]. The FEBS Journal, 2021, 288(14):4242-4248. doi: 10.1111/febs.15608 URL
[15]	雍文明. 国外智能引用系统Scite介绍及其对我国科技期刊评价的启示[J]. 中国科技期刊研究, 2021, 32(9):1126-1134. doi: 10.11946/cjstp.202107090547
[16]	Rosati D. How are journals cited? Characterizing journal citations by type of citation[EB/OL]. (2021-02-22)[2021-05-06]. https://arxiv.org/abs/2102.11043 URL
[17]	Zhu X D, Turney P, Lemire D, et al. Measuring academic influence:Not all citations are equal[J]. Journal of the Association for Information Science and Technology, 2015, 66(2):408-427. doi: 10.1002/asi.23179 URL
[18]	Zhao R Y, Wang X, Liu Z Z, et al. Research on the impact evaluation of academic journals based on altmetrics and citation indicators[J]. Proceedings of the Association for Information Science and Technology, 2019, 56(1):336-345. doi: 10.1002/pra2.27 URL

编号	期刊简称	编号	期刊简称
1	Jama Otolaryngol	22	Int J Audiol
2	Int Forum Allergy Rh	23	Eur Ann Otorhinolary
3	Rhinology	24	Curr Opin Otolaryngo
4	Ear Hearing	25	J Voice
5	Otolaryng Head Neck	26	Auris Nasus Larynx
6	Dysphagia	27	Audiol Neuro-Otol
7	Clin Exp Otorhinolar	28	Braz J Otorhinolar
8	Otolaryng Clin N Am	29	Am J Otolaryng
9	Laryngoscope	30	Ent-Ear Nose Throat
10	Trends Hear	31	Int J Pediatr Otorhi
11	Hearing Res	32	J Am Acad Audiol
12	Head Neck-J Sci Spec	33	Ann Oto Rhinol Laryn
13	Jaro-J Assoc Res Oto	34	Orl J Oto-Rhino-Lary
14	Clin Otolaryngol	35	Acta Oto-Laryngol
15	Eur Arch Oto-Rhino-L	36	Am J Audiol
16	Am J Rhinol Allergy	37	Logop Phoniatr Voco
17	Laryngoscope Invest	38	J Laryngol Otol
18	J Otolaryngol-Head N	39	Hno
19	J Vestibul Res-Equil	40	Laryngo Rhino Otol
20	Otol Neurotol	41	J Int Adv Otol
21	Acta Otorhinolaryngo	42	Folia Phoniatr Logo

编号	期刊简称	编号	期刊简称
1	Jama Otolaryngol	22	Int J Audiol
2	Int Forum Allergy Rh	23	Eur Ann Otorhinolary
3	Rhinology	24	Curr Opin Otolaryngo
4	Ear Hearing	25	J Voice
5	Otolaryng Head Neck	26	Auris Nasus Larynx
6	Dysphagia	27	Audiol Neuro-Otol
7	Clin Exp Otorhinolar	28	Braz J Otorhinolar
8	Otolaryng Clin N Am	29	Am J Otolaryng
9	Laryngoscope	30	Ent-Ear Nose Throat
10	Trends Hear	31	Int J Pediatr Otorhi
11	Hearing Res	32	J Am Acad Audiol
12	Head Neck-J Sci Spec	33	Ann Oto Rhinol Laryn
13	Jaro-J Assoc Res Oto	34	Orl J Oto-Rhino-Lary
14	Clin Otolaryngol	35	Acta Oto-Laryngol
15	Eur Arch Oto-Rhino-L	36	Am J Audiol
16	Am J Rhinol Allergy	37	Logop Phoniatr Voco
17	Laryngoscope Invest	38	J Laryngol Otol
18	J Otolaryngol-Head N	39	Hno
19	J Vestibul Res-Equil	40	Laryngo Rhino Otol
20	Otol Neurotol	41	J Int Adv Otol
21	Acta Otorhinolaryngo	42	Folia Phoniatr Logo

指标	TC	JIF	5YIF	IFJSC	IX	ES	AIS	NE
TC	1
JIF	0.419^**	1
5YIF	0.409^**	0.914^**	1
IFJSC	0.362^*	0.970^**	0.895^**	1
IX	0.347^*	0.551^**	0.378^*	0.475^**	1
ES	0.929^**	0.559^**	0.532^**	0.530^**	0.376^*	1
AIS	0.311^*	0.842^**	0.954^**	0.858^**	0.252	0.449^**	1
NE	0.929^**	0.559^**	0.531^**	0.531^**	0.379^*	1.000^**	0.449^**	1

指标	TC	JIF	5YIF	IFJSC	IX	ES	AIS	NE
TC	1
JIF	0.419^**	1
5YIF	0.409^**	0.914^**	1
IFJSC	0.362^*	0.970^**	0.895^**	1
IX	0.347^*	0.551^**	0.378^*	0.475^**	1
ES	0.929^**	0.559^**	0.532^**	0.530^**	0.376^*	1
AIS	0.311^*	0.842^**	0.954^**	0.858^**	0.252	0.449^**	1
NE	0.929^**	0.559^**	0.531^**	0.531^**	0.379^*	1.000^**	0.449^**	1

指标	校正项总计相关性	项已删除的 α系数	Cronbach's α系数
TC	0.593	0.898
JIF	0.842	0.877
5YIF	0.753	0.883
IFJSC	0.809	0.880	0.901
IX	0.407	0.912
ES	0.733	0.885
AIS	0.706	0.888
NE	0.733	0.885

Please choose a citation manager

Content to export