Abstract:

[Purposes] To explore the application capabilities of four large language models (LLMs) in key information extraction and summarization of medical papers, providing empirical references for the technical pathways of knowledge services in STM journals. [Methods] One hundred research articles published in National Medical Journal of China were selected randomly.Using prompt engineering, ChatGPT 4o, Kimi, ChatGLM 4.0, and iFLYTEK Spark were employed to extract information in JSON format from the papers. The LLMs’ abilities in knowledge extraction, text comprehension, and summarization were evaluated.[Findings] All models returned accurate JSON-format data successfully, demonstrating high accuracy in extracting information such as study sample, sample size, disease, research type, discipline, and keywords. The models also performed well in summary generation, though their understanding of research methods was suboptimal. [Conclusions] The study indicates that LLMs possess strong capabilities in text comprehension, knowledge extraction, and summarization, but certain shortcomings remain. Overcoming these technical challenges could enable GenAI to play a significant role in STM journal dissemination, knowledge services, and decision-making support in vertical domains.

Key words: Scientific journal, Knowledge indexing, Large language models, Generative AI, Knowledge services

摘要：

【目的】 探讨4种大模型技术在科技期刊论文关键信息提取与总结中的应用能力,为科技期刊知识服务技术路径的探索提供实证参考。【方法】 随机选取《中华医学杂志》100篇研究型文献,通过提示语工程利用ChatGPT 4o、Kimi、ChatGLM 4、星火认知大模型从文本中以JSON方式提取信息,并评价各大模型知识抽取、文本理解及总结能力。【结果】 所有大模型均返回准确的JSON格式数据,在提取研究对象、样本量、疾病、研究类型、学科和主题词等信息时,表现出较高的准确性。在概要总结能力上也表现良好,仅在研究方法的理解方面表现不佳。【结论】 大模型具备较强的文本理解、知识提取和总结能力,但也存在一些不足。若能克服技术难点,GenAI有望在科技期刊的内容传播、知识服务以及垂直领域的决策支持等方面发挥重要作用。

关键词: 科技期刊, 知识标引, 大语言模型, 生成式人工智能, 知识服务