多模态知识图谱在农业中的研究进展

doi:10.19788/j.issn.2096-6369.220320

摘要/Abstract

摘要：

多模态知识图谱在传统知识图谱基础上，构建多种模态的实体，以及多模态实体之间的语义关系，能够提供重要的文本、图像以及声音知识，在消除歧义、补充视觉知识等方面具有非常重要的作用。近年来，在农业信息化和智能化飞速发展的背景下，知识图谱技术得到了广泛关注。文章详细介绍了知识图谱和多模态的含义，从图谱构建层面详细叙述了多模态表示学习等技术方法。针对多模态知识图谱在农业领域中的应用，重点总结知识图谱在农业智能问答、病虫害识别、农产品推荐等在农业领域中的应用研究，并对农业多模态知识图谱构建面临的挑战以及多模态知识图谱在农业领域中的发展前景进行展望与分析。

关键词: 多模态, 知识图谱, 问答, 病虫害识别, 农业信息化

Abstract:

Incorporating entities of multiple modalities and their semantic relationships on the basis of traditional knowledge graph, multimodal knowledge graph provides important information in the form of text, image and sound. It plays an important role in eliminating ambiguity and supplementing visual knowledge. In recent years, under the background of the rapid development of agricultural informatization and intelligence, knowledge graph technology has attracted extensive attention. In this article, the concepts of knowledge graph and multimodality are introduced in detail. Meanwhile, technical methods such as multimodal representation learning are elaborated from the perspective of graph construction. For the applications of multimodal knowledge graph in agriculture, we focus on the research of agricultural intelligent question answering system, plant diseases and pests’ identification, agricultural product recommendation and so on. At the same time, the challenges in construction and development of agricultural multimodal knowledge graphs are prospected and analyzed.

Key words: multi-model, knowledge graph, question answering, pest identification, agriculture informatization

中图分类号:

Q939

陈佳云, 徐向英, 章永龙, 周烨, 汪红江, 谭昌伟. 多模态知识图谱在农业中的研究进展[J]. 农业大数据学报, 2022, 4(3): 126-134.

Jiayun Chen, Xiangying Xu, Yonglong Zhang, Ye Zhou, Hongjiang Wang, Changwei Tan. Research Progress of Multimodal Knowledge Graph in Agriculture[J]. Journal of Agricultural Big Data, 2022, 4(3): 126-134.

图/表 3

参考文献 52

1	曹冰雪,李瑾,冯献,等.我国智慧农业的发展现状、路径与对策建议[J].农业现代化研究,2021,42(05):785-794.
	Cao B X, Li j, Feng X, et al. Development status, path, and countermeasures of smart agriculture in China[J]. Research of Agricultural Modernization, 2021,42(05): 785-794.
2	杨艺,朱翠明,王霞.我国农业信息化建设存在的问题、成因与发展对策研究[J].情报科学,2019,37(05):53-57.
	Yang Y, Zhu C M, Wang X. Research on the Problems, Causes and Development Strategies of Agricultural Informatization in China[J]. Information Science, 2019,37(05):53-57.
3	Qin H, Yao Y. Agriculture Knowledge Graph Construction and Application. Journal of Physics: Conference Series. IOP Publishing, 2021, 1756(1): 012010.
4	Yogish D, Manjunath T N, Hegadi R S. Survey on trends and methods of an intelligent answering system. 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT). IEEE, 2017: 346-353.
5	Jusoh, Shaidah, Hejab M, et al. Semantic extraction from texts. of International Conference on Computer Engineering and Applications IPCSIT, Singapore. 2011: 520-525.
6	Lu J, Wu D, Mao M, et al. Recommender system application developments: a survey[J]. Decision Support Systems, 2015, 74: 12-32.
7	Bollacker K, Evans C, Paritosh P, et al. Freebase: a collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 2008: 247-1250.
8	Fabian M, Gjergji K, Gerhard W. Yago: A core of semantic knowledge unifying wordnet and wikipedia.16th International world wide web conference, .
9	Michael K, Ivica L, Juhl J L, et al. The SIDER database of drugs and side effects[J]. Nucleic Acids Research, 2016,(D1): D1075-D1079.
10	Elizabeth, Blakesley, Lindsay. The Internet Movie Database (IMDb)[J]. Electronic Resources Review, 1997, 3(5):56-57.
11	Cheng L Q, Qing S, Peng Z Z, et al. Cn-MAKG: China meteorology and agriculture knowledge graph construction based on semi-structured data. 2018 IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS). IEEE, 2018: 692-696.
12	Chen Y, Kuang J, Cheng D, et al. AgriKG: an agricultural knowledge graph and its applications. International conference on database systems for advanced applications. Springer, Cham, 2019: 533-537.
13	张海瑜,陈庆龙,张斯静,等.基于语义知识图谱的农业知识智能检索方法[J].农业机械学报,2021,52(S1):156-163.
	Zhang H Y, Chen Q L, Zhang S J, et al. lntelligent Retrieval Method of Agricultural Knowledge Based on Semantic Knowledge Graph [J]. Transactions of the Chinese Society for Agricultural Machinery, 2021,52(S1):156-163.
14	Xie R, Liu Z, Luan H, et al. "Image-embodied Knowledge Representation Learning." Proceedings of the Twenty-Sixth International Joint Conference on Artificial IntelligenceMain track, 2016: 3140-3146.
15	陈烨,周刚,卢记仓.多模态知识图谱构建与应用研究综述[J].计算机应用研究,2021,38(12):3535-3543.
	Chen Y, Zhou G, Lu J C. Survey on construction and application research for multi-modal knowledge graphs[J]. Application Research of Computers, 2021,38(12):3535-3543.
16	Wang M, Wang H, Qi G, et al. Richpedia: a large-scale, comprehensive multi-modal knowledge graph[J]. Big Data Research, 2020, 22: 100159.
17	Liu Y, Li H, Garcia-Duran A, et al. MMKG: multi-modal knowledge graphs. European Semantic Web Conference. Springer, Cham, 2019: 459-474.
18	Wilcke W., Bloem P., Boer V.de, et al. End-to-end entity classification on multimodal knowledge graphs. arXiv preprint arXiv:, 2020.
19	Sun R, Cao X, Zhao Y, et al. Multi-modal knowledge graphs for recommender systems. Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020: 1405-1414.
20	Sergieh H M, Botschen T, Gurevych I, et al. A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. 2018: 225-234.
21	Pezeshkpour P, Chen L, Singh S. Embedding Multimodal Relational Data for Knowledge Base Completion[J].Association for Computing Machinery. 2018: 1405–1414.
22	刘建伟, 丁熙浩, 罗雄麟. 多模态深度学习综述[J]. 计算机应用研究, 2020,37(6): 1601-1614.
	Liu J W, Ding X H, Luo X L. Survey of multimodal deep learning [J]. Application Research of Computers, 2020,37(6): 1601-1614.
23	孙影影,贾振堂,朱昊宇.多模态深度学习综述[J].计算机工程与应用,2020,56(21):1-10.
	Sun Y Y, Jia Z T, Zhu H Y. Survey of Multimodal Deep Learning [J]. Computer Engineering and Applications, 2020,56(21):1-10.
24	Han X, Liu Z, Sun M. Joint Representation Learning of Text and Knowledge for Knowledge Graph Completion[J]. CoRR,2016, abs/1611.04125.
25	Yang M, Zhang L, Zhang D, et al. Relaxed collaborative representation for pattern classification. Conference on Computer Vision and Pattern Recognition. IEEE, 2012: 2224-2231.
26	Zaremba Wojciech, Sutskever Ilya, Vinyals Oriol. Recurrent Neural Network Regularization[J]. CoRR,2014, abs/1409.2329.
27	Shi X, Chen Z, Wang H, et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting[J]. Advances in neural information processing systems, 2015: 802-810.
28	Devlin Jacob, Chang Ming-Wei, Lee Kenton, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
29	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017: 6000-6010.
30	Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[J]. Advances in neural information processing systems, 2012: 1106-1114.
31	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations, 2015: 1-14.
32	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
33	Logan B. Mel frequency cepstral coefficients for music modeling. In International Symposium on Music Information Retrieval. 2000: 1-13.
34	Wong E, Sridharan S. Comparison of linear prediction cepstrum coefficients and mel-frequency cepstrum coefficients for language identification. Proceedings of 2001 International Symposium on Intelligent Multimedia, Video and Speech Processing. ISIMP, 2001: 95-98.
35	Manjunath, Bangalore S, Salembier Philippe, et al. Introduction to MPEG-7: multimedia content description interface. John Wiley & Sons, 2002, 1-396.
36	夏迎春. 基于知识图谱的农业知识服务系统研究[D].安徽农业大学,2018.
	Xia Y C. Agriculture Knowledge Service System Based on Knowledge Graph[D]. Anhui Agricultural University, 2018.
37	李岩,胡文岭.基于知识图谱的农业知识问答系统研究[J].智慧农业导刊,2021,1(11):20-22.
	Li Y, Hu W L. Research on Agricultural Knowledge Question and Answer System Based on Knowledge Graph[J]. Journal of Smart Agriculture, 2021,1(11):20-22.
38	Antol S, Agrawal A, Lu J, et al. VQA: Visual Question Answering. 2015 IEEE International Conference on Computer Vision (ICCV), 2015: 2425-2433.
39	周子豪. 基于知识图谱的茶叶知识问答系统研究与实现[D].山东农业大学,2021.
	Zhou Z H. Research and Implementation of Tea Knowledge Question Answering System Based on Knowledge Graph[D]. SHANDONG AGRICULTURAL UNIVERSITY, 2021.
40	Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data[J]. Advances in neural information processing systems, 2013, 26: 1-9.
41	张颖,陈桂芬.基于Citespace的土壤肥力知识图谱可视化挖掘与分析[J].中国农机化学报,2016,37(03):209-213+229.
	Zhang Y, Chen G F. Citespace-based visualized mining and analysis on knowledge mapping of soil fertility[J]. Journal of Chinese Agricultural Mechanization, 2016,37(03):209-213+229.
42	翟肇裕,曹益飞,徐焕良,等.农作物病虫害识别关键技术研究综述[J].农业机械学报,2021,52(7):1-18.
	Zhai Z Y, Cao Y F, Xu H L, et al. Review of Key Techniques for Crop Disease and Pest Detection [J]. Transactions of the Chinese Society for Agricultural Machinery, 2021,52(7):1-18.
43	邹修国.基于计算机视觉的农作物病虫害识别研究现状[J].计算机系统应用,2011,20(6):238-242.
	Zhou X G. Research Status of Crop Pests Recognition over Computer Vision [J]. Computer Systems & Applications, 2011,20(6):238-242.
44	于合龙,沈金梦,毕春光,等.基于知识图谱的水稻病虫害智能诊断系统[J].华南农业大学学报,2021,42(05):105-116.
	Yu H L, Shen J M, Bi C G, et al. Intelligent diagnostic system for rice diseases and pests based on knowledge graph [J]. Journal of South China Agricultural University, 2021,42(05):105-116.
45	Guan L, Zhang J, Geng C. Diagnosis of Fruit Tree Diseases and Pests Based on Agricultural Knowledge Graph. Journal of Physics: Conference Series. IOP Publishing, 2021, 1865(4): 042052.
46	邹修国.基于计算机视觉的农作物病虫害识别研究现状[J].计算机系统应用,2011,20(06):238-242.
	Zou X G. Research Status of Crop Pests Recognition over Computer Vision[J]. Computer Systems & Applications, 2011,20(06):238-242.
47	Choudhary N K, Chukkapalli S S L, Mittal S, et al. Yieldpredict: A crop yield prediction framework for smart farms. 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020: 2340-2349.
48	Chukkapalli S S L, Ranade P, Mittal S, et al. A Privacy Preserving Anomaly Detection Framework for Cooperative Smart Farming Ecosystem[J]. UMBC Student Collection, 2021: 340-347.
49	马波, 田军仓. 作物生长模拟模型研究综述[J].节水灌溉,2010(02):1-5.
	Ma B, Tian J C. A Review on Crop Growth Simulation Model Research[J]. Water Saving Irrigation,2010(02):1-5.
50	Das D, Sahoo L, Datta S. A survey on recommendation system[J]. International Journal of Computer Applications, 2017, 160(7): 6-10.
51	Sharma L, Gera A. A survey of recommendation system: Research challenges[J]. international journal of engineering trends & technology, 2013, 4(5): 1989-1992.
52	Tejaswini H, MM M P, Pai R M. Knowledge Graph for Aquaculture Recommendation System. 2021 IEEE Mysore Sub Section International Conference (MysuruCon). IEEE, 2021: 366-371.

[1]	吕新, 梁斌, 张立福, 马富裕, 王海江, 刘阳春, 高攀, 张泽, 侯彤瑜. 新疆生产建设兵团棉花生产大数据平台建设与探索[J]. 农业大数据学报, 2020, 2(1): 70-78.
[2]	许哲平,邵曾婷,朱学军,王昉,王媛媛,肖曼,马克平. 农业生物多样性大数据平台建设研究和展望[J]. 农业大数据学报, 2019, 1(2): 76-87.