基于空间特征融合ViT的枸杞虫害细粒度分类方法

doi:10.19788/j.issn.2096-6369.000066

农业大数据学报 ›› 2024, Vol. 6 ›› Issue (4): 522-531.doi: 10.19788/j.issn.2096-6369.000066

基于空间特征融合ViT的枸杞虫害细粒度分类方法

孙露露¹(), 刘建平¹^,²^,^*(), 周国民³^,⁴, 王健⁵, 刘立波⁶

1.北方民族大学计算机科学与工程学院，银川 750021
2.图像图形智能处理国家民委重点实验室，银川 750021
3.农业农村部南京农业机械化研究所，南京 210014
4.中国农业科学院农业信息研究所国家农业科学数据中心，北京 100081
5.中国农业科学院农业信息研究所，北京 100091
6.宁夏大学信息工程学院，银川 750021

收稿日期:2024-09-09 接受日期:2024-10-14 出版日期:2024-12-26 发布日期:2024-12-02
通讯作者: 刘建平，E-mail：liujianping01@nmu.edu.cn。
作者简介:孙露露，E-mail：20227515@stu.nmu.edu.cn。
基金资助:
国家自然科学基金项目(32460444);北方民族大学重点科研项目（(2023ZRLG12);北方民族大学研究生创新项目(YCX23168)

Spatial Feature Fusion-Based ViT Method for Fine-Grained Classification of Wolfberry Pests

SUN LuLu¹(), LIU JianPing¹^,²^,^*(), ZHOU GuoMin³^,⁴, WANG Jian⁵, LIU LiBo⁶

1. College of Computer Science and Engineering, North Minzu University, Yinchuan 750021, China
2. The Key Laboratory of Images and Graphics Intelligent Processing of State Ethnic Affairs Commission, Yinchuan 750021, China
3. Nanjing Institute of Agricultural Mechanization, Ministry of Agriculture and Rural Affairs, Nanjing 210014, China
4. National Agriculture Science Data Center, Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China
5. Agricultural Information institute of CAAS, Beijing 100091, China
6. School of Information Engineering Ningxia University, Yinchuan 750021, China

Received:2024-09-09 Accepted:2024-10-14 Published:2024-12-26 Online:2024-12-02

摘要/Abstract

摘要：

针对枸杞种植中面临的虫害细粒度分类难题，提出了一种农业虫害细粒度分类模型——基于空间特征融合的数据增强视觉Transformer（ESF-ViT）。首先，该模型利用自注意力机制裁剪出前景目标的图片以增强图像输入，补充更多细节表示；其次，结合自注意力机制与图卷积网络提取害虫区域的空间信息，学习害虫的空间姿态特征。为了验证本文所提模型的有效性，在CUB-200-2011、IP102以及宁夏枸杞虫害数据集WPIT9K上开展实验研究，实验结果表明本文所提方法相比基础ViT模型分别提升了1.83%、2.09%和2.01%，并且超越了现有最新的虫害分类模型。所提模型能够有效地解决农业虫害识别领域中的细粒度虫害图像分类问题，为虫害的高效监测预警提供视觉模型。

关键词: 枸杞, 视觉 Transformer, 细粒度图像分类, 空间特征融合, 数据增强

Abstract:

To address the fine-grained pest classification challenge faced in wolfberry cultivation, we propose an agricultural pest fine-grained classification model—Spatial Feature Fusion-based Data Augmented Visual Transformer (ESF-ViT). The model first utilizes the self-attention mechanism to crop images of the foreground targets to enhance image input and supplement more detailed representations. Secondly, it combines the self-attention mechanism with a Graph Convolutional Network (GCN) to extract spatial information from the pest regions, learning the spatial posture features of the pests. To validate the effectiveness of the proposed model, we conducted experimental research on the CUB-200-2011, IP102, and Ningxia wolfberry pest dataset WPIT9K. The experimental results show that the proposed method outperforms the baseline ViT model by 1.83%, 2.09%, and 2.01% respectively, and surpasses the existing state-of-the-art pest classification models. The proposed model effectively solves the fine-grained pest image classification problem in the field of agricultural pest recognition, providing a visual model for efficient pest monitoring and early warning.

Key words: wolfberry berry, vision transformer, fine-grained image classification, spatial feature fusion, data augmentation

孙露露, 刘建平, 周国民, 王健, 刘立波. 基于空间特征融合ViT的枸杞虫害细粒度分类方法[J]. 农业大数据学报, 2024, 6(4): 522-531.

SUN LuLu, LIU JianPing, ZHOU GuoMin, WANG Jian, LIU LiBo. Spatial Feature Fusion-Based ViT Method for Fine-Grained Classification of Wolfberry Pests[J]. Journal of Agricultural Big Data, 2024, 6(4): 522-531.

图/表 10

图1

表1

图2

图3

图4

表2

表3

表4

表5

图5

参考文献 37

[1]	LEHMANN P, AMMUNET T, BARTON M, et al. Complex responses of global insect pests to climate warming[J]. Frontiers in Ecology and the Environment, 2020, 18(3):141-150.
[2]	HADDI K, TURCHEN L, JUMBO L, et al. Rethinking biorational insecticides for the pest management: Unintended effects and consequences[J]. Pest Management Science. 2020, 76(7):2286-2293.
[3]	FILHO F, HELDENS W, KONG Z, et al. Drones: Innovative technology for use in precision pest management[J]. Journal of Economic Entomology, 2020, 113(1):1-25. doi: 10.1093/jee/toz268 pmid: 31811713
[4]	THENMOZHI K, REDDY U. Crop pest classification based on deep convolutional neural network and transfer learning[J]. Computers and Electronics in Agriculture, 2019, 164:104906.
[5]	周国民. 迎接农业农村领域数字经济的提速发展[J]. 农业大数据学报, 2023, 5(1): 1-1. doi: 10.19788/j.issn.2096-6369.230101
[6]	DAWEI W, LIMIAO D, JIANGONG N, et al. Recognition pest by image‐based transfer learning[J]. Journal of the Science of Food and Agriculture, 2019, 99(10): 4524-4531.
[7]	JIN X, TAO Z, KONG J. Multi-stream aggregation network for fine-grained crop pests and diseases image recognition[J]. International Journal of Cybernetics and Cyber-Physical Systems, 2020, 1(1):52-67.
[8]	周国民. 我国农业大数据应用进展综述[J]. 农业大数据学报, 2019, 1(1): 16-23. doi: 10.19788/j.issn.2096-6369.190102
[9]	张凌栩, 韩锐, 李文明, 等. 大数据深度学习系统研究进展与典型农业应用[J]. 农业大数据学报, 2019, 1(2): 88-104. doi: 10.19788/j.issn.2096-6369.190208
[10]	YANG G, CHEN G, LI C, et al. Convolutional rebalancing network for the classification of large imbalanced rice pest and disease datasets in the field[J]. Frontiers in Plant Science, 2021, 12:671134.
[11]	LIU J, WANG X, MIAO W, et al. Tomato pest recognition algorithm based on improved yolov4[J]. Frontiers in Plant Science, 2022, 13: 814681.
[12]	LIU B, DING Z, TIAN L, et al. Grape leaf disease identification using improved deep convolutional neural networks[J]. Frontiers in Plant Science, 2020, 11: 1082. doi: 10.3389/fpls.2020.01082 pmid: 32760419
[13]	GU Y, YIN H, JIN D, et al. Image-based hot pepper disease and pest diagnosis using transfer learning and finetuning[J]. Frontiers in Plant Science. 2021, 12: 724487.
[14]	DAI G, FAN J, DEWI C. ITF-WPI: Image and text based cross-modal feature fusion model for wolfberry pest recognition[J]. Computers and Electronics in Agriculture. 2023, 212:108129.
[15]	YANG G, HE Y, YANG Y, et al. Fine-grained image classification for crop disease based on attention mechanism[J]. Frontiers in Plant Science, 2020, 11: 600854.
[16]	YANG J, ZHANG F, QIAN T. Attention-based hierarchical convolution neural network for fine-grained crop image classification[C]// 2020 International Conferences on Internet of Things, 2020: 106-112.
[17]	ZHANG X, GAO H, WAN L. Classification of fine-grained crop disease by dilated convolution and improved channel attention module[J]. Agriculture, 2020, 12(10):1727.
[18]	ZENG Q, NIU L, WANG S, et al. SEViT: a large-scale and fine-grained plant disease classification model based on transformer and attention convolution[J]. Multimedia Systems, 2022, 29(3): 1001-1010.
[19]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[OL]. arXiv:2010.11929. DOI:10.48550/arXiv.2010.11929.
[20]	WAH C, BRANSON S, WELINDER P, et al. The caltech-ucsd birds-200-2011 dataset[J]. 2011.
[21]	WU X, ZHAN C, LAI Y, et al. Ip102: A large-scale benchmark dataset for insect pest recognition[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 8779-8788.
[22]	陈磊, 刘立波, 王晓丽. 2020年宁夏枸杞虫害图文跨模态检索数据集[J]. 中国科学数据, 2022, 7(3):149-156.
[23]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017.
[24]	孙露露, 刘建平, 王健, 等. 细粒度图像分类上Vision Transformer的发展综述[J]. 计算机工程与应用, 2024, 60(10):30-46. doi: 10.3778/j.issn.1002-8331.2310-0395
[25]	BERA A, WHARTON Z, LIU Y, et al. SR-GNN: Spatial relation-aware graph neural network for fine-grained image categorization[C]// IEEE Transactions on Image Processing, 2022. 31: 6017-6031. DOI: 10.1109/TIP.2022.3205215.
[26]	LIU H, ZHANG C, XIE B, et al. Affinity relation-aware fine-grained bird image recognition for robot vision tracking via transformers[C]// 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), Jinghong, China, 2022: 662-667. DOI: 10.1109/ROBIO55434.2022.10011861.
[27]	ZHANG Z C, CHEN Z D, WANG Y, et al. ViT-FOD: A vision transformer based fine-grained object discriminator[OL].arXiv: 2203. 12816.
[28]	XU Q, WANG J, JIANG B, et al. Fine-grained visual classification via internal ensemble learning transformer[J]. IEEE Transactions on Multimedia, 2023, 25:9015-9028. DOI: 10.1109/TMM.2023.3244340.
[29]	LIU H, ZHANG C, DENG Y, et al. TransIFC: Invariant cues-aware feature concentration learning for efficient fine-grained bird image classification[OL]. IEEE Transactions on Multimedia. DOI: 10.1109/TMM.2023.3238548.
[30]	李佳盈, 蒋文婷, 杨林, 等. 基于ViT的细粒度图像分类[J]. 计算机工程与设计, 2023, 44(3):916-921.
[31]	WANG Q, WANG J, DENG H, et al. AA-trans: Core attention aggregating transformer with information entropy selector for fine-grained visual classification[J]. Pattern Recognition, 2023, 140: 109547. https://doi.org/10.1016/j.patcog.2023.109547.
[32]	SUN H, HE X, PENG Y. SIM-Trans: Structure information modeling transformer for fine-grained visual categorization[C]// Proceedings of the 30th ACM International Conference on Multimedia. 2022: 5853-5861.
[33]	HE J, CHEN J, LIU S, et al. TransFG: A transformer architecture for fine-grained recognition[C]// Proceedings of the AAAI conference on artificial intelligence. 2022, 36(1): 852-860.
[34]	WANG J, YU X, GAO Y. Feature fusion vision transformer for fine-grained visual categorization[J]. arXiv preprint arXiv:2107.02341, 2021.
[35]	HU X, ZHU S, PENG T. Hierarchical attention vision transformer for fine-grained visual classification[J]. Journal of Visual Communication and Image Representation, 2023. 91: 103755. https://doi.org/10.1016/j.jvcir.2023.103755.
[36]	DIAO Q, JIANG Y, WEN B, et al. Metaformer: A unified meta framework for fine-grained recognition[OL]. arXiv:2203.02751.
[37]	TOUVRON H, CORD M, DOUZE M, et al. Training data-efficient image transformers distillation through attention[OL]. arXiv:2012.12877.

病虫害类似	任务	主干网络	优势	劣势
水稻病虫害^[10]	普通分类	CRN	解决类别不平衡	计算量大
番茄虫害^[11]	普通分类	YOLOv4	实时性强	数据集少
葡萄叶片病害^[12]	普通分类	DICNN	收敛速度快	泛化能力弱
辣椒病虫害^[13]	普通分类	ResNet	高层特征提取	手动剪裁图像
枸杞虫害^[14]	普通分类	CNN+Transformer	跨模态特征融合	场景单一
作物叶片病害^[15]	细粒度	NASNetLarge	适用自动检测	作物种类少
茶叶质量等级^[16]	细粒度	H-CNN	提取层次特征	类别小
作物叶片病害^[17]	细粒度	ResNet	提取关键特征	泛化能力弱
作物疾病^[18]	细粒度	SEViT	泛化能力强	网络复杂

	CUB-200-2011	IP102	WPIT9K
Train	9421	15183	7644
Test	2367	3800	1852
Images	11788	18983	9496
Category	200	96	17

模型	Backbone	CUB-200-2011	IP102	WPIT9K
CAL	ResNet-101	89.16%	75.26%	98.75%
ViT	ViT-B-16	90.61%	75.63%	97.93%
TransFG	ViT-B-16	91.76%	76.99%	99.89%
SIM	ViT-B-16	91.76%	77.14%	99.89%
FFVT	ViT-B-16	91.61%	77.14%	99.84%
HAVT	ViT-B-16	91.84%	77.48%	99.74%
IELT	ViT-B-16	91.87%	76.61%	99.84%
ESF-ViT	ViT-B-16	92.44%	77.72%	99.94%

Model	CUB-200-2011	WPIT9K
Baseline	90.61%	97.93%
Baseline +数据增强	91.76%	99.74%
Baseline +数据增强+空间信息学习	92.28%	99.97%
Baseline +数据增强+空间信息学习+特征融合	92.44%	99.94%

阈值	IP102
0.2-0.4	77.42%
0.3-0.5	77.67%
0.4-0.6	77.72%
0.5-0.7	77.64%
0.6-0.8	77.48%

基于空间特征融合ViT的枸杞虫害细粒度分类方法

Spatial Feature Fusion-Based ViT Method for Fine-Grained Classification of Wolfberry Pests

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 37

相关文章 1

Metrics

本文评价

推荐阅读 0