With the development and application of artificial intelligence, computer vision, deep learning and other science and technology in the field of agriculture, the data-driven deep learning model for agriculture has become a new research paradigm for agricultural information extraction, and agricultural datasets are the basis for deep learning model training, and high-quality, large-scale, and diverse datasets can effectively improve the model performance, thus boosting the application of deep learning in the field of smart agriculture. To help researchers in related fields better understand the driving force of data for deep learning and give full play to the application of deep learning in the field of agriculture, this paper analyzes the datasets through metrology and summarizes the basic qualities of agricultural datasets such as type, scale, and source, which are divided into four categories according to the deep learning methods, such as target detection, image segmentation, and image recognition, and into seven categories according to the application areas, such as visual navigation, feature recognition, non-destructive testing and other 7 categories. The results show that the type of dataset is dominated by image data, and the data volume of images is concentrated in the range of 500 to 1500, and due to the specificity of agricultural data collection, most of the dataset is constructed by individuals and some of them are from public datasets, and the dataset is mainly utilized to carry out feature recognition. In the future, as the scale of the model becomes larger and larger, the requirements for the dataset are also upgraded, and it is necessary to continuously construct large-scale, balanced distribution, and accurately labeled datasets.In this paper, we provide a theoretical basis for data to promote deep learning agricultural applications by emphasizing the driving force and the importance of data to the deep learning model.