As grain pests cause a major post-harvest loss in stored grains, early detection and monitoring of grain pest activities become necessary for applying appropriate actions to reduce storage losses. With the development of artificial intelligence, image detection methods based on deep learning have been widely used in agriculture. However, current research in stored grain pest detection is relatively limited. The quality of the dataset will determine the level of knowledge that deep learning models can learn. Therefore, constructing a specialized dataset for grain pest detection and counting is of great significance. The proposed dataset GrainPest includes 500 original images of grain insects, 500 pixel-level saliency annotation images, 420 files with insect bounding boxes and 500 entries of pest counts. The data set covers various grain pests such as corn weevil, wheat moth, grain beetle, and corn borer, as well as different types of grain backgrounds such as wheat, corn, and rice. Due to the fact that many grains are not infected with pests, the GrainPest also includes 80 pure grain background images without any pest, which bring more challenge for saliency detection. The GrainPest provides a benchmark dataset to promote the research of saliency detection, object detection, and pests counting in stored grains, and the work will provide support for reducing grain storage losses and ensuring food security.