Research Article
Open Access
Reliability-Aware Active Crowdsourcing and Ontology-Constrained Multi-Evidence Clustering for Agricultural Multimedia Knowledge-Base Construction
1
National Engineering Research Center for Information Technology in Agriculture, Beijing 100101, China
2
Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China
3
Key Laboratory of Digital Village Technology, Ministry of Agriculture and Rural Affairs, Beijing 100125, China
4
Beijing Research Institute of Agricultural Artificial Intelligence and Robotics, Beijing 100097, China
* Corresponding Author:
Peng Hao,
[email protected]
Volume 2, Issue 4
2026
Received: 09 May 2026
Revised: 03 November 2026
Accepted: 05 November 2026
Published: 10 November 2026
You have full access to this open access article
·
CC BY 4.0 License
Article Information
Abstract
Constructing agricultural multimedia knowledge bases is complicated by heterogeneous evidence, noisy crowdsourced annotations, variations in sample difficulty, and inadequate integration of multisource information. Using tomato disease and pest data, this study proposes a unified framework comprising Multi-evidence Reliability-Aware Active Crowdsourcing (MRAC) and Ontology-Constrained Multi-Evidence Clustering and Refinement (OMCR). MRAC jointly models annotator reliability, sample difficulty, content-evidence quality, and evidence availability. It selects sample-annotator pairs according to the expected information gain per unit cost and produces calibrated soft-label posteriors. OMCR integrates frozen visual features, crowdsourcing posteriors, and agricultural ontology constraints, and uses class prototypes, pairwise constraints, and risk scores for sample clustering, mislabeled-instance detection, and label refinement. Under 30% synthetic annotation noise, MRAC achieved an Accuracy of 0.9732 and a Macro-F1 of 0.9676. MRAC-EIG achieved the highest mean Macro-F1 (0.9631) and the lowest mean relative annotation cost (1.0080) among the evaluated policies. The complete OMCR model achieved ACC, NMI, and ARI values of 0.9708, 0.9293, and 0.9500, respectively. The unified framework achieved Precision@5, Precision@10, and nDCG@10 values of 0.9251, 0.9223, and 0.9233, respectively, with a 14.97% relative improvement in nDCG@10 over visual-only retrieval. These results show that the proposed framework improves noisy-label aggregation, multi-evidence knowledge organization, and retrieval ranking while controlling annotation costs, offering a practical route to reliable agricultural multimedia knowledge-base construction.
Graphical Abstract
Keywords
agricultural multimedia knowledge construction
tomato diseases and pests
active crowdsourcing
multimodal reliability awareness
ontology-constrained clustering
image retrieval
Data Availability Statement
The data supporting the findings of this study are available from the corresponding author upon reasonable request.
Funding
This work was supported by the National Major Agricultural Science and Technology Project under Grant 0304.
Conflicts of Interest
The authors declare no conflicts of interest.
AI Use Statement
The authors declare that no generative AI was used in the preparation of this manuscript.
Ethical Approval and Consent to Participate
Not applicable.
References
- Benos, L., Tagarakis, A. C., Dolias, G., Berruto, R., Kateris, D., & Bochtis, D. (2021). Machine learning in agriculture: A comprehensive updated review. Sensors, 21(11), 3758.
[CrossRef] [Google Scholar] - Lv, B., Wu, H., Chen, W., Chen, C., Miao, Y., & Zhao, C. (2024). VEG-MMKG: Multimodal knowledge graph construction for vegetables based on pre-trained model extraction. Computers and electronics in agriculture, 226, 109398.
[CrossRef] [Google Scholar] - Liu, X., Bai, X., Wang, L., Ren, B., Lu, S., & Lin, L. (2019). Review and trend analysis of knowledge graphs for crop pest and diseases. IEEE Access, 7, 62251-62264.
[CrossRef] [Google Scholar] - Zhao, X., Chen, B., Ji, M., Wang, X., Yan, Y., Zhang, J., ... & Lv, C. (2024). Implementation of large language models and agricultural knowledge graphs for efficient plant disease detection. Agriculture, 14(8), 1359.
[CrossRef] [Google Scholar] - Dolatabadian, A., Neik, T. X., Danilevicz, M. F., Upadhyaya, S. R., Batley, J., & Edwards, D. (2025). Image-based crop disease detection using machine learning. Plant Pathology, 74(1), 18-38.
[CrossRef] [Google Scholar] - Zhu, H., Qin, S., Su, M., Lin, C., Li, A., & Gao, J. (2025). Harnessing large vision and language models in agriculture: A review. Frontiers in Plant Science, 16, 1579355.
[CrossRef] [Google Scholar] - Wang, L., Jin, T., Yang, J., Leonardis, A., Wang, F., & Zheng, F. (2024). Agri-llava: Knowledge-infused large multimodal assistant on agricultural pests and diseases. arXiv preprint arXiv:2412.02158.
[CrossRef] [Google Scholar] - Liu, X., Liu, Z., Hu, H., Chen, Z., Wang, K., Wang, K., & Lian, S. (2024, September). A multimodal benchmark dataset and model for crop disease diagnosis. In European conference on computer vision (pp. 157-170). Cham: Springer Nature Switzerland.
[CrossRef] [Google Scholar] - Zhang, M., Xu, Z., Wang, P., Li, R., Wang, L., Liu, Q., ... & Wu, S. (2026, May). Agridoctor: A multimodal intelligent assistant for agriculture. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2741-2745). IEEE.
[CrossRef] [Google Scholar] - Truong, G. H., Quoc, K. N., & Quach, L. D. (2026). TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases. arXiv preprint arXiv:2608.08727.
[CrossRef] [Google Scholar] - Zhang, J., Wu, M., Zhou, C., & Sheng, V. S. (2022). Active crowdsourcing for multilabel annotation. IEEE Transactions on Neural Networks and Learning Systems, 35(3), 3549-3559.
[CrossRef] [Google Scholar] - Majdi, M. S., & Rodriguez, J. J. (2023). Crowd-Certain: label aggregation in crowdsourced and ensemble learning classification. arXiv preprint arXiv:2310.16293.
[CrossRef] [Google Scholar] - Goh, H. W., & Mueller, J. (2023). Activelab: Active learning with re-labeling by multiple annotators. arXiv preprint arXiv:2301.11856.
[CrossRef] [Google Scholar] - Xu, J., Tang, H., Ren, Y., Peng, L., Zhu, X., & He, L. (2022, June). Multi-level feature learning for contrastive multi-view clustering. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 16030-16039). IEEE.
[CrossRef] [Google Scholar] - Chao, G., Jiang, Y., & Chu, D. (2024, March). Incomplete contrastive multi-view clustering with high-confidence guiding. In Proceedings of the AAAI conference on artificial intelligence (Vol. 38, No. 10, pp. 11221-11229).
[CrossRef] [Google Scholar] - Wang, K., Miao, Y., Wang, X., Li, Y., Li, F., & Song, H. (2024). Research on the construction of a knowledge graph for tomato leaf pests and diseases based on the named entity recognition model. Frontiers in Plant Science, 15, 1482275.
[CrossRef] [Google Scholar] - Yan, R., An, P., Meng, X., Li, Y., Li, D., Xu, F., & Dang, D. (2025). A knowledge graph for crop diseases and pests in China. Scientific Data, 12, 222.
[CrossRef] [Google Scholar] - Mohanty, S. P., Hughes, D. P., & Salathé, M. (2016). Using deep learning for image-based plant disease detection. Frontiers in plant science, 7, 215232.
[CrossRef] [Google Scholar] - Cooper, L., Walls, R. L., Elser, J., Gandolfo, M. A., Stevenson, D. W., Smith, B., Preece, J., Athreya, B., Mungall, C. J., & Rensing, S. (2013). The plant ontology as a tool for comparative plant anatomy and genomic analyses. Plant and Cell Physiology, 54(2), e1.
[CrossRef] [Google Scholar] - He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778).
[CrossRef] [Google Scholar] - Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321-1330). PMLR. http://proceedings.mlr.press/v70/guo17a.html
[Google Scholar] - Chen, Z., Jiang, L., & Li, C. (2022). Label augmented and weighted majority voting for crowdsourcing. Information Sciences, 606, 397-409.
[CrossRef] [Google Scholar] - Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1), 20-28.
[CrossRef] [Google Scholar] - Whitehill, J., Wu, T., Bergsma, J., Movellan, J., & Ruvolo, P. (2009). Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, 22.
[Google Scholar] - Zhang, H., Li, S., Zeng, D., Yan, C., & Ge, S. (2024). Coupled confusion correction: Learning from crowds with sparse annotations. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15), 16732-16740.
[CrossRef] [Google Scholar] - Chu, Z., Ma, J., & Wang, H. (2021). Learning from crowds by modeling common confusions. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7), 5832-5840.
[CrossRef] [Google Scholar]
Cite This Article
APA Style
Li, H., Zhu, Z., Ren, G., Miao, Y., & Hao, P. (2026). Reliability-Aware Active Crowdsourcing and Ontology-Constrained Multi-Evidence Clustering for Agricultural Multimedia Knowledge-Base Construction. Digital Intelligence in Agriculture, 2(4), 214-229. https://doi.org/10.62762/DIA.2026.512829
Export Citation
RIS Format
Compatible with EndNote, Zotero, Mendeley, and other reference managers
TY - JOUR AU - Li, Hongda AU - Zhu, Zixin AU - Ren, Guanxu AU - Miao, Yisheng AU - Hao, Peng PY - 2026 DA - 2026/11/10 TI - Reliability-Aware Active Crowdsourcing and Ontology-Constrained Multi-Evidence Clustering for Agricultural Multimedia Knowledge-Base Construction JO - Digital Intelligence in Agriculture T2 - Digital Intelligence in Agriculture JF - Digital Intelligence in Agriculture VL - 2 IS - 4 SP - 214 EP - 229 DO - 10.62762/DIA.2026.512829 UR - https://www.icck.org/article/abs/dia.2026.512829 KW - agricultural multimedia knowledge construction KW - tomato diseases and pests KW - active crowdsourcing KW - multimodal reliability awareness KW - ontology-constrained clustering KW - image retrieval AB - Constructing agricultural multimedia knowledge bases is complicated by heterogeneous evidence, noisy crowdsourced annotations, variations in sample difficulty, and inadequate integration of multisource information. Using tomato disease and pest data, this study proposes a unified framework comprising Multi-evidence Reliability-Aware Active Crowdsourcing (MRAC) and Ontology-Constrained Multi-Evidence Clustering and Refinement (OMCR). MRAC jointly models annotator reliability, sample difficulty, content-evidence quality, and evidence availability. It selects sample-annotator pairs according to the expected information gain per unit cost and produces calibrated soft-label posteriors. OMCR integrates frozen visual features, crowdsourcing posteriors, and agricultural ontology constraints, and uses class prototypes, pairwise constraints, and risk scores for sample clustering, mislabeled-instance detection, and label refinement. Under 30% synthetic annotation noise, MRAC achieved an Accuracy of 0.9732 and a Macro-F1 of 0.9676. MRAC-EIG achieved the highest mean Macro-F1 (0.9631) and the lowest mean relative annotation cost (1.0080) among the evaluated policies. The complete OMCR model achieved ACC, NMI, and ARI values of 0.9708, 0.9293, and 0.9500, respectively. The unified framework achieved Precision@5, Precision@10, and nDCG@10 values of 0.9251, 0.9223, and 0.9233, respectively, with a 14.97% relative improvement in nDCG@10 over visual-only retrieval. These results show that the proposed framework improves noisy-label aggregation, multi-evidence knowledge organization, and retrieval ranking while controlling annotation costs, offering a practical route to reliable agricultural multimedia knowledge-base construction. SN - 3069-3187 PB - Institute of Central Computation and Knowledge LA - English ER -
BibTeX Format
Compatible with LaTeX, BibTeX, and other reference managers
@article{Li2026Reliabilit,
author = {Hongda Li and Zixin Zhu and Guanxu Ren and Yisheng Miao and Peng Hao},
title = {Reliability-Aware Active Crowdsourcing and Ontology-Constrained Multi-Evidence Clustering for Agricultural Multimedia Knowledge-Base Construction},
journal = {Digital Intelligence in Agriculture},
year = {2026},
volume = {2},
number = {4},
pages = {214-229},
doi = {10.62762/DIA.2026.512829},
url = {https://www.icck.org/article/abs/dia.2026.512829},
abstract = {Constructing agricultural multimedia knowledge bases is complicated by heterogeneous evidence, noisy crowdsourced annotations, variations in sample difficulty, and inadequate integration of multisource information. Using tomato disease and pest data, this study proposes a unified framework comprising Multi-evidence Reliability-Aware Active Crowdsourcing (MRAC) and Ontology-Constrained Multi-Evidence Clustering and Refinement (OMCR). MRAC jointly models annotator reliability, sample difficulty, content-evidence quality, and evidence availability. It selects sample-annotator pairs according to the expected information gain per unit cost and produces calibrated soft-label posteriors. OMCR integrates frozen visual features, crowdsourcing posteriors, and agricultural ontology constraints, and uses class prototypes, pairwise constraints, and risk scores for sample clustering, mislabeled-instance detection, and label refinement. Under 30\% synthetic annotation noise, MRAC achieved an Accuracy of 0.9732 and a Macro-F1 of 0.9676. MRAC-EIG achieved the highest mean Macro-F1 (0.9631) and the lowest mean relative annotation cost (1.0080) among the evaluated policies. The complete OMCR model achieved ACC, NMI, and ARI values of 0.9708, 0.9293, and 0.9500, respectively. The unified framework achieved Precision@5, Precision@10, and nDCG@10 values of 0.9251, 0.9223, and 0.9233, respectively, with a 14.97\% relative improvement in nDCG@10 over visual-only retrieval. These results show that the proposed framework improves noisy-label aggregation, multi-evidence knowledge organization, and retrieval ranking while controlling annotation costs, offering a practical route to reliable agricultural multimedia knowledge-base construction.},
keywords = {agricultural multimedia knowledge construction, tomato diseases and pests, active crowdsourcing, multimodal reliability awareness, ontology-constrained clustering, image retrieval},
issn = {3069-3187},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
Scan to read this article