ICCK Journal of Image Analysis and Processing Home About Editors Current Issue
Knee Osteoarthritis Severity Detection Using Multimodal Data
Research Article  ·  Published: 10 September 2026
Issue cover
ICCK Journal of Image Analysis and Processing
Volume 2, Issue 4, 2026: 206-217
Research Article Open Access

Knee Osteoarthritis Severity Detection Using Multimodal Data

by
Ayesha Malik Ayesha Malik 1
Rabbia Mahum Rabbia Mahum 1 * ORCID
1 Department of Computer Science, University of Engineering and Technology (UET) Taxila, Taxila 47050, Pakistan
* Corresponding Author: Rabbia Mahum, [email protected]
Volume 2, Issue 4
Volume 2, Issue 4 2026
Received: 26 May 2025 Revised: 20 July 2026 Accepted: 21 July 2026 Published: 10 September 2026 CrossMark
Download PDF 2.08 MB Cite Metrics
DOI: 10.62762/JIAP.2026.275643
ARK: ark:/57805/jiap.2026.275643
You have full access to this open access article · CC BY 4.0 License

Article Information

Published in ICCK Journal of Image Analysis and Processing
Volume/Issue Volume 2, Issue 4, 2026
Pages 206-217

Abstract

Osteoarthritis of the knee (KOA) is one of the main causes of disability; hence, it requires precise and early evaluation of the severity level of the disease. In this paper, we propose a multimodal deep learning architecture based on a self-supervised Swin transformer combined with a cross-modal attention mechanism (SWIN-MULTI-ATTEN) for combining radiological images and patients' information (age, sex, and BMI). This framework captures both structural and contextual information to achieve better classification accuracy. Experimental results on the Osteoarthritis Initiative (OAI) dataset show that the proposed model achieves an accuracy of 91.4% and a QWK of 0.903, which outperforms other CNN-based and transformer-based architectures.

Graphical Abstract

Knee Osteoarthritis Severity Detection Using Multimodal Data

Keywords

Deep learning Transformers Multi-attention Osteoarthritis

Data Availability Statement

The raw knee radiograph data and clinical metadata analysed in this study are derived from the Osteoarthritis Initiative (OAI) database, which is publicly available at https://nda.nih.gov/oai/. The processed data subsets and model code generated during this study are not publicly available due to institutional policies but may be made available from the corresponding author upon reasonable request.

Funding

This work was supported without any funding.

Conflicts of Interest

Rabbia Mahum served as an Associate Editor of the ICCK Journal of Image Analysis and Processing at the time of manuscript submission. To ensure the integrity of the peer-review process, Rabbia Mahum was not involved in the editorial handling, peer review, or decision-making process for this manuscript, which was handled independently by another editor. The remaining authors declare no conflicts of interest.

AI Use Statement

The authors declare that no generative AI was used in the preparation of this manuscript.

Ethical Approval and Consent to Participate

The Osteoarthritis Initiative (OAI) study was approved by the institutional review boards at each of the participating clinical sites and the coordinating center, and written informed consent was obtained from all original study participants prior to enrolment. The present study constitutes a secondary analysis of publicly available, de-identified OAI data and did not involve the recruitment of new participants or the collection of additional personal data; accordingly, separate ethical approval and informed consent were not required for this analysis.

References

  1. Hunter, D. J., & Bierma-Zeinstra, S. (2019). Osteoarthritis. The Lancet, 393(10182), 1745-1759.
    [CrossRef] [Google Scholar]
  2. World Health Organization. (2023). Osteoarthritis. Retrieved from https://www.who.int/news-room/fact-sheets/detail/osteoarthritis
    [Google Scholar]
  3. Kellgren, J. H., & Lawrence, J. S. (1957). Radiological assessment of osteo-arthrosis. Annals of the Rheumatic Diseases, 16(4), 494-502.
    [CrossRef] [Google Scholar]
  4. Guermazi, A., Hayashi, D., Roemer, F. W., & Felson, D. T. (2013). Osteoarthritis: A review of strengths and weaknesses of different imaging options. Rheumatic Disease Clinics of North America, 39(3), 567-591.
    [CrossRef] [Google Scholar]
  5. Shen, D., Wu, G., & Suk, H. I. (2017). Deep learning in medical image analysis. Annual Review of Biomedical Engineering, 19, 221-248.
    [CrossRef] [Google Scholar]
  6. Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P., & Saarakkala, S. (2018). Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Scientific reports, 8(1), 1727.
    [CrossRef] [Google Scholar]
  7. Thomas, K. A., Kidziński, Ł., Halilaj, E., Fleming, S. L., Venkataraman, G. R., Oei, E. H., ... & Delp, S. L. (2020). Automated classification of radiographic knee osteoarthritis severity using deep neural networks. Radiology: Artificial Intelligence, 2(2), e190065.
    [CrossRef] [Google Scholar]
  8. Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., \dots & \ S\'anchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60-88.
    [CrossRef] [Google Scholar]
  9. Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018). Convolutional neural networks: an overview and application in radiology. Insights into imaging, 9(4), 611-629.
    [CrossRef] [Google Scholar]
  10. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770-778).
    [CrossRef] [Google Scholar]
  11. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
    [CrossRef] [Google Scholar]
  12. Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & M\"uller, K. R. (2021). Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3), 247-278.
    [CrossRef] [Google Scholar]
  13. Takahashi, S., Sakaguchi, Y., Kouno, N., Takasawa, K., Ishizu, K., Akagi, Y., ... & Hamamoto, R. (2024). Comparison of vision transformers and convolutional neural networks in medical image analysis: A systematic review. Journal of medical systems, 48(1), 84.
    [CrossRef] [Google Scholar]
  14. Wang, Z., Chetouani, A., & Jennane, R. (2023, September). Transformer with Selective Shuffled Position Embedding for Early Detection of Knee Osteoarthritis. In Proceedings of the 20th International Conference on Content-based Multimedia Indexing (pp. 261-264).
    [CrossRef] [Google Scholar]
  15. Jahan, M., Hasan, M. Z., Samia, I. J., Fatema, K., Rony, M. A. H., Arefin, M. S., & Moustafa, A. (2024). KOA-CCTNet: An enhanced knee osteoarthritis grade assessment framework using modified compact convolutional transformer model. IEEE Access, 12, 107719-107741.
    [CrossRef] [Google Scholar]
  16. Patel, U., Patel, S., Patel, D., Jain, N., Patel, S., & Gangavani, R. (2026). Hybrid CNN-transformer for knee osteoarthritis severity grading. Journal of Innovative Image Processing, 8(2), 617-643.
    [CrossRef] [Google Scholar]
  17. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021, October). Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF international conference on computer vision (ICCV) (pp. 9992-10002). IEEE.
    [CrossRef] [Google Scholar]
  18. Sudha, K., & Kannan, A. R. (2026). Enhanced Swin transformer with dual attention for knee osteoarthritis severity grading from X-ray images. Scientific Reports, 16, 10617.
    [CrossRef] [Google Scholar]
  19. Huang, Y., Shao, Z., Wang, R., & Qiu, H. (2026). Clinically integrated multi-modal transformer framework with cross-modal gated fusion and clinical nomogram for automated Kellgren-Lawrence grading of knee osteoarthritis on x-ray images. BMC Musculoskeletal Disorders, 27(1), 528.
    [CrossRef] [Google Scholar]
  20. Roy, B., Das, J., Baidya, A., & Banik, D. (2026). Osteotrans-Net: An interpretable transformer framework for knee osteoarthritis severity grading in X-ray scans. Biomedical Signal Processing and Control, 113, 109096.
    [CrossRef] [Google Scholar]
  21. Zhang, C., Chen, S., Cigdem, O., Rajamohan, H. R., Cho, K., Kijowski, R., & Deniz, C. M. (2025). MR-Transformer: A Vision Transformer-based Deep Learning Model for Total Knee Replacement Prediction Using MRI. Radiology: Artificial Intelligence, 7(5), e240373.
    [CrossRef] [Google Scholar]
  22. Shamir, L., Ling, S. M., Scott, W., Hochberg, M., Ferrucci, L., & Goldberg, I. G. (2009). Early detection of radiographic knee osteoarthritis using computer-aided analysis. Osteoarthritis and Cartilage, 17(10), 1307-1312.
    [CrossRef] [Google Scholar]
  23. Tiulpin, A., Klein, S., Bierma-Zeinstra, S. M., Thevenot, J., Rahtu, E., Meurs, J. V., ... & Saarakkala, S. (2019). Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. Scientific reports, 9(1), 20038.
    [CrossRef] [Google Scholar]
  24. Mahum, R., Rehman, S. U., Meraj, T., Rauf, H. T., Irtaza, A., El-Sherbeeny, A. M., & El-Meligy, M. A. (2021). A novel hybrid approach based on deep CNN features to detect knee osteoarthritis. Sensors, 21(18), 6189.
    [CrossRef] [Google Scholar]
  25. Pi, S. W., Lee, B. D., Lee, M. S., & Lee, H. J. (2023). Ensemble deep-learning networks for automated osteoarthritis grading in knee X-ray images. Scientific Reports, 13(1), 22887.
    [CrossRef] [Google Scholar]
  26. Zhao, H., Ou, L., Zhang, Z., Zhang, L., Liu, K., & Kuang, J. (2025). The value of deep learning-based X-ray techniques in detecting and classifying KL grades of knee osteoarthritis: a systematic review and meta-analysis. European Radiology, 35(1), 327-340.
    [CrossRef] [Google Scholar]
  27. Esmez, O., Deniz, G., Bilek, F., Gurger, M., Barua, P. D., Dogan, S., ... & Tuncer, T. (2025). TurkerNeXtV2: An Innovative CNN Model for Knee Osteoarthritis Pressure Image Classification. Diagnostics, 15(19), 2478.
    [CrossRef] [Google Scholar]
  28. Pan, J., Wu, Y., Tang, Z., Sun, K., Li, M., Sun, J., ... & Shen, B. (2024). Automatic knee osteoarthritis severity grading based on X-ray images using a hierarchical classification method. Arthritis Research & Therapy, 26(1), 203.
    [CrossRef] [Google Scholar]
  29. Ren, X., Hou, L., Liu, S., Wu, P., Liang, S., Fu, H., ... & Cheng, Y. (2025). OA-MEN: a fusion deep learning approach for enhanced accuracy in knee osteoarthritis detection and classification using X-Ray imaging. Frontiers in Bioengineering and Biotechnology, 12, 1437188.
    [CrossRef] [Google Scholar]
  30. Amjad, J., Sajid, M. Z., Amjad, A., Hamid, M. F., Youssef, A., & Sharif, M. I. (2025). Knee osteoarthritis detection and classification using autoencoders and extreme learning machines. AI, 6(7), 151.
    [CrossRef] [Google Scholar]
  31. Hu, J., Zheng, C., Yu, Q., Zhong, L., Yu, K., Chen, Y., ... & Zhang, X. (2023). DeepKOA: a deep-learning model for predicting progression in knee osteoarthritis using multimodal magnetic resonance images from the osteoarthritis initiative. Quantitative Imaging in Medicine and Surgery, 13(8), 4852.
    [CrossRef] [Google Scholar]
  32. Lavanya, R., & Senthilkumar, S. (2026). Deep learning in knee osteoarthritis: A task-based systematic review of recent advances. Biomedical Signal Processing and Control, 119, 109771.
    [CrossRef] [Google Scholar]
  33. Rani, S., Rout, A., Soni, P., Gupta, M., Kumar, N., & Kumar, K. (2026). Review of CNN-Based Approaches for Preprocessing, Segmentation and Classification of Knee Osteoarthritis. Diagnostics, 16(3), 461.
    [CrossRef] [Google Scholar]
  34. Yeoh, P. S. Q., Lai, K. W., Goh, S. L., Hasikin, K., Wu, X., & Li, P. (2023). Transfer learning-assisted 3D deep learning models for knee osteoarthritis detection: Data from the osteoarthritis initiative. Frontiers in Bioengineering and Biotechnology, 11, 1164655.
    [CrossRef] [Google Scholar]
  35. Teoh, Y. X., Othmani, A., Lai, K. W., Goh, S. L., & Usman, J. (2023). Stratifying knee osteoarthritis features through multitask deep hybrid learning: data from the osteoarthritis initiative. Computer methods and programs in biomedicine, 242, 107807.
    [CrossRef] [Google Scholar]
  36. Subha, B., Jeyakumar, V., & Deepa, S. N. (2024). Gaussian Aquila optimizer based dual convolutional neural networks for identification and grading of osteoarthritis using knee joint images. Scientific Reports, 14(1), 7225.
    [CrossRef] [Google Scholar]
  37. Yao, W., Bai, J., Liao, W., Chen, Y., Liu, M., & Xie, Y. (2024). From cnn to transformer: A review of medical image segmentation models. Journal of Imaging Informatics in Medicine, 37(4), 1529-1547.
    [CrossRef] [Google Scholar]
  38. Ju, R. Y., & Cai, W. (2023). Fracture detection in pediatric wrist trauma X-ray images using YOLOv8 algorithm. Scientific reports, 13(1), 20077.
    [CrossRef] [Google Scholar]
  39. Prezja, F., Paloneva, J., Pölönen, I., Niinimäki, E., & Äyrämö, S. (2022). DeepFake knee osteoarthritis X-rays from generative adversarial neural networks deceive medical experts and offer augmentation potential to automatic classification. Scientific Reports, 12(1), 18573.
    [CrossRef] [Google Scholar]
  40. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
    [Google Scholar]
  41. Huang, S. C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020). Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, 3(1), 136.
    [CrossRef] [Google Scholar]
  42. Dihin, R. A., Al Shemmary, E. N., & Al-Jawher, W. A. M. (2023). Implementation of the swin transformer and its application in image classification. Journal Port Science Research, 6(4), 318-331.
    [CrossRef] [Google Scholar]
  43. Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., ... & Hu, H. (2022, June). Simmim: A simple framework for masked image modeling. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 9643-9653). IEEE.
    [CrossRef] [Google Scholar]
  44. Ullah, Z., & Kim, J. (2026). VL-OrdinalFormer: Vision–language-guided ordinal transformers for interpretable knee osteoarthritis grading. Mathematics, 14(6), 963.
    [CrossRef] [Google Scholar]
  45. Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220.
    [CrossRef] [Google Scholar]
  46. Shamir, L., Ling, S. M., Scott, W. W., Bos, A., Orlov, N., Macura, T. J., ... & Goldberg, I. G. (2008). Knee x-ray image analysis method for automated detection of osteoarthritis. IEEE Transactions on Biomedical Engineering, 56(2), 407-415.
    [CrossRef] [Google Scholar]

Cite This Article

APA Style
Malik, A., & Mahum, R. (2026). Knee Osteoarthritis Severity Detection Using Multimodal Data. ICCK Journal of Image Analysis and Processing, 2(4), 206-217. https://doi.org/10.62762/JIAP.2026.275643
Export Citation
RIS Format
Compatible with EndNote, Zotero, Mendeley, and other reference managers
TY  - JOUR
AU  - Malik, Ayesha
AU  - Mahum, Rabbia
PY  - 2026
DA  - 2026/09/10
TI  - Knee Osteoarthritis Severity Detection Using Multimodal Data
JO  - ICCK Journal of Image Analysis and Processing
T2  - ICCK Journal of Image Analysis and Processing
JF  - ICCK Journal of Image Analysis and Processing
VL  - 2
IS  - 4
SP  - 206
EP  - 217
DO  - 10.62762/JIAP.2026.275643
UR  - https://www.icck.org/article/abs/JIAP.2026.275643
KW  - Deep learning
KW  - Transformers
KW  - Multi-attention
KW  - Osteoarthritis
AB  - Osteoarthritis of the knee (KOA) is one of the main causes of disability; hence, it requires precise and early evaluation of the severity level of the disease. In this paper, we propose a multimodal deep learning architecture based on a self-supervised Swin transformer combined with a cross-modal attention mechanism (SWIN-MULTI-ATTEN) for combining radiological images and patients' information (age, sex, and BMI). This framework captures both structural and contextual information to achieve better classification accuracy. Experimental results on the Osteoarthritis Initiative (OAI) dataset show that the proposed model achieves an accuracy of 91.4% and a QWK of 0.903, which outperforms other CNN-based and transformer-based architectures.
SN  - 3068-6679
PB  - Institute of Central Computation and Knowledge
LA  - English
ER  -
BibTeX Format
Compatible with LaTeX, BibTeX, and other reference managers
@article{Malik2026Knee,
  author = {Ayesha Malik and Rabbia Mahum},
  title = {Knee Osteoarthritis Severity Detection Using Multimodal Data},
  journal = {ICCK Journal of Image Analysis and Processing},
  year = {2026},
  volume = {2},
  number = {4},
  pages = {206-217},
  doi = {10.62762/JIAP.2026.275643},
  url = {https://www.icck.org/article/abs/JIAP.2026.275643},
  abstract = {Osteoarthritis of the knee (KOA) is one of the main causes of disability; hence, it requires precise and early evaluation of the severity level of the disease. In this paper, we propose a multimodal deep learning architecture based on a self-supervised Swin transformer combined with a cross-modal attention mechanism (SWIN-MULTI-ATTEN) for combining radiological images and patients' information (age, sex, and BMI). This framework captures both structural and contextual information to achieve better classification accuracy. Experimental results on the Osteoarthritis Initiative (OAI) dataset show that the proposed model achieves an accuracy of 91.4\% and a QWK of 0.903, which outperforms other CNN-based and transformer-based architectures.},
  keywords = {Deep learning, Transformers, Multi-attention, Osteoarthritis},
  issn = {3068-6679},
  publisher = {Institute of Central Computation and Knowledge}
}

Article Metrics

Citations
Crossref
0
Scopus
0
Views
47
PDF Downloads
7

Publisher's Note

ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and Permissions

CC BY Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
ICCK Journal of Image Analysis and Processing
ICCK Journal of Image Analysis and Processing
ISSN: 3068-6679 (Online)
Portico
Preserved at
Portico

Article QR Code

Article QR Code
Scan to read this article
Submit Manuscript Edit a Special Issue

Popular Articles

Share This Article