An Explainable Hybrid Deep Learning Framework with Multi-Scale Feature Fusion and Uncertainty-Aware Clinical Decision Support for Brain Tumor Classification
Article Information
Abstract
Accurate diagnosis of brain tumors from magnetic resonance imaging (MRI) is crucial but is challenged by inter-observer variability and by the limited explainability of deep models. Most existing studies use standalone CNNs or Transformers without uncertainty assessment or explanation capabilities. We propose an explainable hybrid deep learning framework that combines multi-scale feature fusion with uncertainty-aware clinical decision support. On a four-class brain MRI benchmark, it was compared with EfficientNetV2, ConvNeXt, and Swin Transformer. Five explainability methods (Grad-CAM, SHAP, LIME, Integrated Gradients, and Attention Rollout) explain predictions, while Monte Carlo dropout quantifies predictive uncertainty; confidence and entropy form a Trust Score for three-tier referral. The hybrid model outperformed the baselines, achieving a test accuracy of 95.7%, a macro F1-score of 95.5%, and an AUC of 0.989. A preliminary 12-reader evaluation suggests that the clinical decision support system (CDSS) report is usable (mean System Usability Scale 78.3/100) and reduces mean reading time per case from 42.6~s to 34.8~s versus radiologist-alone reading, although underpowered for clinical endpoints. Prospective multi-institutional validation remains future work.
Graphical Abstract
Keywords
Data Availability Statement
Funding
Conflicts of Interest
AI Use Statement
Ethical Approval and Consent to Participate
References
- Dai, Y., Gao, Y., & Liu, F. (2021). TransMed: Transformers advance multi-modal medical image classification. Diagnostics, 11(8), 1384.
[CrossRef] [Google Scholar] - Wang, W., Chen, C., Ding, M., Yu, H., Zha, S., & Li, J. (2021). TransBTS: Multimodal brain tumor segmentation using transformer. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 (pp. 109-119). Springer.
[CrossRef] [Google Scholar] - Noreen, N., Palaniappan, S., Qayyum, A., Ahmad, I., Imran, M., & Shoaib, M. (2020). A deep learning model based on concatenation approach for the diagnosis of brain tumor. IEEE Access, 8, 55135-55144.
[CrossRef] [Google Scholar] - Díaz-Pernas, F. J., Martínez-Zarzuela, M., Antón-Rodríguez, M., & González-Ortega, D. (2021, February). A deep learning approach for brain tumor classification and segmentation using a multiscale convolutional neural network. In Healthcare (Vol. 9, No. 2, p. 153). MDPI.
[CrossRef] [Google Scholar] - Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020). An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ digital medicine, 3(1), 17.
[CrossRef] [Google Scholar] - Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., ... & Kittler, H. (2020). Human-computer collaboration for skin cancer recognition. Nature Medicine, 26(8), 1229-1234.
[CrossRef] [Google Scholar] - Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1050-1059). PMLR. https://proceedings.mlr.press/v48/gal16.html
[Google Scholar] - Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., ... & Snoek, J. (2019). Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019).
[Google Scholar] - Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
[Google Scholar] - Jiang, H., Kim, B., Guan, M., & Gupta, M. (2018). To trust or not to trust a classifier. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018).
[Google Scholar] - Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 618-626).
[CrossRef] [Google Scholar] - Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017) (pp. 4765-4774).
[Google Scholar] - Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). ``Why should I trust you?'' Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135-1144).
[CrossRef] [Google Scholar] - Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 3319-3328). PMLR. https://proceedings.mlr.press/v70/sundararajan17a.html
[Google Scholar] - Abnar, S., & Zuidema, W. (2020). Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4190-4197).
[CrossRef] [Google Scholar] - Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1409.1556
[Google Scholar] - He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778).
[CrossRef] [Google Scholar] - Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2261-2269).
[CrossRef] [Google Scholar] - Tan, M., & Le, Q. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (pp. 6105-6114). PMLR. https://proceedings.mlr.press/v97/tan19a.html
[Google Scholar] - Roy, S., Koehler, G., Ulrich, C., Baumgartner, M., Petersen, J., Isensee, F., ... & Maier-Hein, K. H. (2023). MedNeXt: Transformer-driven scaling of ConvNets for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2023 (pp. 405-415). Springer.
[CrossRef] [Google Scholar] - Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2010.11929
[Google Scholar] - Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9992-10002).
[CrossRef] [Google Scholar] - Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021, July). Training data-efficient image transformers & distillation through attention. In International conference on machine learning (pp. 10347-10357). PMLR. https://proceedings.mlr.press/v139/touvron21a
[Google Scholar] - Liu, Z., Mao, H., Wu, C. Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022, June). A convnet for the 2020s. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 11966-11976). IEEE.
[CrossRef] [Google Scholar] - Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023). Transformers in medical imaging: A survey. Medical Image Analysis, 88, 102802.
[CrossRef] [Google Scholar] - Asiri, A. A., Shaf, A., Ali, T., Pasha, M. A., Khan, A., Irfan, M., ... & Alamri, S. (2024). Advancing brain tumor detection: Harnessing the Swin Transformer's power for accurate classification and performance analysis. PeerJ Computer Science, 10, e1867.
[CrossRef] [Google Scholar] - Van der Velden, B. H., Kuijf, H. J., Gilhuijs, K. G., & Viergever, M. A. (2022). Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Medical image analysis, 79, 102470.
[CrossRef] [Google Scholar] - Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., ... & Kalpathy-Cramer, J. (2021). Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiology: Artificial Intelligence, 3(6), e200267.
[CrossRef] [Google Scholar] - Chefer, H., Gur, S., & Wolf, L. (2021). Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 782-791).
[CrossRef] [Google Scholar] - Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020). Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 180-186).
[CrossRef] [Google Scholar] - Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018).
[Google Scholar] - Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., & Kapur, T. (2020). Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Transactions on Medical Imaging, 39(12), 3868-3878.
[CrossRef] [Google Scholar] - Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., ... & Nahavandi, S. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76, 243-297.
[CrossRef] [Google Scholar] - Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
[Google Scholar] - Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019). Causability and explainability of artificial intelligence in medicine. Wiley interdisciplinary reviews: data mining and knowledge discovery, 9(4), e1312.
[CrossRef] [Google Scholar] - Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC medicine, 17(1), 195.
[CrossRef] [Google Scholar] - Roy, A. G., Navab, N., & Wachinger, C. (2018). Concurrent spatial and channel `squeeze & excitation' in fully convolutional networks. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 (pp. 421-429). Springer.
[CrossRef] [Google Scholar] - Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018). Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11), e1002683.
[CrossRef] [Google Scholar] - Nickparvar, M. (2021). Brain tumor MRI dataset [Data set]. Kaggle. Retrieved from https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset
[Google Scholar] - Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
[CrossRef] [Google Scholar] - Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019). CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6022-6031).
[CrossRef] [Google Scholar] - Tan, M., & Le, Q. (2021). EfficientNetV2: Smaller models and faster training. In Proceedings of the 38th International Conference on Machine Learning (pp. 10096-10106). PMLR. https://proceedings.mlr.press/v139/tan21a.html
[Google Scholar] - Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7132-7141).
[CrossRef] [Google Scholar] - Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018, September). Cbam: Convolutional block attention module. In European conference on computer vision (pp. 3-19). Cham: Springer International Publishing.
[CrossRef] [Google Scholar] - Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).
[CrossRef] [Google Scholar] - Johnson, J. M., & Khoshgoftaar, T. M. (2019). Survey on deep learning with class imbalance. Journal of big data, 6(1), 27.
[CrossRef] [Google Scholar] - Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1711.05101
[Google Scholar] - Loshchilov, I., & Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1608.03983
[Google Scholar] - Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(2), 281-305. https://jmlr.org/papers/v13/bergstra12a.html
[Google Scholar] - Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2623-2631).
[CrossRef] [Google Scholar] - Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., ... & Van Leemput, K. (2015). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10), 1993-2024.
[CrossRef] [Google Scholar] - Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., ... & Jambawalikar, S. R. (2018). Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629. https://arxiv.org/abs/1811.02629
[Google Scholar] - McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282). PMLR. https://proceedings.mlr.press/v54/mcmahan17a.html
[Google Scholar] - Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. https://arxiv.org/abs/1503.02531
[Google Scholar]
Cite This Article
TY - JOUR AU - Mirza, Mubashshera AU - Rasool, Saleem Pasha Choudhari AU - Baig, Mohammed Salman AU - Hassan, Saadiah AU - Ahmad, Rehan AU - Pandey, Anish PY - 2026 DA - 2026/11/18 TI - An Explainable Hybrid Deep Learning Framework with Multi-Scale Feature Fusion and Uncertainty-Aware Clinical Decision Support for Brain Tumor Classification JO - ICCK Transactions on Emerging Topics in Artificial Intelligence T2 - ICCK Transactions on Emerging Topics in Artificial Intelligence JF - ICCK Transactions on Emerging Topics in Artificial Intelligence VL - 3 IS - 4 SP - 267 EP - 305 DO - 10.62762/TETAI.2026.693634 UR - https://www.icck.org/article/abs/tetai.2026.693634 KW - brain tumor classification KW - explainable AI KW - hybrid CNN-Transformer KW - clinical decision support KW - multi-scale feature fusion KW - MRI KW - Grad-CAM AB - Accurate diagnosis of brain tumors from magnetic resonance imaging (MRI) is crucial but is challenged by inter-observer variability and by the limited explainability of deep models. Most existing studies use standalone CNNs or Transformers without uncertainty assessment or explanation capabilities. We propose an explainable hybrid deep learning framework that combines multi-scale feature fusion with uncertainty-aware clinical decision support. On a four-class brain MRI benchmark, it was compared with EfficientNetV2, ConvNeXt, and Swin Transformer. Five explainability methods (Grad-CAM, SHAP, LIME, Integrated Gradients, and Attention Rollout) explain predictions, while Monte Carlo dropout quantifies predictive uncertainty; confidence and entropy form a Trust Score for three-tier referral. The hybrid model outperformed the baselines, achieving a test accuracy of 95.7%, a macro F1-score of 95.5%, and an AUC of 0.989. A preliminary 12-reader evaluation suggests that the clinical decision support system (CDSS) report is usable (mean System Usability Scale 78.3/100) and reduces mean reading time per case from 42.6~s to 34.8~s versus radiologist-alone reading, although underpowered for clinical endpoints. Prospective multi-institutional validation remains future work. SN - 3068-6652 PB - Institute of Central Computation and Knowledge LA - English ER -
@article{Mirza2026An,
author = {Mubashshera Mirza and Saleem Pasha Choudhari Rasool and Mohammed Salman Baig and Saadiah Hassan and Rehan Ahmad and Anish Pandey},
title = {An Explainable Hybrid Deep Learning Framework with Multi-Scale Feature Fusion and Uncertainty-Aware Clinical Decision Support for Brain Tumor Classification},
journal = {ICCK Transactions on Emerging Topics in Artificial Intelligence},
year = {2026},
volume = {3},
number = {4},
pages = {267-305},
doi = {10.62762/TETAI.2026.693634},
url = {https://www.icck.org/article/abs/tetai.2026.693634},
abstract = {Accurate diagnosis of brain tumors from magnetic resonance imaging (MRI) is crucial but is challenged by inter-observer variability and by the limited explainability of deep models. Most existing studies use standalone CNNs or Transformers without uncertainty assessment or explanation capabilities. We propose an explainable hybrid deep learning framework that combines multi-scale feature fusion with uncertainty-aware clinical decision support. On a four-class brain MRI benchmark, it was compared with EfficientNetV2, ConvNeXt, and Swin Transformer. Five explainability methods (Grad-CAM, SHAP, LIME, Integrated Gradients, and Attention Rollout) explain predictions, while Monte Carlo dropout quantifies predictive uncertainty; confidence and entropy form a Trust Score for three-tier referral. The hybrid model outperformed the baselines, achieving a test accuracy of 95.7\%, a macro F1-score of 95.5\%, and an AUC of 0.989. A preliminary 12-reader evaluation suggests that the clinical decision support system (CDSS) report is usable (mean System Usability Scale 78.3/100) and reduces mean reading time per case from 42.6~s to 34.8~s versus radiologist-alone reading, although underpowered for clinical endpoints. Prospective multi-institutional validation remains future work.},
keywords = {brain tumor classification, explainable AI, hybrid CNN-Transformer, clinical decision support, multi-scale feature fusion, MRI, Grad-CAM},
issn = {3068-6652},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
Portico