AI Enabled Resource-Constrained Computing Architectures for IoT Devices

Ishfaq Ahmad Malik; Gousia Habib

doi:10.62762/TMI.2025.225921

Article Information

Published in ICCK Transactions on Machine Intelligence

Volume/Issue Volume 1, Issue 3, 2025

Pages 138-147

Cited by 4 (Crossref) 4 (Scopus)

Abstract

Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.

Graphical Abstract

AI Enabled Resource-Constrained Computing Architectures for IoT Devices

Keywords

knowledge distillation high-performance computing transformers DistilBERT self-attention mechanism

Data Availability Statement

Not applicable.

Funding

This work was supported without any funding.

Conflicts of Interest

The authors declare no conflicts of interest.

Ethical Approval and Consent to Participate

Not applicable.

References

Khan, M. A., Hamila, R., Erbad, A., & Gabbouj, M. (2022). Distributed inference in resource-constrained IoT for real-time video surveillance. IEEE Systems Journal, 17(1), 1512-1523.
[CrossRef] [Google Scholar]
Deng, L., Li, G., Han, S., Shi, L., & Xie, Y. (2020). Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4), 485-532.
[CrossRef] [Google Scholar]
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
[CrossRef] [Google Scholar]
Zhang, L., Chen, X., Tu, X., Wan, P., Xu, N., & Ma, K. (2022, June). Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12454-12464). IEEE.
[CrossRef] [Google Scholar]
Mansourian, A. M., Ahmadi, R., Ghafouri, M., Babaei, A. M., Golezani, E. B., Ghamchi, Z. Y., ... & Kasaei, S. (2025). A Comprehensive Survey on Knowledge Distillation. arXiv preprint arXiv:2503.12067.
[CrossRef] [Google Scholar]
Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., & Zhang, Q. (2022). Cross-image relational knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12319-12328).
[CrossRef] [Google Scholar]
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
[CrossRef] [Google Scholar]
Menghani, G. (2023). Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12), 1-37.
[CrossRef] [Google Scholar]
Setyanto, A., Sasongko, T. B., Fikri, M. A., Ariatmanto, D., Agastya, I. M. A., Rachmanto, R. D., ... & Kim, I. K. (2025). Knowledge Distillation in Object Detection for Resource-Constrained Edge Computing. IEEE Access.
[CrossRef] [Google Scholar]
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2022). Efficient acceleration of deep learning inference on resource-constrained edge devices: A review. Proceedings of the IEEE, 111(1), 42-91.
[CrossRef] [Google Scholar]
Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
[CrossRef] [Google Scholar]
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019) (pp. 4171--4186).
[CrossRef] [Google Scholar]
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
[CrossRef] [Google Scholar]
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
[CrossRef] [Google Scholar]
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., ... & Liu, Q. (2020, November). Tinybert: Distilling bert for natural language understanding. In Findings of the association for computational linguistics: EMNLP 2020 (pp. 4163-4174).
[CrossRef] [Google Scholar]
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018, November). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353-355).
[CrossRef] [Google Scholar]
Nityasya, M. N., Wibowo, H. A., Chevi, R., Prasojo, R. E., & Aji, A. F. (2022). Which student is best? a comprehensive knowledge distillation exam for task-specific bert models. arXiv preprint arXiv:2201.00558.
[CrossRef] [Google Scholar]
Tahaei, M., Charlaix, E., Nia, V., Ghodsi, A., & Rezagholizadeh, M. (2022). KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2116–2127).
[CrossRef] [Google Scholar]
Cui, B., Li, Y., & Zhang, Z. (2021). Joint structured pruning and dense knowledge distillation for efficient transformer model compression. Neurocomputing, 458, 56–69.
[CrossRef] [Google Scholar]
Haider, M. H., Valarezo-Plaza, S., Muhsin, S., Zhang, H., & Ko, S. B. (2024, May). Optimized Transformer Models: $\ell$' BERT with CNN-like Pruning and Quantization. In 2024 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1-5). IEEE.
[CrossRef] [Google Scholar]
Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461, 370-403.
[CrossRef] [Google Scholar]
Tian, Y., Chen, H., Guo, T., Xu, C., & Wang, Y. (2023). Towards higher ranks via adversarial weight pruning. Advances in Neural Information Processing Systems, 36, 1189-1207.
[Google Scholar]
de Resende Oliveira, F. D., Batista, E. L. O., & Seara, R. (2024). On the compression of neural networks using $\ell$0-norm regularization and weight pruning. Neural Networks, 171, 343-352.
[CrossRef] [Google Scholar]
Xu, K., Li, Z., Wang, S., & Zhang, X. (2024). PTMQ: Post-training Multi-Bit Quantization of Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 14, pp. 16193–16201).
[CrossRef] [Google Scholar]
Zhang, A., Yang, Z., Wang, N., Qi, Y., Xin, J., Li, X., & Yin, P. (2025). Comq: A backpropagation-free algorithm for post-training quantization. IEEE Access.
[CrossRef] [Google Scholar]
Kuzmin, A., Nagel, M., Van Baalen, M., Behboodi, A., & Blankevoort, T. (2023). Pruning vs quantization: Which is better?. Advances in neural information processing systems, 36, 62414-62427.
[Google Scholar]
Cantini, R., Orsino, A., & Talia, D. (2024). Xai-driven knowledge distillation of large language models for efficient deployment on low-resource devices. Journal of Big Data, 11(1), 63.
[CrossRef] [Google Scholar]
Jouhari, M., Al-Ali, A. K., Baccour, E., Mohamed, A., Erbad, A., Guizani, M., & Hamdi, M. (2021). Distributed CNN inference on resource-constrained UAVs for surveillance systems: Design and optimization. IEEE Internet of Things Journal, 9(2), 1227-1242.
[CrossRef] [Google Scholar]
Pfeiffer, M., & Pfeil, T. (2018). Deep learning with spiking neurons: Opportunities and challenges. Frontiers in neuroscience, 12, 409662.
[CrossRef] [Google Scholar]
Sun, W., Chen, D., Chen, J., Feng, Y., Chen, C., & Wang, C. (2024). Knowledge Translation: A New Pathway for Model Compression. arXiv preprint arXiv:2401.05772.
[Google Scholar]

Cited By (4)

Shubhani Aggarwal, Arzoo Miglani, Norah Saleh Alghamdi, Gaurav Dhiman. Resilient and decentralized demand-side management in smart grids using blockchain. Scientific Reports, 2026 , 16 (1).
[CrossRef]
Gaurav Dhiman, Kiran Deep Singh, Prabh Deep Singh, Norah Saleh Alghamdi, Ghadah Shukri Albakri. A novel approach to reliable and flexible distributed computing with virtualization in smart healthcare applications. Scientific Reports, 2026 , 16 (1).
[CrossRef]
Marco Fiore, Francesca Lanera. Edge Intelligence in the IoT Era: A Review of Architectural Paradigms. Electronics, 2026 , 15 (16).
[CrossRef]
İlknur Dönmez, Faruk Bulut. Multidimensional diversity in video recommender systems: a holistic framework of literature gaps and future directions. Intelligent Systems with Applications, 2026 , 31 .
[CrossRef]

* Citation data provided by Crossref Cited-by.

Cite This Article

APA Style

Malik, I. A., & Habib, G. (2025). AI Enabled Resource-Constrained Computing Architectures for IoT Devices. ICCK Transactions on Machine Intelligence, 1(3), 138-147. https://doi.org/10.62762/TMI.2025.225921

Export Citation

RIS Format

Compatible with EndNote, Zotero, Mendeley, and other reference managers

TY  - JOUR
AU  - Malik, Ishfaq Ahmad
AU  - Habib, Gousia
PY  - 2025
DA  - 2025/11/13
TI  - AI Enabled Resource-Constrained Computing Architectures for IoT Devices
JO  - ICCK Transactions on Machine Intelligence
T2  - ICCK Transactions on Machine Intelligence
JF  - ICCK Transactions on Machine Intelligence
VL  - 1
IS  - 3
SP  - 138
EP  - 147
DO  - 10.62762/TMI.2025.225921
UR  - https://www.icck.org/article/abs/TMI.2025.225921
KW  - knowledge distillation
KW  - high-performance computing
KW  - transformers
KW  - DistilBERT
KW  - self-attention mechanism
AB  - Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.
SN  - 3068-7403
PB  - Institute of Central Computation and Knowledge
LA  - English
ER  -

BibTeX Format

Compatible with LaTeX, BibTeX, and other reference managers

@article{Malik2025AI,
  author = {Ishfaq Ahmad Malik and Gousia Habib},
  title = {AI Enabled Resource-Constrained Computing Architectures for IoT Devices},
  journal = {ICCK Transactions on Machine Intelligence},
  year = {2025},
  volume = {1},
  number = {3},
  pages = {138-147},
  doi = {10.62762/TMI.2025.225921},
  url = {https://www.icck.org/article/abs/TMI.2025.225921},
  abstract = {Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.},
  keywords = {knowledge distillation, high-performance computing, transformers, DistilBERT, self-attention mechanism},
  issn = {3068-7403},
  publisher = {Institute of Central Computation and Knowledge}
}

Article Metrics

Citations

Crossref

4

Scopus

4

Views

1611

PDF Downloads

413

Publisher's Note

ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and Permissions

Institute of Central Computation and Knowledge (ICCK) or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

ICCK Transactions on Machine Intelligence

ISSN: 3068-7403 (Online)

[email protected]

Preserved at
Portico

User

Unlimited Downloads

Complete Library Access

Membership Eligibility

Community Leadership Opportunities