AI Enabled Resource-Constrained Computing Architectures for IoT Devices

Ishfaq Ahmad Malik; Gousia Habib

doi:10.62762/TMI.2025.225921

CiteScore

Impact Factor

Volume 1, Issue 3, ICCK Transactions on Machine Intelligence

Volume 1, Issue 3, 2025

Submit Manuscript Edit a Special Issue

Article QR Code

Scan the QR code for reading

Popular articles

Case Studies on Integrating Artificial Intelligence in Finance to Transform Decision Making and Risk Management for Enhanced Financial Outcomes Reinforcement Learning for Prompt Optimization in Language Models: A Comprehensive Survey of Methods, Representations, and Evaluation Challenges AI and the Future of Education: Advancing Personalized Learning and Intelligent Tutoring Systems Plant Disease Detection Using Deep Learning Techniques Modeling Brain Functional Networks Using Graph Neural Networks: A Review and Clinical Application Bridging Modalities: A Survey of Cross-Modal Image-Text Retrieval Research on A Ship Trajectory Classification Method Based on Deep Learning Enhancing Fake News Detection with a Hybrid NLP-Machine Learning Framework Acrylamide in Food: Sources and Prevention Analyzing the Translation and Impact of Popular Science Literature in China: A Case Study Approach

ICCK Transactions on Machine Intelligence, Volume 1, Issue 3, 2025: 138-147

Free to Read | Review Article | 13 November 2025

AI Enabled Resource-Constrained Computing Architectures for IoT Devices

Ishfaq Ahmad Malik 1 *

Gousia Habib 1

1 Yogananda School of AI, Computer and Data Sciences, Shoolini University, Solan 173229, Himachal Pradesh, India

* Corresponding Author: Ishfaq Ahmad Malik, [email protected]

DOI: 10.62762/TMI.2025.225921

Received: 14 August 2025, Accepted: 18 October 2025, Published: 13 November 2025

PDF (1.96 MB)

Article Metrics Cite This Article

Abstract

Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.

Graphical Abstract

AI Enabled Resource-Constrained Computing Architectures for IoT Devices

Keywords

knowledge distillation

high-performance computing

transformers

DistilBERT

self-attention mechanism

Data Availability Statement

Not applicable.

Funding

This work was supported without any funding.

Conflicts of Interest

The authors declare no conflicts of interest.

Ethical Approval and Consent to Participate

Not applicable.

References

Bouguettaya, A., Zarzour, H., Taberkit, A. M., & Kechida, A. (2022). A review on early wildfire detection from unmanned aerial vehicles using deep learning-based computer vision algorithms. Signal Processing, 190, 108309.
[CrossRef] [Google Scholar]
Le Mero, L., Yi, D., Dianati, M., & Mouzakitis, A. (2022). A survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 23(9), 14128-14147.
[CrossRef] [Google Scholar]
Lin, S., Xie, H., Wang, B., Yu, K., Chang, X., Liang, X., & Wang, G. (2022, June). Knowledge Distillation via the Target-aware Transformer. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10905-10914). IEEE.
[CrossRef] [Google Scholar]
Zhang, L., Chen, X., Tu, X., Wan, P., Xu, N., & Ma, K. (2022, June). Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12454-12464). IEEE.
[CrossRef] [Google Scholar]
Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., & Zhang, Q. (2022). Cross-image relational knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12319-12328).
[CrossRef] [Google Scholar]
Hassan, S., Andrianomena, S., & Doughty, C. (2020). Constraining the astrophysics and cosmology from 21 cm tomography using deep learning with the SKA. Monthly Notices of the Royal Astronomical Society, 494(4), 5761–5774.
[CrossRef] [Google Scholar]
Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. (2021). Attention is all you need in speech separation. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 21–25).
[CrossRef] [Google Scholar]
Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
[Google Scholar]
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., ... & Liu, Q. (2020, November). Tinybert: Distilling bert for natural language understanding. In Findings of the association for computational linguistics: EMNLP 2020 (pp. 4163-4174).
[CrossRef] [Google Scholar]
Nityasya, M. N., Wibowo, H. A., Chevi, R., Prasojo, R. E., & Aji, A. F. (2022). Which student is best? a comprehensive knowledge distillation exam for task-specific bert models. arXiv preprint arXiv:2201.00558.
[Google Scholar]
Cui, B., Li, Y., & Zhang, Z. (2021). Joint structured pruning and dense knowledge distillation for efficient transformer model compression. Neurocomputing, 458, 56–69.
[CrossRef] [Google Scholar]
Haider, M. H., Valarezo-Plaza, S., Muhsin, S., Zhang, H., & Ko, S. B. (2024, May). Optimized Transformer Models: $\ell$' BERT with CNN-like Pruning and Quantization. In 2024 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1-5). IEEE.
[CrossRef] [Google Scholar]
Tahaei, M., Charlaix, E., Nia, V., Ghodsi, A., & Rezagholizadeh, M. (2022). KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2116–2127).
[CrossRef] [Google Scholar]
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
[Google Scholar]
Adoma, A. F., Henry, N. M., & Chen, W. (2020, December). Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition. In 2020 17th international computer conference on wavelet active media technology and information processing (ICCWAMTIP) (pp. 117-121). IEEE.
[CrossRef] [Google Scholar]
Feng, L., Yu, J., Cai, D., Liu, S., Zheng, H., & Wang, Y. (2021). ASR-GLUE: A new multi-task benchmark for asr-robust natural language understanding. arXiv preprint arXiv:2108.13048.
[Google Scholar]
Tian, Y., Chen, H., Guo, T., Xu, C., & Wang, Y. (2023). Towards higher ranks via adversarial weight pruning. Advances in Neural Information Processing Systems, 36, 1189-1207.
[Google Scholar]
de Resende Oliveira, F. D., Batista, E. L. O., & Seara, R. (2024). On the compression of neural networks using ℓ0-norm regularization and weight pruning. Neural Networks, 171, 343-352.
[CrossRef] [Google Scholar]
Xu, K., Li, Z., Wang, S., & Zhang, X. (2024). PTMQ: Post-training Multi-Bit Quantization of Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 14, pp. 16193–16201).
[CrossRef] [Google Scholar]
Zhang, A., Yang, Z., Wang, N., Qi, Y., Xin, J., Li, X., & Yin, P. (2025). Comq: A backpropagation-free algorithm for post-training quantization. IEEE Access.
[CrossRef] [Google Scholar]
Kuzmin, A., Nagel, M., Van Baalen, M., Behboodi, A., & Blankevoort, T. (2023). Pruning vs quantization: Which is better?. Advances in neural information processing systems, 36, 62414-62427.
[Google Scholar]
Hemmati, A., Raoufi, P., & Rahmani, A. M. (2024). Edge artificial intelligence for big data: a systematic review. Neural Computing and Applications, 36(19), 11461-11494.
[CrossRef] [Google Scholar]
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., ... & Lu, W. D. (2023). Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 111(9), 1016-1054.
[CrossRef] [Google Scholar]
Che, C., Zheng, H., Huang, Z., Jiang, W., & Liu, B. (2024). Intelligent robotic control system based on computer vision technology. arXiv preprint arXiv:2404.01116.
[Google Scholar]
Sun, W., Chen, D., Chen, J., Feng, Y., Chen, C., & Wang, C. (2024). Knowledge Translation: A New Pathway for Model Compression. arXiv preprint arXiv:2401.05772.
[Google Scholar]
Mansourian, A. M., Ahmadi, R., Ghafouri, M., Babaei, A. M., Golezani, E. B., Ghamchi, Z. Y., ... & Kasaei, S. (2025). A Comprehensive Survey on Knowledge Distillation. arXiv preprint arXiv:2503.12067.
[Google Scholar]
Cantini, R., Orsino, A., & Talia, D. (2024). Xai-driven knowledge distillation of large language models for efficient deployment on low-resource devices. Journal of Big Data, 11(1), 63.
[CrossRef] [Google Scholar]
Setyanto, A., Sasongko, T. B., Fikri, M. A., Ariatmanto, D., Agastya, I. M. A., Rachmanto, R. D., ... & Kim, I. K. (2025). Knowledge Distillation in Object Detection for Resource-Constrained Edge Computing. IEEE Access.
[CrossRef] [Google Scholar]

Cite This Article

APA Style

Malik, I. A., & Habib, G. (2025). AI Enabled Resource-Constrained Computing Architectures for IoT Devices. ICCK Transactions on Machine Intelligence, 1(3), 138–147. https://doi.org/10.62762/TMI.2025.225921

Export Citation

RIS Format

Compatible with EndNote, Zotero, Mendeley, and other reference managers

RIS format data for reference managers

TY  - JOUR
AU  - Malik, Ishfaq Ahmad
AU  - Habib, Gousia
PY  - 2025
DA  - 2025/11/13
TI  - AI Enabled Resource-Constrained Computing Architectures for IoT Devices
JO  - ICCK Transactions on Machine Intelligence
T2  - ICCK Transactions on Machine Intelligence
JF  - ICCK Transactions on Machine Intelligence
VL  - 1
IS  - 3
SP  - 138
EP  - 147
DO  - 10.62762/TMI.2025.225921
UR  - https://www.icck.org/article/abs/TMI.2025.225921
KW  - knowledge distillation
KW  - high-performance computing
KW  - transformers
KW  - DistilBERT
KW  - self-attention mechanism
AB  - Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.
SN  - 3068-7403
PB  - Institute of Central Computation and Knowledge
LA  - English
ER  -

BibTeX Format

Compatible with LaTeX, BibTeX, and other reference managers

BibTeX format data for LaTeX and reference managers

@article{Malik2025AI,
  author = {Ishfaq Ahmad Malik and Gousia Habib},
  title = {AI Enabled Resource-Constrained Computing Architectures for IoT Devices},
  journal = {ICCK Transactions on Machine Intelligence},
  year = {2025},
  volume = {1},
  number = {3},
  pages = {138-147},
  doi = {10.62762/TMI.2025.225921},
  url = {https://www.icck.org/article/abs/TMI.2025.225921},
  abstract = {Deep learning is a great success primarily because it encodes large amounts of data and manipulates billions of model parameters. Despite this, it is challenging to deploy these cumbersome deep models on devices with limited resources, such as mobile phones and embedded devices, due to the high computational complexity and the amount of storage required. Various techniques are available to compress and accelerate models for this purpose. Knowledge distillation is a novel technique for model compression and acceleration, which involves learning a small student model from a large teacher model. Then, that student network is fine-tuned on any downstream task to be applicable for resource-constrained applications. This paper explores various state-of-the-art model compression techniques, including knowledge distillation, for compressing large deep neural networks to make them deployable on resource-constrained devices.},
  keywords = {knowledge distillation, high-performance computing, transformers, DistilBERT, self-attention mechanism},
  issn = {3068-7403},
  publisher = {Institute of Central Computation and Knowledge}
}

Article Metrics

Citations:

Google Scholar

Crossref

Scopus

Web of Science

Article Access Statistics:

PDF Downloads: 36

Publisher's Note

ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and Permissions

Institute of Central Computation and Knowledge (ICCK) or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

ICCK Transactions on Machine Intelligence

ISSN: 3068-7403 (Online)

Email: [email protected]

Portico

All published articles are preserved here permanently:
https://www.portico.org/publishers/icck/

Google Scholar

Crossref

Scopus

Web of Science

We use cookies