A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition
Research Article  ·  Published: 26 September 2025
Issue cover
ICCK Transactions on Machine Intelligence
Volume 1, Issue 2, 2025: 103-116
Research Article Free to Read

A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition

1 Gujarat Technological University, Ahmedabad 382424, India
* Corresponding Author: Ketan Sarvakar, [email protected]
Volume 1, Issue 2
You have access to this article · Limited-Time Free Access

Article Information

Abstract

Deep learning has substantially enhanced facial emotion recognition, an essential element of human--computer interaction. This study evaluates the performance of multiple architectures, including a custom CNN, VGG-16, ResNet-50, and a hybrid CNN-LSTM framework, across FER2013 and CK+ datasets. Preprocessing steps involved grayscale conversion, image resizing, and pixel normalization. Experimental results show that ResNet-50 achieved the highest accuracy on FER2013 (76.85%), while the hybrid CNN-LSTM model attained superior performance on CK+ (92.30%). Performance metrics such as precision, recall, and F1-score were used for evaluation. Findings highlight the trade-off between computational efficiency and recognition accuracy, offering insights for developing robust, real-time emotion recognition systems.

Graphical Abstract

A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition

Keywords

face expressions face emotion recognition deep learning CNN VGG-16 ResNet-50 LSTM transfer learning

Data Availability Statement

Data will be made available on request.

Funding

This work was supported without any funding.

Conflicts of Interest

The authors declare no conflicts of interest.

Ethical Approval and Consent to Participate

Not applicable.

References

  1. Naga, P., Marri, S. D., & Borreo, R. (2023). Facial emotion recognition methods, datasets and technologies: A literature survey. Materials Today: Proceedings, 80, 2824-2828.
    [CrossRef] [Google Scholar]
  2. Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2023). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 35(32), 23311-23328.
    [CrossRef] [Google Scholar]
  3. Ming, Y., Qian, H., & Guangyuan, L. (2022). CNN‐LSTM Facial Expression Recognition Method Fused with Two‐Layer Attention Mechanism. Computational Intelligence and Neuroscience, 2022(1), 7450637.
    [CrossRef] [Google Scholar]
  4. Sarvakar, K., Senkamalavalli, R., Raghavendra, S., Kumar, J. S., Manjunath, R., & Jaiswal, S. (2023). Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings, 80, 3560-3564.
    [CrossRef] [Google Scholar]
  5. Dias, W., Andalo, F., Padilha, R., Bertocco, G., Almeida, W., Costa, P., & Rocha, A. (2022). Cross-dataset emotion recognition from facial expressions through convolutional neural networks. Journal of Visual Communication and Image Representation, 82, 103395.
    [CrossRef] [Google Scholar]
  6. Hernandez-Luquin, F., & Escalante, H. J. (2023). Multi-branch deep radial basis function networks for facial emotion recognition. Neural Computing and Applications, 35(25), 18131-18145.
    [CrossRef] [Google Scholar]
  7. Kopalidis, T., Solachidis, V., Vretos, N., & Daras, P. (2024). Advances in facial expression recognition: a survey of methods, benchmarks, models, and datasets. Information, 15(3), 135.
    [CrossRef] [Google Scholar]
  8. Nan, Y., Ju, J., Hua, Q., Zhang, H., & Wang, B. (2022). A-MobileNet: An approach of facial expression recognition. Alexandria Engineering Journal, 61(6), 4435-4444.
    [CrossRef] [Google Scholar]
  9. Shahzad, H. M., Bhatti, S. M., Jaffar, A., Akram, S., Alhajlah, M., & Mahmood, A. (2023). Hybrid facial emotion recognition using CNN-based features. Applied Sciences, 13(9), 5572.
    [CrossRef] [Google Scholar]
  10. Ahmad, M., Alfandi, O., Khattak, A. M., Qadri, S. F., Saeed, I. A., Khan, S., ... & Ahmad, A. (2023). Facial expression recognition using lightweight deep learning modeling. Mathematical Biosciences and Engineering, 20(5), 8208.
    [CrossRef] [Google Scholar]
  11. Borgalli, R. A., & Surve, S. (2022, March). Deep learning framework for facial emotion recognition using CNN architectures. In 2022 International Conference on Electronics and Renewable Systems (ICEARS) (pp. 1777-1784). IEEE.
    [CrossRef] [Google Scholar]
  12. Khattak, A., Asghar, M. Z., Ali, M., & Batool, U. (2022). An efficient deep learning technique for facial emotion recognition. Multimedia Tools and Applications, 81(2), 1649-1683.
    [CrossRef] [Google Scholar]
  13. Singh, R., Saurav, S., Kumar, T., Saini, R., Vohra, A., & Singh, S. (2023). Facial expression recognition in videos using hybrid CNN & ConvLSTM. International Journal of Information Technology, 15(4), 1819-1830.
    [CrossRef] [Google Scholar]
  14. Magherini, R., Mussi, E., Servi, M., & Volpe, Y. (2022). Emotion recognition in the times of COVID19: Coping with face masks. Intelligent Systems with Applications, 15, 200094.
    [CrossRef] [Google Scholar]
  15. Dar, T., Javed, A., Bourouis, S., Hussein, H. S., & Alshazly, H. (2022). Efficient-SwishNet based system for facial emotion recognition. IEEE Access, 10, 71311-71328.
    [CrossRef] [Google Scholar]
  16. Slimani, K., Ruichek, Y., & Messoussi, R. (2022). Compound facial emotional expression recognition using cnn deep features. Engineering Letters, 30(4), 1402-1416. https://www.engineeringletters.com/issues_v30/issue_4/EL_30_4_25.pdf
    [Google Scholar]
  17. Pise, A., Vadapalli, H., & Sanders, I. (2022). Facial emotion recognition using temporal relational network: an application to E-learning. Multimedia Tools and Applications, 81(19), 26633-26653.
    [CrossRef] [Google Scholar]
  18. Abate, A. F., Cimmino, L., Mocanu, B. C., Narducci, F., & Pop, F. (2023). The limitations for expression recognition in computer vision introduced by facial masks. Multimedia Tools and Applications, 82(8), 11305-11319.
    [CrossRef] [Google Scholar]
  19. Khan, A. R. (2022). Facial emotion recognition using conventional machine learning and deep learning methods: current achievements, analysis and remaining challenges. Information, 13(6), 268.
    [CrossRef] [Google Scholar]
  20. Minaee, S., Minaei, M., & Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 21(9), 3046.
    [CrossRef] [Google Scholar]
  21. Dwijayanti, S., Iqbal, M., & Suprapto, B. Y. (2022). Real-time implementation of face recognition and emotion recognition in a humanoid robot using a convolutional neural network. IEEE Access, 10, 89876-89886.
    [CrossRef] [Google Scholar]
  22. Gupta, S., Kumar, P., & Tekchandani, R. K. (2023). Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimedia Tools and Applications, 82(8), 11365-11394.
    [CrossRef] [Google Scholar]
  23. Canal, F. Z., Müller, T. R., Matias, J. C., Scotton, G. G., de Sa Junior, A. R., Pozzebon, E., & Sobieranski, A. C. (2022). A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences, 582, 593-617.
    [CrossRef] [Google Scholar]
  24. ELsayed, Y., ELSayed, A., & Abdou, M. A. (2023). An automatic improved facial expression recognition for masked faces. Neural Computing and Applications, 35(20), 14963-14972.
    [CrossRef] [Google Scholar]
  25. Alsharekh, M. F. (2022). Facial emotion recognition in verbal communication based on deep learning. Sensors, 22(16), 6105.
    [CrossRef] [Google Scholar]
  26. Farkhod, A., Abdusalomov, A. B., Mukhiddinov, M., & Cho, Y. I. (2022). Development of real-time landmark-based emotion recognition CNN for masked faces. Sensors, 22(22), 8704.
    [CrossRef] [Google Scholar]
  27. Dewi, C., Gunawan, L. S., Hastoko, S. G., & Christanto, H. J. (2024). Real-time facial expression recognition: advances, challenges, and future directions. Vietnam Journal of Computer Science, 11(02), 167-193.
    [CrossRef] [Google Scholar]
  28. Mukhiddinov, M., Djuraev, O., Akhmedov, F., Mukhamadiyev, A., & Cho, J. (2023). Masked face emotion recognition based on facial landmarks and deep learning approaches for visually impaired people. Sensors, 23(3), 1080.
    [CrossRef] [Google Scholar]
  29. Sajjad, M., Ullah, F. U. M., Ullah, M., Christodoulou, G., Cheikh, F. A., Hijji, M., ... & Rodrigues, J. J. (2023). A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines. Alexandria Engineering Journal, 68, 817-840.
    [CrossRef] [Google Scholar]
  30. Mohana, M., Subashini, P., & Krishnaveni, M. (2023). Emotion recognition from facial expression using hybrid CNN–LSTM network. International Journal of Pattern Recognition and Artificial Intelligence, 37(08), 2356008.
    [CrossRef] [Google Scholar]
  31. Pereira, R., Mendes, C., Ribeiro, J., Ribeiro, R., Miragaia, R., Rodrigues, N., ... & Pereira, A. (2024). Systematic review of emotion detection with computer vision and deep learning. Sensors, 24(11), 3484.
    [CrossRef] [Google Scholar]
  32. Varma, S., Shinde, M., & Chavan, S. S. (2019, November). Analysis of PCA and LDA features for facial expression recognition using SVM and HMM classifiers. In Techno-Societal 2018: Proceedings of the 2nd International Conference on Advanced Technologies for Societal Applications-Volume 1 (pp. 109-119). Cham: Springer International Publishing.
    [CrossRef] [Google Scholar]
  33. Reddy, C. V. R., Reddy, U. S., & Kishore, K. V. K. (2019). Facial emotion recognition using NLPCA and SVM. Traitement du Signal, 36(1), 13-22.
    [CrossRef] [Google Scholar]
  34. Sajjad, M., Nasir, M., Ullah, F. U. M., Muhammad, K., Sangaiah, A. K., & Baik, S. W. (2019). Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services. Information Sciences, 479, 416-431.
    [CrossRef] [Google Scholar]
  35. Nazir, M., Jan, Z., & Sajjad, M. (2017). Facial expression recognition using weber discrete wavelet transform. Journal of Intelligent & Fuzzy Systems, 33(1), 479-489.
    [CrossRef] [Google Scholar]
  36. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., & Dobaie, A. M. (2018). Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 273, 643-649.
    [CrossRef] [Google Scholar]
  37. Uddin, M. Z., Hassan, M. M., Almogren, A., Zuair, M., Fortino, G., & Torresen, J. (2017). A facial expression recognition system using robust face features from depth videos and deep learning. Computers & Electrical Engineering, 63, 114-125.
    [CrossRef] [Google Scholar]
  38. Al-agha, L. S. A., Saleh, P. H. H., & Ghani, P. R. F. (2017). Geometric-based feature extraction and classification for emotion expressions of 3D video film. Journal of Advances in Information Technology, 8(2).
    [CrossRef] [Google Scholar]
  39. Ghimire, D., Lee, J., Li, Z. N., & Jeong, S. (2017). Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools and Applications, 76(6), 7921-7946.
    [CrossRef] [Google Scholar]
  40. Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE transactions on affective computing, 13(3), 1195-1215.
    [CrossRef] [Google Scholar]
  41. Wu, P. P., Liu, H., Zhang, X. W., & Gao, Y. (2017). Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics. Frontiers of Information Technology & Electronic Engineering, 18(7), 955-967.
    [CrossRef] [Google Scholar]
  42. Ekundayo, O. S., & Viriri, S. (2021). Facial expression recognition: A review of trends and techniques. IEEE Access, 9, 136944-136973.
    [CrossRef] [Google Scholar]
  43. Kim, D. J. (2016). Facial expression recognition using ASM-based post-processing technique. Pattern Recognition and Image Analysis, 26(3), 576-581.
    [CrossRef] [Google Scholar]
  44. Cornejo, J. Y. R., Pedrini, H., & Flórez-Revuelta, F. (2015, October). Facial expression recognition with occlusions based on geometric representation. In Iberoamerican Congress on Pattern Recognition (pp. 263-270). Cham: Springer International Publishing.
    [CrossRef] [Google Scholar]
  45. Siddiqi, M. H., Ali, R., Khan, A. M., Kim, E. S., Kim, G. J., & Lee, S. (2015). Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimedia Systems, 21(6), 541-555.
    [CrossRef] [Google Scholar]
  46. Chang, K. Y., Chen, C. S., & Hung, Y. P. (2013, October). Intensity rank estimation of facial expressions based on a single image. In 2013 IEEE International Conference on Systems, Man, and Cybernetics (pp. 3157-3162). IEEE.
    [CrossRef] [Google Scholar]
  47. Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the national academy of sciences, 111(15), E1454-E1462.
    [CrossRef] [Google Scholar]
  48. Mavani, V., Raman, S., & Miyapuram, K. P. (2017, October). Facial Expression Recognition Using Visual Saliency and Deep Learning. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) (pp. 2783-2788). IEEE.
    [CrossRef] [Google Scholar]
  49. Slimani, K., Lekdioui, K., Messoussi, R., & Touahni, R. (2019, March). Compound facial expression recognition based on highway CNN. In Proceedings of the new challenges in data sciences: acts of the second conference of the Moroccan Classification Society (pp. 1-7).
    [CrossRef] [Google Scholar]
  50. Li, S., Deng, W., & Du, J. (2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852-2861).
    [CrossRef] [Google Scholar]
  51. Fan, Y., Lam, J. C., & Li, V. O. (2018, September). Multi-region ensemble convolutional neural network for facial expression recognition. In International Conference on Artificial Neural Networks (pp. 84-94). Cham: Springer International Publishing.
    [CrossRef] [Google Scholar]
  52. Ghosh, S., Dhall, A., & Sebe, N. (2018, October). Automatic group affect analysis in images via visual attribute and feature networks. In 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 1967-1971). IEEE.
    [CrossRef] [Google Scholar]
  53. Shen, F., Liu, J., & Wu, P. (2018, October). Double complete d-lbp with extreme learning machine auto-encoder and cascade forest for facial expression analysis. In 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 1947-1951). IEEE.
    [CrossRef] [Google Scholar]
  54. Vielzeuf, V., Kervadec, C., Pateux, S., Lechervy, A., & Jurie, F. (2018, October). An occam's razor view on learning audiovisual emotion recognition with small training sets. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (pp. 589-593).
    [CrossRef] [Google Scholar]
  55. Acharya, D., Huang, Z., Paudel, D. P., & Van Gool, L. (2018, June). Covariance Pooling for Facial Expression Recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 480-4807). IEEE.
    [CrossRef] [Google Scholar]
  56. Li, Y., Zeng, J., Shan, S., & Chen, X. (2018, August). Patch-gated CNN for occlusion-aware facial expression recognition. In 2018 24th international conference on pattern recognition (ICPR) (pp. 2209-2214). IEEE.
    [CrossRef] [Google Scholar]
  57. Deng, J., Pang, G., Zhang, Z., Pang, Z., Yang, H., & Yang, G. (2019). cGAN based facial expression recognition for human-robot interaction. IEEE Access, 7, 9848-9859.
    [CrossRef] [Google Scholar]
  58. Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020). Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, 29, 4057-4069.
    [CrossRef] [Google Scholar]
  59. Vo, T. H., Lee, G. S., Yang, H. J., & Kim, S. H. (2020). Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access, 8, 131988-132001.
    [CrossRef] [Google Scholar]
  60. Yu, J., Cai, Z., Li, R., Zhao, G., Xie, G., Zhu, J., ... & Zheng, W. (2023, June). Exploring Large-scale Unlabeled Faces to Enhance Facial Expression Recognition. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 5803-5810). IEEE.
    [CrossRef] [Google Scholar]
  61. Yu, S., Jia, W., Shu, X., Yuan, X., Gui, J., Tang, J., ... & Liu, Q. (Eds.). (2025). Biometric Recognition. In 18th Chinese Conference, CCBR 2024, Nanjing, China, November 22–24, 2024, Proceedings, Part II. Springer Nature.
    [CrossRef] [Google Scholar]

Cited By (6)

  1. Meenakshi Garg, Nor Asilah Wati Abdul Hamid, Dafik, Sunder R.. AI-driven feature descriptor design using CNNs and transfer learning for enhanced image retrieval accuracy and efficiency. Scientific Reports, 2026 , 16 (1).
    [CrossRef]
  2. Mengyun Wang, Manru Xun, Qinglin Yun. Fine-Grained Human Action Recognition Via Laban Movement Analysis and Attentive Bi-LSTM. International Journal of Pattern Recognition and Artificial Intelligence, 2026 .
    [CrossRef]
  3. Gaurav Dhiman, Kiran Deep Singh, Prabh Deep Singh, Norah Saleh Alghamdi, Ghadah Shukri Albakri. A novel approach to reliable and flexible distributed computing with virtualization in smart healthcare applications. Scientific Reports, 2026 , 16 (1).
    [CrossRef]
  4. Bin Sun. Lightweight GIS-based large-scale urban fire spread simulation method. Journal of Industrial Information Integration, 2026 , 50 .
    [CrossRef]
  5. İlknur Dönmez, Faruk Bulut. Multidimensional diversity in video recommender systems: a holistic framework of literature gaps and future directions. Intelligent Systems with Applications, 2026 , 31 .
    [CrossRef]
  6. Shubhani Aggarwal, Arzoo Miglani, Norah Saleh Alghamdi, Gaurav Dhiman. Resilient and decentralized demand-side management in smart grids using blockchain. Scientific Reports, 2026 , 16 (1).
    [CrossRef]
* Citation data provided by Crossref Cited-by.

Cite This Article

APA Style
Sarvakar, K., & Rana, K. (2025). A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition. ICCK Transactions on Machine Intelligence, 1(2), 103-116. https://doi.org/10.62762/TMI.2025.572412
Export Citation
RIS Format
Compatible with EndNote, Zotero, Mendeley, and other reference managers
TY  - JOUR
AU  - Sarvakar, Ketan
AU  - Rana, Kaushik
PY  - 2025
DA  - 2025/09/26
TI  - A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition
JO  - ICCK Transactions on Machine Intelligence
T2  - ICCK Transactions on Machine Intelligence
JF  - ICCK Transactions on Machine Intelligence
VL  - 1
IS  - 2
SP  - 103
EP  - 116
DO  - 10.62762/TMI.2025.572412
UR  - https://www.icck.org/article/abs/TMI.2025.572412
KW  - face expressions
KW  - face emotion recognition
KW  - deep learning
KW  - CNN
KW  - VGG-16
KW  - ResNet-50
KW  - LSTM
KW  - transfer learning
AB  - Deep learning has substantially enhanced facial emotion recognition, an essential element of human--computer interaction. This study evaluates the performance of multiple architectures, including a custom CNN, VGG-16, ResNet-50, and a hybrid CNN-LSTM framework, across FER2013 and CK+ datasets. Preprocessing steps involved grayscale conversion, image resizing, and pixel normalization. Experimental results show that ResNet-50 achieved the highest accuracy on FER2013 (76.85%), while the hybrid CNN-LSTM model attained superior performance on CK+ (92.30%). Performance metrics such as precision, recall, and F1-score were used for evaluation. Findings highlight the trade-off between computational efficiency and recognition accuracy, offering insights for developing robust, real-time emotion recognition systems.
SN  - 3068-7403
PB  - Institute of Central Computation and Knowledge
LA  - English
ER  - 
BibTeX Format
Compatible with LaTeX, BibTeX, and other reference managers
@article{Sarvakar2025A,
  author = {Ketan Sarvakar and Kaushik Rana},
  title = {A Hybrid Framework Combining CNN, LSTM, and Transfer Learning for Emotion Recognition},
  journal = {ICCK Transactions on Machine Intelligence},
  year = {2025},
  volume = {1},
  number = {2},
  pages = {103-116},
  doi = {10.62762/TMI.2025.572412},
  url = {https://www.icck.org/article/abs/TMI.2025.572412},
  abstract = {Deep learning has substantially enhanced facial emotion recognition, an essential element of human--computer interaction. This study evaluates the performance of multiple architectures, including a custom CNN, VGG-16, ResNet-50, and a hybrid CNN-LSTM framework, across FER2013 and CK+ datasets. Preprocessing steps involved grayscale conversion, image resizing, and pixel normalization. Experimental results show that ResNet-50 achieved the highest accuracy on FER2013 (76.85\%), while the hybrid CNN-LSTM model attained superior performance on CK+ (92.30\%). Performance metrics such as precision, recall, and F1-score were used for evaluation. Findings highlight the trade-off between computational efficiency and recognition accuracy, offering insights for developing robust, real-time emotion recognition systems.},
  keywords = {face expressions, face emotion recognition, deep learning, CNN, VGG-16, ResNet-50, LSTM, transfer learning},
  issn = {3068-7403},
  publisher = {Institute of Central Computation and Knowledge}
}

Article Metrics

Citations
Views
2768
PDF Downloads
872

Publisher's Note

ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and Permissions

Institute of Central Computation and Knowledge (ICCK) or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
ICCK Transactions on Machine Intelligence
ICCK Transactions on Machine Intelligence
ISSN: 3068-7403 (Online)
Portico
Preserved at
Portico