Deepfake Detection Based on Visual Lip-sync Match and Blink Rate
DOI:
https://doi.org/10.22399/ijcesen.755Keywords:
Deepfake Detection, visual lip-sync matching, Blink Rate, Artificial IntelligenceAbstract
Deepfake technology has emerged as a significant challenge to the authenticity of digital media, necessitating innovative detection methods. This paper introduces TrueSync, an advanced application for detecting deepfake videos by integrating two critical detection features: lip-sync analysis and blink rate monitoring. Leveraging a hybrid approach combining CNN-LSTM and SyncNet models, TrueSync processes visual and temporal features to identify anomalies in lip movement synchronization and eye blinking patterns. The application utilizes a modular pipeline to analyse these features independently and then fuses the results for a comprehensive detection score. This approach enhances detection accuracy and provides users with reliable tools to combat sophisticated manipulations. By proposing this scalable solution, TrueSync addresses the increasing difficulty in distinguishing authentic videos from manipulated content, fostering trust in digital media.
References
Gupta, H. (2024). Perceptual synchronization scoring of dubbed content using phoneme-viseme agreement. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 392–402.
Zhou, Y., & Lim, S. N. (2021). Joint audio-visual deepfake detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, 14800–14809.
Halperin, T., Ephrat, A., & Peleg, S. (2019). Dynamic temporal alignment of speech to lips. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3980–3984. IEEE.
Park, S. J., Kim, M., Choi, J., & Ro, Y. M. (2024, April). Exploring phonetic context-aware lip-sync for talking face generation. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4325–4329. IEEE.
Yoon, D., & Cho, H. (2024). Lip and voice synchronization using visual attention. The Transactions of the Korea Information Processing Society, 13(4), 166–173.
Guera, D., & Delp, E. J. (2018). Deepfake video detection using recurrent neural networks. Proceedings of the 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1–6.
Li, Y., Chang, M. C., & Lyu, S. (2018). In ictu oculi: Exposing AI generated fake face videos by detecting eye blinking. arXiv preprint arXiv:1806.02877.
Farid, H. (2020). Creating, identifying, and combating deep fakes. IEEE International Workshop on Information Forensics and Security (WIFS). Retrieved from https://farid.berkeley.edu/downloads/publications/wifs20.pdf
Gholipour, A., Taheri, A., & Mohammadzade, H. (2021). Automated lip-reading robotic system based on convolutional neural network and long short-term memory. In Social Robotics: 13th International Conference, ICSR 2021, Singapore, November 10–13, 2021, Proceedings 13 (pp. 73–84). Springer International Publishing.
Almutairi, Z., & Elgibreen, H. (2022). A review of modern audio deepfake detection methods: Challenges and future directions. Algorithms, 15(5), 155.
Al-Khazraji, S. H., Saleh, H. H., Khalid, A. I., & Mishkhal, I. A. (2023). Impact of deepfake technology on social media: Detection, misinformation, and societal implications. The Eurasia Proceedings of Science Technology Engineering and Mathematics, 23, 429–441.
Mallet, J., Krueger, N., Vanamala, M., & Dave, R. (2023). Hybrid deepfake detection utilizing MLP and LSTM. arXiv. Retrieved from https://arxiv.org/pdf/2304.14504
Arshad, S., & Shah, S. C. A. (2024). Hybrid optimized deep feature fusion-based deepfake detection in videos using spotted hyena optimizer. Computers & Security, 134, 102848. https://doi.org/10.1016/j.cose.2023.102848
GeeksforGeeks. (2023). Residual networks (ResNet) – Deep learning. Retrieved from https://www.geeksforgeeks.org/residual-networks-resnet-deep-learning/
Cozzolino, D., Poggi, G., & Verdoliva, L. (2017). Recasting residual-based local descriptors as convolutional neural networks: An application to image forgery detection. Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, 159–164.
Yao, H., Huang, Z., & Tan, S. (2021). Detection of AI-synthesized video using temporal cues: Eye blinks and mouth movements. Journal of Information Security Research, 13(3), 155–167.
Raina, A., & Arora, V. (2022). SyncNet: Using causal convolutions and correlating objective for time delay estimation in audio signals. arXiv preprint arXiv:2203.14639.
Jung, T., Kim, S., & Kim, K. (2020). DeepVision: Deepfakes detection using human eye blinking patterns. IEEE Access, 8, 83144–83154.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
Chollet, F. (2017). Deep learning with Python. Manning Publications.
Wang, Y., & Zhang, H. (2024). Hybrid approaches for deepfake detection. Journal of Multimedia Tools and Applications, 2(4), 73–90.
Heidari, A., Jafari Navimipour, N., Dag, H., & Unal, M. (2024). Deepfake detection using deep learning methods: A systematic and comprehensive review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 14(2), e1520.
Robert, N. R., A. Cecil Donald, & K. Suresh. (2025). Artificial Intelligence Technique Based Effective Disaster Recovery Framework to Provide Longer Time Connectivity in Mobile Ad-hoc Networks. International Journal of Computational and Experimental Science and Engineering, 11(1). https://doi.org/10.22399/ijcesen.713
ZHANG, J. (2025). Artificial intelligence contributes to the creative transformation and innovative development of traditional Chinese culture. International Journal of Computational and Experimental Science and Engineering, 11(1). https://doi.org/10.22399/ijcesen.860
M.K. Sarjas, & G. Velmurugan. (2025). Bibliometric Insight into Artificial Intelligence Application in Investment. International Journal of Computational and Experimental Science and Engineering, 11(1). https://doi.org/10.22399/ijcesen.864
S. Esakkiammal, & K. Kasturi. (2024). Advancing Educational Outcomes with Artificial Intelligence: Challenges, Opportunities, And Future Directions. International Journal of Computational and Experimental Science and Engineering, 10(4). https://doi.org/10.22399/ijcesen.799
Bandla Raghuramaiah, & Suresh Chittineni. (2025). BreastHybridNet: A Hybrid Deep Learning Framework for Breast Cancer Diagnosis Using Mammogram Images. International Journal of Computational and Experimental Science and Engineering, 11(1). https://doi.org/10.22399/ijcesen.812
Almutairi, Z., & Elgibreen, H. (2022). Advanced detection of audio-visual deepfake patterns. Algorithms, 15(6), 155–160.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
