Semantic Analysis of ChatGPT’s Behavior in Contextual Code Correction for Python
DOI:
https://doi.org/10.22399/ijcesen.3419Keywords:
Semantic Code Analysis, ChatGPT, Python, Large Language Models, Error Detection Rate, Artificial IntelligenceAbstract
Software debugging remains one of the most time-consuming and cognitively demanding phases in the software development lifecycle. The paper considers the analytical capabilities of transformers transformer-based language models—specifically ChatGPT—in detecting and correcting semantic faults in multi-module Python projects. Ten synthetic Python programs (100–200 lines each), containing a total of 30 deliberately injected faults (distributed among order faults, variable leakage, and edge-case omissions), were submitted to ChatGPT under a standardized prompting scheme. Model responses were benchmarked against conventional static analysis tools (Pylint, MyPy) and a human expert baseline. Quantitatively, ChatGPT achieved an average error detection rate (EDR) of 76.7%, outperforming Pylint (23.3%) and MyPy (15%) across fault categories. In repair accuracy (RA), ChatGPT resolved 62–75% of the identified errors correctly, versus 10–25% for static tools. Statistical validation using a Chi-square test (χ² = 36.27, p < 0.001) and one-way ANOVA (F(3, 27) = 14.92, p < 0.001) confirms the significance of these differences. Qualitative clarity was also assessed using ordinal metrics and validated via Kruskal-Wallis H tests (H = 11.56, p < 0.01). These results suggest that ChatGPT possesses substantial semantic reasoning capabilities, particularly in contexts requiring non-local inference across modules. However, limitations persist in handling implicit dependencies and dynamic runtime conditions. The study concludes that such models can be meaningfully integrated into debugging pipelines as assistive agents, provided their outputs are cross-validated with expert oversight and static tools. Future research should explore hybrid frameworks that combine statistical inference with formal verification techniques.
References
[1] Stack Overflow. (2024). Developer survey results. https://insights.stackoverflow.com/survey
[2] GitHub. (2024). State of the Octoverse report. https://octoverse.github.com
[3] IEEE Spectrum. (2024). Top challenges in software development. IEEE.
[4] Ernst, M., et al. (2023). Manual debugging efficiency and practices. Empirical Software Engineering, 28(2), 223–245. https://doi.org/10.1007/s10664-023-10123-5
[5] Jones, D., & Liang, P. (2024). Quantifying debugging effort: A multicase study. Journal of Systems and Software, 195(6), 111283. https://doi.org/10.1016/j.jss.2024.111283
[6] van Rossum, G., & Warsaw, B. (2010). Pylint: A Python static code analyzer. Python Software Foundation.
[7] Lehtosalo, J. (2018). MyPy: Optional static typing for Python [GitHub repository]. https://github.com/python/mypy
[8] Zhuang, Y. Y., Kao, C. W., & Yen, W. H. (2025). A static analysis approach for detecting array shape errors in Python. Journal of Information Science & Engineering, 41(1).
[9] Fernández Poolan, R. O. (2024). Optimizing Python software through clean code: Practices and principles.
[10] Sharma, R., & Whitehead, J. (2019). Beyond lint: Deep code analysis for fault detection. Empirical Software Engineering, 24(3), 459–474. https://doi.org/10.1007/s10664-018-9653-x
[11] Zhang, Z., Wang, L., Li, Y., & Chen, S. (2024). Reasoning runtime behavior of a program with LLM: How far are we?. https://arxiv.org/abs/2403.16437
[12] Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. https://arxiv.org/abs/2107.03374
[13] Li, Y., et al. (2018). Neural machine translation for program repair. In Proceedings of ICLR.
[14] Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374
[15] Ahmad, W. U., et al. (2022). CodeXGLUE: A Benchmark Dataset for Code Intelligence.
[16] Wang, X., Liu, H., & Zhang, Y. (2023). On the effectiveness of language models for code error correction. ACM Transactions on Software Engineering and Methodology, 32(4), 45–67. https://doi.org/10.1145/3571736
[17] Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., & Rozière, B. (2024). Code Llama: Open foundation models for code. Meta AI.
[18] Liu, H., et al. (2024). Integrating LLMs into developer environments for enhanced debugging. IEEE Software, 41(2).
[19] Ahmad, W. U., et al. (2023). Evaluating pre-trained transformers for code completion and summarization. https://arxiv.org/abs/2303.01859
[20] Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., … Sun, M. (2024). DebugBench: Evaluating debugging capability of large language models. https://arxiv.org/abs/2401.04621
[21] Chirkova, N., Babii, H., D’Antoni, L., & Krishnamurthi, S. (2022). Measuring the effectiveness of LLM-based bug fixing. https://arxiv.org/abs/2210.04273
[22] Wang, X., Liu, H., & Zhang, Y. (2023). On the effectiveness of language models for code error correction. ACM Transactions on Software Engineering and Methodology, 32(4), 45–67. https://doi.org/10.1145/3571736
[23] Sharma, R., & Whitehead, J. (2019). Beyond lint: Deep code analysis for fault detection. Empirical Software Engineering, 24(3), 459–474. https://doi.org/10.1007/s10664-018-9653-x
[24] Lima, R. (2019). Review of Python static analysis tools: Pylint vs Flake8 vs MyPy. Medium. https://medium.com/@codacy/review-of-python-static-analysis-tools-29ede4342674
[25] MyPy. (n.d.). Optional static typing for Python. https://www.mypy-lang.org/
[26] Zhu, X., Li, Y., Wang, Y., Wang, H., & Zhou, J. (2021). Syntax- and semantic-aware neural bug fix. https://arxiv.org/abs/2106.08253
[27] Li, Y., Wang, Y., & Wang, Y. (2021). DLFix: Context-based code transformation learning for automated program repair. https://arxiv.org/abs/2106.08253
[28] Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating large language models trained on code. https://arxiv.org/abs/2107.03374
[29] Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. (2021). CodeXGLUE: A benchmark dataset and open challenge for code intelligence. https://arxiv.org/abs/2102.04664
[30] Wang, J., Zhang, X., & Lin, T. (2023). An empirical study on large language models for debugging tasks.
[31] Liu, Z., Liu, Y., Gu, X., & Chen, H. (2021). LLMs meet IDEs: Enhancing software development with code completion and feedback. https://arxiv.org/abs/2102.04664
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
