Comparative Analysis of GitHub Copilot and ChatGPT in Web Application Development: An Experimental Study
DOI:
https://doi.org/10.22399/ijcesen.1846Keywords:
AI Tools, Web Development Application, Comparative Analysis, Experimental Study, EducationAbstract
Nowadays, the artificial intelligence has been rapidly developed and with this development has initiated the transformation of web development by changing the way students deal with code, problem solving and soft skills. The key innovations in this changing are AI tools like GitHub Copilot and ChatGPT which provides intelligent assistance in boost the productivity and learning. The GitHub Copilot is created by GitHub in collaboration with OpenAI, and it is integrated into Visual Studio Code, while ChatGPT is also created by OpenAI, but it facilitated interactive communication through chat and code explanation. Although, several studies treated the AI Tools in education especially ChatGPT but there is limited research comparing these two tools in web development tasks. This experimental study treats the usage of AI tools by students in web application development to show the impact on their learning, development and soft skills through a comparative analysis. Findings suggest both AI Tools integration in educational settings in terms of code generation, but in task completion ChatGPT is slightly faster than GitHub Copilot. While GitHub Copilot was found with stronger impact in collaboration, both tools are equally in critical thinking and adaptability. Based on these findings, this study provides recommendation for integration AI Tools in curriculum design and teaching strategies in computer science education.
References
[1]GitHub. (n.d.). GitHub Copilot plans. Retrieved August 31, 2024, from https://github.com/features/copilot/plans
[2]ChatGPT. (n.d.). ChatGPT. Retrieved August 31, 2024, from https://chatgpt.com
[3]Pekkola, E. (Ed.). (2018). Theoretical and methodological perspectives on higher education management and transformation: An advanced reader for PhD (p. 218).
[4]Bryman, A. (2012). Social research methods (4th ed.).
[5]Chou, C. P. (2014). Bray, M., Adamson, B., & Mason, M. (Eds.). Comparative education research: Approaches and methods (Book review). CEPS Journal, 4(3), 131–136. DOI: https://doi.org/10.26529/cepsj.199
[6]Morse, J. M. (2016). Mixed method design: Principles and procedures. Routledge. https://doi.org/10.4324/9781315424538 DOI: https://doi.org/10.4324/9781315424538
[7]Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Handbook of mixed methods in social and behavioral research (1st ed., pp. 13–50). https://www.worldcat.org/title/49558717
[8]Gupta, B. N., & Gupta, N. (2022). Research methodology. SBPD Publications.
[9]Cameron, R. (2009). A sequential mixed model research design: Design, analytical, and display issues. International Journal of Multiple Research Approaches, 3(2), 140–152. DOI: https://doi.org/10.5172/mra.3.2.140
[10]Peng, G. C., Nunes, J. M. B., & Annansingh, F. (2011). Investigating information systems with mixed-methods research. In Proceedings of the IADIS International Workshop on Information Systems Research Trends, Approaches and Methodologies (pp. 39–48). Sheffield.
[11]Hyseni, L. N., & Dika, Z. (2017, August). An integrated framework of conceptual modeling for performance improvement of the information systems. In 2017 Seventh International Conference on Innovative Computing Technology (INTECH) (pp. 174–180). IEEE. DOI: https://doi.org/10.1109/INTECH.2017.8102438
[12]Freeman, J. V., & Julious, S. A. (2007). The analysis of categorical data. Scope, 16(1), 18–21.
[13]Strzelecki, A. (2024). Students’ acceptance of ChatGPT in higher education: An extended unified theory of acceptance and use of technology. Innovative Higher Education, 49(2), 223–245. DOI: https://doi.org/10.1007/s10755-023-09686-1
[14]Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024). The effects of generative AI on computing students’ help-seeking preferences. In Proceedings of the 26th Australasian Computing Education Conference (pp. 39–48). DOI: https://doi.org/10.1145/3636243.3636248
[15]Jo, H. (2024). From concerns to benefits: A comprehensive study of ChatGPT usage in education. International Journal of Educational Technology in Higher Education, 21(1), 35. DOI: https://doi.org/10.1186/s41239-024-00471-4
[16]Peslak, A., & Kovalchick, L. (2024). AI for coders: An analysis of the usage of ChatGPT and GitHub Copilot. Issues in Information Systems, 25(4), 252–260.
[17]Popovici, M.-D. (2023). ChatGPT in the classroom: Exploring its potential and limitations in a functional programming course. International Journal of Human–Computer Interaction, 1–12.
[18]Kapakos, W. A., & Fulk, H. K. (2024). GitHub Copilot: Introducing the artificial intelligence tool in an information systems course. Issues in Information Systems, 25(4).
[19]Zhang, B., Liang, P., Zhou, X., Ahmad, A., & Waseem, M. (2023). Practices and challenges of using GitHub Copilot: An empirical study. arXiv preprint arXiv:2303.08733. DOI: https://doi.org/10.18293/SEKE2023-077
[20]Jošt, G., Taneski, V., & Karakatič, S. (2024). The impact of large language models on programming education and student learning outcomes. Applied Sciences, 14(10), 4115. DOI: https://doi.org/10.3390/app14104115
[21]Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022). Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1–7). DOI: https://doi.org/10.1145/3491101.3519665
[22]Denny, P., Kumar, V., & Giacaman, N. (2023). Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education (Vol. 1, pp. 1136–1142). DOI: https://doi.org/10.1145/3545945.3569823
[23]Wermelinger, M. (2023). Using GitHub Copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education (Vol. 1, pp. 172–178).
[24]Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187–192. DOI: https://doi.org/10.1126/science.adh2586
[25]Yetiştiren, B., Özsoy, I., Ayerdem, M., & Tüzün, E. (2023). Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. arXiv preprint arXiv:2304.10778.
[26]Beer, R., Feix, A., Guttzeit, T., Muras, T., Müller, V., Rauscher, M., Schäffler, F., & Löwe, W. (2024). Examination of code generated by large language models. arXiv preprint arXiv:2408.16601.
[27]Georgsen, R. E. (2023). Beyond code assistance with GPT-4: Leveraging GitHub Copilot and ChatGPT for peer review in VSE engineering. In Norsk IKT-konferanse for forskning og utdanning (No. 2).
[28]Solohubov, I., Moroz, A., Tiahunova, M. Y., Kyrychek, H. H., & Skrupsky, S. (2023). Accelerating software development with AI: Exploring the impact of ChatGPT and GitHub Copilot. In CTE (pp. 76–86).
[29]Nizamudeen, F., Gatti, L., Bouali, N., & Ahmed, F. (2024). Investigating the impact of code generation tools (ChatGPT & GitHub Copilot) on programming education. In CSEDU (2) (pp. 221–229).
[30]Wermelinger, M. (2023). Using GitHub Copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education (Vol. 1, pp. 172–178). DOI: https://doi.org/10.1145/3545945.3569830
[31]Biswas, S. (2023). Role of ChatGPT in computer programming. Mesopotamian Journal of Computer Science, 2023, 9–15. DOI: https://doi.org/10.58496/MJCSC/2023/002
[32]Cagáňová, D., & Horňaková, N. (2024). Soft skills development in the context of digitalization: Insights from international projects. In International Conference Innovation in Engineering (pp. 290–299). Cham: Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-031-61582-5_24
[33]Dolev, N., & Itzkovich, Y. (2020). In the AI era, soft skills are the new hard skills. In Artificial Intelligence and Its Impact on Business, 55.
[34]Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022). Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1 (pp. 27–43). DOI: https://doi.org/10.1145/3501385.3543957
[35]Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023). Programming is hard—Or at least it used to be: Educational opportunities and challenges of AI code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education (Vol. 1, pp. 500–506). DOI: https://doi.org/10.1145/3545945.3569759
[36]Marwan, S., Williams, J. J., & Price, T. (2019). An evaluation of the impact of automated programming hints on performance and learning. In Proceedings of the 2019 ACM Conference on International Computing Education Research (pp. 61–70). DOI: https://doi.org/10.1145/3291279.3339420
[37]Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher education: A systematic review of empirical research from 2011 to 2020. Education and Information Technologies, 27(6), 7893–7925. DOI: https://doi.org/10.1007/s10639-022-10925-9
[38]Nizamudeen, F., Gatti, L., Bouali, N., & Ahmed, F. (2024). Investigating the impact of code generation tools (ChatGPT & GitHub Copilot) on programming education. In CSEDU (2) (pp. 221–229). DOI: https://doi.org/10.5220/0012628000003693
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
