SK Planet Developers' Experience Using GitHub Copilot (AI-assisted Coding and Developer Productivity #2)

SK Planet conducted a developer productivity experiment through the adoption of GitHub Copilot from April to August 2024. This experiment, involving a total of 30 developers, focused on quantitatively measuring the actual effectiveness of AI coding tools across various roles including backend, frontend, and data engineering.

Productivity Improvement Effects

The experiment results showed that using Copilot reduced average development completion time by 41.7% and improved development speed by approximately 1.71x. This is similar to Microsoft's previous research results (55.76% reduction), demonstrating that meaningful productivity gains occur in real work environments as well. Notably, the standard deviation of completion time among Copilot users decreased, confirming an Augmentation effect that reduces the performance gap based on skill level between senior and junior developers.

Key Use Cases

Developers utilized Copilot in various areas beyond simple code autocompletion.

• Code Generation and Refactoring: Generated code from API specifications or comments based on natural language prompts, and improved readability by separating duplicate code into Methods.
• Legacy Code Analysis: Received detailed explanations of the intent and procedures of complex legacy code using the /explain command, and automatically generated Javadoc-style documentation using /doc.
• Data Processing: When performing schema change work on approximately 200 tables and 5,000 columns, automatic comment generation completed 2 days' worth of work in just 2 hours.

Developer Experience and Considerations

In the survey, 87% of respondents positively evaluated that the adoption of Copilot would be helpful, with almost no negative opinions. However, due to the risk of Hallucination characteristic of generative AI, a process for identifying and correcting generated code is essential. Additionally, limitations were pointed out, such as limited support for iOS app development environments and constraints in specialized areas like blockchain smart contracts (Solidity). Based on this feedback, SK Planet plans to continue education and case-sharing efforts to establish an AI-based development culture.