It's Time to Collect Data on How Software Is Developed

As LLM adoption spreads in development environments, the importance of collecting data not just on code generation but on 'how' developers build software is emerging. Existing source code or Git history is merely outcome-focused data, lacking the step-by-step process and contextual decision-making that developers went through, which limits improvements in LLM performance.

Continue is developing an open-source coding autopilot to fill this gap by automatically collecting development data. This tool captures implicit feedback generated during the process of developers interacting with LLMs (whether suggestions are accepted, code modifications, etc.), and uses it to build a development data engine that boosts LLM performance and ROI across the entire team.

The Value and Use of Development Data

• Solving the Problem of Data Staleness: Commercial LLMs like GPT-4 often fail to reflect the latest libraries or internal style guides, but this can be supplemented through self-collected development data.
• Proving ROI and Securing Budget: By clarifying the return on investment of LLM usage through collected data, this contributes to increasing LLM budgets within organizations and attracting top talent.
• Continuous Model Improvement: By identifying developers' complaints or challenges, models can be continuously tuned to resolve those issues.