Using Local LLMs for Agentic Coding

In response to price increases and usage-based billing structures for cloud models, this covers how to build an efficient coding environment using local LLMs. Local models may perform below SOTA models, but a deterministic harness can compensate for quality, and they offer excellent cost-to-performance.

For coding, Gemma 4 is recommended. Gemma 4 has a good balance between general tasks and code generation, and it supports Tools Use, Vision, Reasoning capabilities, making it suitable for IDE integration such as VS Code. In particular, the 26B A4B (MoE) model shows high efficiency in environments with 8-12GB VRAM.

The key components for running local models are as follows.

• Runtime (Inference Engine): Llama.cpp, MLX (Apple), ONNX Runtime, vLLM, etc.
• Model Manager: Ollama, LM Studio, Jan, etc.
• Harness: Tools that wrap the model, such as VS Code Copilot, Pi, etc.

For efficient use, it is recommended to use LM Studio's REST API to connect to a custom endpoint in VS Code. At this point, due to the nature of coding workloads, a context length of at least 100k is needed to maintain the system prompt and context.