AI BriefingLocal LLM VSCode
Key point
A setup connecting llama.cpp with VSCode to run Gemma-4-26B-A4B-IT-Q8_0 has been shared, along with its performance results.
Details
On an Ubuntu 24.04 system with an Intel i7-9700, 32GB DDR4, and a Radeon AI PRO R9700, Gemma-4-26B-A4B-it-Q8_0.gguf was run using llama.cpp server (vulkan).
The server configuration used options such as --ctx-size 80000, --threads 7, --gpu-layers 99, --batch-size 2048, --ubatch-size 512, --cache-type-k q8_0, --cache-type-v q8_0, and --cache-ram 8192, along with an updated chat template.
With this setup, GPU VRAM usage was about 83%, CPU usage was under 10%, and speed was around 1600 tps for prompt and 60 tps for response. It was noted that performance drops somewhat as the context grows larger.
On the VSCode side, instead of the Continue extension, the johnny-zhao.oai-compatible-copilot extension was used to connect the local LLM in a Copilot-like manner, and it was reported that Agent / Plan / Ask modes all work.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Ollaya Launches Local Runtime for Open-Source Decision Models with TypeSafe SDK Compatibility