Breaking Through the 8GB Barrier

A personal project to build a fully local, offline companion robot was introduced, aimed at reducing the isolation felt by the author's disabled husband.

The setup is as follows.

• LLM: Mistral-7B-Instruct via llama.cpp
• Inference PC: an old Lenovo ThinkPad with 8GB RAM
• Speech recognition: faster-whisper (base, INT8) on Jetson Nano
• TTS: Piper (en_us-ryan-medium)
• OS: Linux Mint 22.3 Cinnamon 64-bit

The core bottleneck is 8GB RAM, making small quantized models and memory savings essential. The question was how to maximize usable RAM and performance in llama.cpp.

The comments gathered a lot of practical advice.

• Switching to smaller, newer models like Gemma 4 or Qwen 3.5
• Strictly limiting the context window, such as -c 2048 or -c 4096
• Reducing OS memory usage with a lightweight desktop like XFCE or running headless
• Using SSD/NVMe where possible
• Leveraging swap/zram to reduce out-of-memory issues
• Alternatives such as Kokoro for TTS and KoboldCPP/SillyTavern/LettuceAI for the frontend were also mentioned

Some comments also pointed out that interaction quality matters more than model choice. Things like interruptions while speaking, synchronizing streaming output with TTS, and designing long-term memory (RAG/memory files) are what actually determine the perceived quality.

Overall, this is a real-world case where concrete design tips and model recommendations were exchanged for building an offline voice companion system on low-spec hardware.