Building RAG Applications with RNGD | FuriosaAI

RAG (Retrieval-Augmented Generation) is a key technology that goes beyond the limitations of relying on an LLM's pretrained knowledge, enabling the use of up-to-date data and domain-specific information. However, building a RAG system comes with challenges of complexity, latency, and energy consumption arising from integrating an LLM, an embedding model, and a vector database.

FuriosaAI's RNGD is a high-performance, low-power inference accelerator designed to address these issues. When running the Llama 3.1 8B model (FP8 precision), it processes 3,200–3,300 tokens per second (TPS), and a single server can support up to 60 concurrent users.

In particular, RNGD features an efficient 180W power profile and 48GB HBM3 memory, enabling cost-effective RAG operation with far less power than conventional GPUs. This is a major advantage for on-premises solutions or data center deployments.

As a real-world implementation example, it shows integration with Kotaemon, an open-source RAG project. Instead of the conventional approach of calling OpenAI's cloud, RNGD can be used as a local endpoint through configuration, strengthening data privacy and reducing latency.