Getting Started with ggml, the Foundation of Local LLMs

ggml is a C/C++-based machine learning library specialized for Transformer inference. It is the core technology underlying major local LLM execution tools such as llama.cpp, ollama, and LM Studio.

Key Advantages:

• Minimalism: The core library consists of fewer than 5 files, making it extremely lightweight.
• Easy Compilation: It can be built using just GCC or Clang, without complex build tools.
• Lightweight and Efficient: The binary size is under 1MB, with very low memory overhead.
• High Compatibility: Supports a variety of hardware including x86_64, ARM, Apple Silicon, and CUDA.
• Quantization Support: Tensor quantization can reduce memory usage and optimize performance.

Caveats:

• Not all tensor operations are supported on every backend (CPU, CUDA, etc.).
• Low-level programming knowledge is required, and frequent API changes may occur due to active development.

Core Concepts:

• ggml_context: A container that holds tensors, graphs, and data.
• ggml_cgraph: A computation graph representing the order of operations to be passed to the backend.
• ggml_backend: An interface that executes operations, such as CPU, CUDA, or Metal.
• ggml_backend_buffer_type: Acts as a memory allocator tied to each backend.