MiniCPM-V-4.7-35B-A3B-gguf: 35B MoE Multimodal, Run Locally with llama.cpp

OpenBMB's MiniCPM-V 4.7 model has been released in GGUF format. As a multimodal LLM that understands both text and images simultaneously, it adopts a Mixture of Experts structure, maintaining active parameters at around 3B out of a total of 35B parameters. This design reduces memory load during inference, enhancing the feasibility of execution in local environments.

1 / 2

Various quantization versions are provided for compatibility with the llama.cpp ecosystem. Files with a wide range of bit counts, from Q4_K_M to F16, are included, allowing users to select based on their hardware specifications. Additionally, a version supporting dspark-based speculative decoding to improve processing speed is also distributed.

It can be used directly in major local inference frameworks such as llama.cpp, Ollama, and vLLM. Integration is possible in various ways, including launching an OpenAI-compatible API server or performing interactive inference directly from the terminal. Distributed under the Apache 2.0 license, it can be freely applied to commercial projects.

It is suitable for developers who need high-performance multimodal inference while reducing reliance on cloud APIs and maintaining data privacy. It is a useful option, especially when you want to experience the performance of a 35B-class model in on-device or edge computing environments.