JVM-only AI inference engine 'jinfer' released

The open-source engine jinfer has been released, enabling AI inference in JVM environments without Python runtimes, ONNX, or sidecar processes. The engine supports various modalities, including chat, vision, audio transcription, embeddings, reranking, and TTS, with the entire stack designed for the JVM.

Key Components

• Tok'n'Roll: A high-speed tokenizer for LLMs implemented in pure Java, with no dependencies.
• gguf / safetensors: Provides native read/write capabilities for two major model formats.
• jam: Quantized matrix multiplication routines leveraging the Vector API and optional native backends, delivering performance competitive with llama.cpp on CPU.
• jota: A Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo.

Ecosystem Integration and Status

It integrates with Spring AI and LangChain4j, including first-class support for GraalVM Native Image. Currently in the early release stage with CPU as the primary target, GPU support is under development via jota. It is released under the Apache 2.0 license and is led by the author of llama3.java.