Ollaya Launches Local Runtime for Open-Source Decision Models with TypeSafe SDK Compatibility

Ollaya has released a local runtime for open-source decision models, allowing users to run models such as Laya (Convai Innovations), Decider (Mapika), NLI (Moritz Laurer), and gliclass (Knowledgator) on their own hardware. These models answer structured questions (choice, score, yes/no) in a single forward pass rather than generating tokens.

Performance and Latency

The runtime achieves significantly lower latency than hosted alternatives. A five-question request to the Laya model takes approximately 8–10 ms on an NVIDIA RTX 4090. This is compared to the 236–276 ms median latency of the hosted TypeSafe Jev API, a figure derived from third-party benchmarks that include network overhead. The source notes that setups differ, so the comparison is an order-of-magnitude estimate.

API Compatibility

Ollaya is designed to be a drop-in replacement for the hosted Jev service. It maintains compatibility with the TypeSafe Python SDK 0.7.1, allowing developers to route requests to a local server (defaulting to 127.0.0.1) with minimal code changes by setting environment variables.

Calibration and Privacy

The models provide calibrated probabilities; Laya’s Expected Calibration Error (ECE) is 0.081 after temperature fitting, compared to 0.246 for Jev. Data remains local, with weights pulled from authors’ Hugging Face repositories and checked against SHA256 hashes. The runtime is licensed under Apache-2.0 and supports macOS, Windows, Linux, and Docker, with NVIDIA GPU support (CUDA 13) for accelerated inference.