AI Briefing#inference-server
The latest AI and developer news about #inference-server, with the original source and a short summary.
The latest AI and developer news about #inference-server, with the original source and a short summary.
The new release supports continuous batching, prefix-caching, and speculative decoding across NVIDIA, AMD, Intel, TPU, and Metal hardware.
Apache 2.0 licensed tool processes 491 questions per second on an RTX 4090 by reading content once for multiple decision queries.
A bug has been reported where llama.cpp's llama-server fails to pass the reasoning_effort parameter to the chat template.
SIE unifies serving of embeddings, reranking, OCR, and LLMs through a single OpenAI-compatible API.
It's a CLI tool that analyzes vLLM server metrics to diagnose queue pressure, TTFT, KV cache status, and more.
The llama.cpp server now supports native tool-calling capabilities, including file manipulation and shell command execution.
Ollama vulnerabilities reveal prompt/API key leakage and potential Windows RCE.