AI Briefing#tokenizer
The latest AI and developer news about #tokenizer, with the original source and a short summary.
The latest AI and developer news about #tokenizer, with the original source and a short summary.
A new method specializes Small Language Model vocabularies to reduce measured inference costs by 77% while improving quality scores on domain-specific tasks.
New pricing and tokenizer changes make Haiku 5.5 cheaper per token but require more tokens per task, with savings dependent on prompt length and effort settings.
The Claude tokenizer was reproduced with approximately 15,000 entries, analyzing Anthropic’s rationale and benefits for reducing vocabulary size.
The tokenizer change in Claude Sonnet 5 can increase costs by up to 2x for the same tasks.
Liquid AI conducted an experiment to verify the autonomous production problem-solving capabilities of coding agents.
Analyzed the independent development of LLMs using public configurations, tokenizers, and weights.
This analysis examines methods for estimating LLM lineage by comparing the structure, tokenizer, and embeddings of public model files.
Liquid AI has unveiled a technique that expands the vocabulary of the LFM2.5-8B-A1B model from 65K to 128K without retraining from scratch.
Gigatoken is an open-source tokenizer that delivers speeds up to 1353x faster than the HuggingFace tokenizer.
LG AI Research has developed K-EXAONE, a large-scale AI model that maximizes both efficiency and performance.
An analysis of how differences in each model's tokenizer have a massive impact on actual API costs.
This proposes an algorithm that practically solves the theoretically intractable problem of finding the optimal Tokenizer via Integer Linear Programming (ILP).
Tokenizer support for the MiniCPM5 model has been added to the llama.cpp open-source project.
ztok, a multithreaded Zig-based tokenizer that is 2 to 5 times faster than existing tiktoken and Hugging Face tokenizers, has been released.
Opus 4.7's new tokenizer increased real-world costs by 12-27%.
A SQL parser, formatter, and LSP tool built by directly compiling SQLite's grammar has been released.
Opus 4.7 keeps the same price, but token counts can increase up to 1.35x.
A calculator comparing why the same prompt becomes more expensive on Opus 4.7.
Claude 4.7 produces 1.3–1.45x more tokens while showing a modest improvement in instruction compliance.
Claude Opus 4.7's new tokenizer was measured to use more tokens than officially stated.
Claude 4.7 uses 1.3–1.45x more tokens, while strict instruction following improved slightly.
Because of the new tokenizer in Opus 4.7, the same input burns through tokens up to 35% faster than 4.6.
Mistral collaborated with NVIDIA to release Mistral NeMo, a 12B model with enhanced Korean language performance.