New 'Scatter' sampler added to llama.cpp
Key point
An experimental Scatter sampler has been introduced to llama.cpp, which locally smooths the probability distribution to improve generation quality.
Details
A new experimental sampler called Scatter has been implemented in llama.cpp. This sampler finely adjusts the probability mass among the top candidates within the model's next-token probability distribution, preventing the generated output from becoming overly rigid.
The existing Temperature adjustment method smooths the entire distribution, allowing low-probability 'tail' tokens to participate in generation, which can lead to inappropriate word choices or context deviation. In contrast, Scatter has the following characteristics.
- Local Smoothing: Exchanges probability only between adjacent ranks within the selected top K candidates.
- Tail Distribution Protection: Smooths only the head of the probability distribution, preventing probability from leaking into the low-probability tail.
- Key Features: Includes fixed strength adjustment, adaptive strength adjustment via entropy feedback, and repeated token absorption functionality.
Currently, this feature is implemented as an experimental sampler, and while it is included within the default sampler chain, it is disabled (noop) by default.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.