LaCy: What Small Language Models Should Learn Isn't Just About Loss

Small Language Models(SLMs) cannot fully internalize world knowledge due to their limited parameter size, and often produce outputs that diverge from facts. To compensate for this, they need the ability to delegate missing information via tokens, calling on external models, documents, or databases.

The key question is not simply "should we predict tokens with high loss", but which tokens should be learned and which should be delegated. While loss is useful for predicting mismatches with the correct token, it should not force every acceptable token—one that can still be a factual alternative within the pre-training context—to become a target for delegation.

To address this, a spaCy grammar parser was used to augment the loss signal, separating tokens the SLM can safely predict on its own from tokens where it should ask for help. Based on this philosophy, a pre-training method called LaCy was proposed.

In experiments, LaCy models learned more effectively which tokens to predict and where to delegate. As a result, FactScore improved in cascaded generation with larger models, and the approach outperformed SLMs trained with Rho or LLM-judge, while also being simpler and cheaper.