Vercel Releases AI SDK for Python with Experimental Jev evaluate() API

Vercel’s Python team has released the latest version of the AI SDK for Python, introducing an experimental evaluate() API designed to interact with Jev, a new class of AI model. Jev functions as a universal classifier that takes data and multiple-choice questions as input, returning structured JSON answers with confidence scores rather than generated text. This approach leverages the broad training of Large Language Models but optimizes for speed and cost in narrow decision-making tasks.

How Jev Works

Unlike traditional classifiers that require domain-specific training data, Jev utilizes pre-trained LLM weights to classify inputs without additional tuning. The API is built around a single function, evaluate(), which accepts a model, a state (string or JSON), and a set of questions. These questions can be defined as ChoiceQuestion (selecting one answer), ScoreQuestion (rating on a scale), or NoulQuestion (estimating probability).

Practical Experiments

The article details two experiments to test Jev’s capabilities:

• Language Classification: The author tested Jev’s ability to distinguish between English and Python code in a REPL context. While Jev performed better than a custom-trained classifier, it still struggled with ambiguous inputs like partial expressions.
• Code Generation: Attempting to use Jev to write Python code by selecting characters or words proved ineffective. The author instead used an LLM (GPT-5.6) to plan the code structure, then guided Jev to build a Python Abstract Syntax Tree (AST) node by node. This produced syntactically valid but often incorrect code, highlighting the difficulty of using classifiers for generative tasks.

Availability

Developers can install the SDK using uv add ai and access Jev by setting an AI Gateway API key. The experiments and source code are available on GitHub for further exploration.