From Zero to Production with AI21 Studio

AI21 Studio compared Jurassic-1 prompt engineering with custom models using an AG News news classification case. Based on the task of choosing one of 4 topics from a title and summary, they first experimented with zero-shot and few-shot prompts.

In zero-shot, J1-Large achieved 32% accuracy and J1-Jumbo achieved 56.9% accuracy. Adding more correct-answer examples greatly improved performance, and the J1-Jumbo prompt with 16 examples recorded 86% accuracy on the test set.

However, as prompts got longer, every request had to re-read all the examples, increasing latency, power, and cost. There was also a 2048 token input limit, which made scaling difficult for tasks involving long article bodies or more classes.

The custom model presented as an alternative could be trained with only a small amount of correct-answer data, with 50–100 examples recommended as a starting point. The trained model was served immediately on AI21 Studio and could be used right away just like calling an API; with only 10 examples it approached the level of J1-Large, and at 80 examples it surpassed the few-shot performance of J1-Jumbo.

The speed difference was also significant. While the general model took about 250ms to process a 16-example prompt, the custom model responded in under 50ms, making it more than 5x faster.

The difference in security was also clear. In prompt injection attacks, when the malicious phrase was repeated 2 or more times, half of the outputs from the prompt engineering approach became toxic, and when repeated 3 or more times, virtually all outputs turned toxic. For the custom model, the attack only worked after 4 or more repetitions, and the success probability was also lower.

In the end, AI21 Studio concluded that custom models make it easier to balance accuracy, cost, latency, and safety, giving them an advantage for production deployment.