Lessons Learned at OpenAI (7-min read)

Now that AGI is expected to arrive within 2 to 5 years, an unprecedented opportunity is opening up for builders with high execution ability.

Designing Evals(evaluation) is an extremely tricky task. A good eval must be simple yet specific, providing fast feedback and a clear signal. An excellent Benchmark becomes an industry Schelling point, causing every researcher to optimize their models against it, thereby determining the direction of the technology.

Designing a Post-training data mix is more art than engineering, because human taste and creativity play a key role. In particular, implementing subjective capabilities such as emotional intelligence, humor, and creative judgment in models will be the next frontier of AI advancement. Also, products like ChatGPT canvas function not merely as interfaces but as tools for collecting Training signal by turning users' collaboration patterns into data.

For efficient AI research, internal tooling becomes a powerful competitive advantage. Rather than pouring massive compute into every experiment, it's important to design experiments that extract the maximum information per FLOP. Research results should also be distilled into a single clear claim and visual so that anyone can understand them.