Rethinking Lean Analytics for the Age of AI and Agents

The core principles of Lean Analytics, published in 2013, still hold true, but in the age of AI and agents, products work in fundamentally different ways, requiring a redefinition of metrics.

1. Drastic Reduction in Time to Value Unlike traditional SaaS, AI product users expect immediate, high-quality results from their very first attempt. As the learning curve disappears, Time to Competency collapses along with it, making the rate at which users get their first useful result a key metric.

2. Shift of Activation Toward Quality In AI products, activation is not simply a binary event of completing certain steps. Even after completing the funnel, the output may still be inadequate, so step-completion metrics need to be tracked alongside quality signals of the output.

3. Understanding the Direction of Engagement Long time spent is not inherently good—what matters is how that time is used. Time spent struggling to obtain useful results is a bad metric, while time where AI performs the task on the user's behalf or time spent exploring/creating should be distinguished as high-quality engagement. GitHub Copilot's suggestion acceptance rate is a representative example.

4. Changes in Stickiness and Moats Rather than barriers (Moats) that trap users, what matters is naturally blending into the user's workflow (Flow). The diversity of tasks a user performs acts as a new growth vector.

5. Token-Based Cost Structure and Profitability Due to the variable cost structure based on tokens, heavy power users can actually cause losses. Therefore, tracking Gross Margin based on active users and adopting outcome-based pricing models are essential.