Philadelphia Inquirer Uses Scrape AI Tool to Surface Hyperlocal News Leads

The Philadelphia Inquirer has implemented Scrape, an AI-driven tool designed to surface newsworthy leads for its hyperlocal newsletters from fragmented sources like municipal meetings, school calendars, and social media. Developed by Lenfest Institute AI Fellow Kevin Hoffman, the tool addresses the inefficiency of manual news gathering, which previously required editors to spend approximately 12 hours a week solely on finding items.

Workflow and Training Methodology

Scrape was built around the Inquirer’s existing editorial workflows rather than as a standalone generic solution. The development process involved a tight human-in-the-loop feedback loop spanning three to four months:

• Baseline Creation: The team used an editor’s existing manual tip sheets as ground truth for AI training and evaluation.
• Iterative Refinement: Scrape generated daily email digests which editors annotated to identify useful leads versus noise. This process allowed the team to refine prompts to filter out irrelevant content, such as routine business personnel changes or stock market news, while highlighting community-impact stories.
• Contextual Embedding: Instead of relying solely on geographic boundaries, the system incorporates detailed newsletter-specific descriptions, including audience demographics, relevant school districts, and explicit exclusion criteria.

Operational Impact and Scaling

The tool has transitioned from a pilot to a critical infrastructure component for the newsroom:

• Current Reach: Scrape currently supports six newsletters and is followed by nearly 20 journalists across the organization.
• Future Expansion: The Inquirer plans to expand the tool’s coverage to eight newsletters soon.
• Efficiency Gains: By automating the initial sifting process, editors can focus on writing and shaping the publication’s voice rather than data collection.

Key Lessons for Implementation

Hoffman highlights several strategic insights for organizations deploying similar AI tools:

• Start with Real Workflows: Begin by identifying a specific team performing the task manually and measuring their current effort to establish a clear baseline.
• Explicit Cost-Quality Trade-offs: Early attempts to balance search depth with API costs were inefficient. The team recommends defining acceptable cost-quality constraints from day one and designing prompts and architecture to adhere to them.
• Encode Editorial Judgment: Replace abstract concepts like "newsworthiness" with concrete, structured prompts that define audience interests, geographic scope, and specific exclusion rules.