Crawlee, a Data Collection Library for AI/LLM, Released

Developed by Apify, Crawlee is a Node.js-based web scraping library designed to efficiently collect the data needed for training and using AI, LLM, and RAG models.

Key features include:

• Hybrid crawling approach: Supports both lightweight HTTP crawling (using Cheerio, JSDOM) and browser crawling (using Playwright, Puppeteer) for sites requiring JavaScript rendering, all through a single interface.
• Bot blocking evasion: Designed to behave like a human to bypass the latest bot protection measures, with HTTP2 support and browser TLS fingerprint spoofing capabilities.
• Operational automation features: Provides integrated request queue management (persistent queue), autoscaling, proxy rotation, error retries, and data storage plugins.
• Flexible environment: Based on the Node.js environment, with Crawlee for Python also available separately for Python users.

It is released under the Apache 2.0 license, allowing free use for both personal and commercial purposes.