GoScrapy: An Ultra-Fast Web Scraping Framework Built in Go

A high-performance web scraping framework that natively implements the structure of Python Scrapy in Go.

A single command, goscrapy startproject, automatically handles project structure generation, Go module initialization, and dependency resolution.

The core flow proceeds as Spider → Engine → Scheduler → Worker → Middleware → HTTP Client.

• When a Spider sends a request, the Engine passes it to the Scheduler.
• The Scheduler assigns an available worker from the Worker Queue and delivers the task.
• The Worker and Executor perform the HTTP request after passing through Middleware.
• The response returns in reverse order and is delivered to the Spider callback, and yielded records are exported via the PipelineManager to DB, CSV, files, etc.

It also comes with many built-in features. It includes export pipelines for CSV, JSON, MongoDB, Google Sheets, Firebase, and has built-in middleware such as Azure TLS and Dupefilter.

HTML parsing supports chainable CSS/XPath selectors, and the structure allows middleware and pipelines to be configured centrally in settings.go.

As practical examples, it provides a Google Maps scraper, bot detection evasion via Fingerprint Spoofing, and TUI real-time monitoring.

It is currently at the v0.x development stage, and the license is Business Source License (BSL), which allows use in commercial products but restricts reselling the framework itself as a competing service.