Concurrent Web Crawler
A concurrent web crawler built with Go and Next.js — featuring bounded worker pools, crawl cancellation, SSRF protection, result analytics, interactive graph visualization, and Dockerized deployment.
Problem
I wanted to build a crawler that went beyond recursively fetching links. The goal was to understand bounded concurrency in Go, request cancellation, URL normalization, crawl limits, failure handling, and how to expose that backend through a useful full-stack interface.
Prior Art
Many crawler examples stop at recursively fetching pages or spawning goroutines without much control over concurrency, cancellation, security, or observability. This project intentionally adds those concerns so the crawler behaves more like a real service rather than a small scraping script.
Design Decisions
The crawler uses a bounded worker pool instead of creating an unbounded goroutine per URL. Crawls are organized by depth, context cancellation propagates from the browser through the HTTP request into the Go workers, and public target validation blocks localhost and private network addresses to reduce SSRF risk. The API also adds structured errors, request IDs, panic recovery, request logging, crawl limits, and graceful shutdown.
Architecture
The backend is written in Go and is split into crawler and HTTP server packages. The crawler handles URL normalization, HTML parsing, bounded concurrency, crawl depth, page limits, delays, cancellation, and target validation. The server layer exposes health and crawl endpoints with CORS, request IDs, structured JSON errors, recovery middleware, logging, and graceful shutdown. The frontend is built with Next.js, React, TypeScript, Tailwind CSS, and React Flow. It provides crawl configuration, loading and cancellation states, summary metrics, HTTP status and depth breakdowns, search, filtering, sorting, pagination, page details, JSON/CSV export, and an interactive crawl graph. Both services are Dockerized and deployed separately on Render.
Reflection
The most valuable part of this project was seeing how concurrency decisions affect the entire application. Worker limits, cancellation, request lifetime, redirects, URL validation, and error propagation all had to fit together correctly. It also reinforced that a useful backend project becomes much stronger when the frontend exposes the underlying behavior clearly instead of hiding it behind a single submit button.