Design a web crawler that systematically downloads and indexes billions of web pages. Handle URL discovery, politeness policies, duplicate detection, distributed coordination, and content extraction at the scale of crawling the entire indexable web.
What to design
What are the core functional requirements for a web crawler?
What are the non-functional requirements? Consider crawl throughput, politeness, freshness, and storage.
Design the APIs and interfaces — crawl job management, URL submission, and content retrieval.
What storage architecture would you use for the URL database, page content, and crawl metadata?
How would you design the URL frontier to manage billions of URLs with prioritization?
How would you respect robots.txt and implement politeness policies?
How would you detect and handle duplicate content across different URLs?
How would you determine recrawl frequency to keep the index fresh without wasting resources?
How would you distribute the crawl across multiple nodes and handle node failures?