Loading…
Design a Web Crawler is a medium system design interview question covering BFS, politeness, deduplication, distributed systems — the kind asked at FAANG (Facebook/Meta, Apple, Amazon, Netflix, Google) and AI-first companies like Anthropic, OpenAI, and Nvidia.
Design a large-scale web crawler. The system should systematically browse the internet, download pages, and extract links for indexing.
Interactive walkthrough on Pro
Upgrade to Pro to unlock the clarifying questions, architecture, capacity calculations, tradeoffs, follow-up answers, and the model answer.
What's inside this guide
Design rounds are decided on capacity estimates. Both tools are free.
Longer written walkthroughs of the ideas behind it.
What each one means, and how interviewers probe it.
Fully unlocked, no account needed — solutions, walkthroughs and follow-up answers included.
Or start the Study Plan — a week-by-week order for your loop, and week 1 is entirely free.
A full interview loop is more than one round. These are the others, each with what is on the site for it.