crawlerThe spider in this playground is a visual metaphor. Real crawlers are software. They request and read resources; they do not highlight, enlarge, tilt, or knock words off a website. Those effects make the act of discovery visible.
Start somewhere.
Then follow a link.
A crawler needs a starting point: a known URL, links from a previous crawl, or a sitemap supplied by a website. It keeps discovered addresses in a queue often called the URL frontier.
That queue is carefully managed. URLs may be normalized and checked against addresses already seen. A scheduler decides what to visit next, balancing freshness, priority, and the load placed on each host.
Read the room.
Before fetching pages, a well-behaved crawler checks the site's /robots.txt. This file tells matching crawlers which paths they may request. Responsible crawlers also pace requests, limit concurrency, and back off when servers struggle.
# A simple robots.txt example
User-agent: *
Disallow: /private-drafts/
Sitemap: https://example.com/sitemap.xmlA robots rule controls crawling. It is not a password, and it does not reliably keep a URL out of search. A noindex directive can tell supporting search engines not to index a page, but the crawler must be allowed to fetch it to see the directive. Protect confidential content with authentication.
Ask. Receive. Inspect.
The crawler makes an HTTP request, identifying itself with a user agent. The server responds with a status code, headers, and usually a body. A successful HTML response gives the crawler the source of a page.
200 OKA resource is available to process.301 / 308A permanent redirect points to a new address.404 / 410The resource is missing or gone.429 / 503Slow down or try again later.Some crawlers also render JavaScript in a browser engine to see content created after the initial HTML arrives. Others only parse the response. Rendering takes extra resources, so discovery and rendering may happen at different times.
A page is full
of possibilities.
The response is parsed into a structure the crawler can inspect. It can extract headings, visible text, metadata, canonical hints, and links. Relative links are resolved against the page's address to produce usable URLs.
<a href="/journal/curiosity">
Follow your curiosity
</a>
// Resolved against https://example.com
https://example.com/journal/curiosityNew eligible links are deduplicated and returned to the frontier. Crawling policy determines which hosts, paths, file types, and link relationships are followed. This repeating loop lets a crawler explore a connected part of the web.
Finding is only
the beginning.
Crawling collects resources. A search engine's indexing systems then analyze eligible content, group duplicates, choose representative URLs, and organize signals so pages can be retrieved efficiently.
For example, an inverted index maps a term to documents that contain it. When someone searches, a separate retrieval and ranking process evaluates relevant candidates. A page being crawled does not guarantee that it will be indexed or appear for a particular query.
/journal/curiosity/notes/asking-better-questions/reading-listThe web never
stands still.
Pages change, links break, and new documents appear. The scheduler revisits known URLs, using signals such as past changes and server responses to decide when another visit is useful.
Conditional requests can avoid downloading unchanged content. A server may respond with 304 Not Modified when a stored version is still current. There is no universal schedule: crawl frequency depends on the crawler, the site, and available resources.
Discover. Fetch. Understand. Repeat.
A little exploration makes a big web usable.