Skip to content

Web Crawler

How web crawlers work, the difference between crawling and scraping, and how crawlers are used with screenshot capture tools.

A web crawler, also called a spider or bot, is a program that discovers web pages by following links. It starts from a list of seed URLs, downloads each page, extracts the links it contains, and adds new URLs to a queue. Repeating this builds a map of a site or of the wider web.

Search engines run the best-known crawlers to index pages. Other crawlers check sites for broken links, build archives, audit SEO, or collect a list of URLs for later processing. Well-behaved crawlers identify themselves with a user agent string, follow the Robots Exclusion Protocol published as RFC 9309 in robots.txt, and limit how fast they request pages.

How web crawlers work

  1. Start with seed URLs or a sitemap.
  2. Check robots.txt for rules that apply to the crawler.
  3. Fetch a page and record its status and content.
  4. Extract links, normalize them, and skip ones already seen.
  5. Add new URLs to the queue and continue until a limit is reached.

Modern sites that build content with JavaScript may need a headless browser to render before links appear.

Crawling vs scraping

Crawling finds pages. Scraping extracts specific data from pages, such as prices or titles. A crawler often feeds a scraper, but the two can run separately: a scraper may work on a fixed list of URLs without any link discovery.

Where web crawlers are used

  • Search engine indexing.
  • Site audits for broken links, redirects and missing metadata.
  • Web archives that preserve pages over time.
  • Building a URL list for screenshots or content review.

Common mistakes

  • Ignoring robots.txt and site terms.
  • Requesting pages too fast and overloading the server.
  • Following URL parameters endlessly, such as calendars and filters.
  • Crawling pages behind a login without permission.

Taking the same screenshots again and again?

Temposhot does not crawl. Website Capture captures the URLs you enter, and Bulk insert accepts a list exported from a crawler or sitemap. See website screenshot automation.

Common questions

It depends on the site's terms, the data involved and local law. Follow robots.txt and the site's terms, and get advice for commercial use.

What is a sitemap?

An XML file that lists a site's URLs, which crawlers use as a starting point.

Sources

Taking the same screenshots again and again?

Temposhot captures a screen, window or area on a timer and exports the set as a PDF, images, ZIP or MP4. The free browser tools fix single images.

Download for MacSee features

Related resources

Related terms

All glossary terms