Screaming Frog SEO Spider
A desktop crawler for technical audits: broken links, redirects, meta tag duplicates, and site structure visualization.
In brief
Screaming Frog SEO Spider is a desktop application that crawls a website like a search engine bot. It helps find technical errors, analyze titles, meta tags, redirects, and visualize internal link structure.
What the crawler does
Screaming Frog mimics Googlebot: it crawls a website following internal links and collects data about every URL. The free version scans up to 500 URLs; the paid version has no limit.
What Screaming Frog checks
- Broken links (404, 500, DNS errors) — both internal and external.
- Redirect chains and loops that waste crawl budget.
- Duplicate titles and meta descriptions, missing H1s, overly long descriptions.
- Indexability signals: response codes, canonical tags, meta robots.
- Page speed analysis.
Link graph visualization
The tool builds an interactive internal link graph, helping you spot orphan pages, isolated sections, and weak internal linking areas.
Crawl with the user-agent 'Googlebot' and enable robots.txt compliance to see the site through a search engine's eyes.
FAQ
Common questions
Crawl the site in sections or use alternatives like Sitebulb or Netpeak Spider. For larger sites, the paid license quickly pays for itself.
Yes, Screaming Frog supports cookies, HTTP authentication and form-based login—useful for auditing members-only areas.
You can adjust the number of threads and the delay between requests. For slower servers, reduce the speed to avoid overloading.
It analyzes the site at the local network level, does not rely on an external server, and can pull data from Google Analytics and Search Console.
Direct contacts
Discuss your project?
Share your goals and website context — I will suggest a practical next step.