How to Fix Crawl Errors in Google Search Console: A Practical Walkthrough

You open Google Search Console, click the Pages report, and see a wall of red labels: Not found (404), Server error (5xx), Redirect error, Blocked by robots.txt, Soft 404. Each label means something different, and each one needs a different fix. Applying the wrong remedy is how sites end up with the same errors six months later.

This guide is built as a diagnostic table you can work through. Find the exact label you see in your own account, apply the matching fix, then validate and monitor the recrawl. No theory, no filler.

First: the old “Crawl Errors” report no longer exists

If you are searching for a report literally named Crawl Errors, you will not find it. Google retired that report years ago and split its functionality across three places:

• Indexing > Pages (often called the Page indexing report): per-URL errors and exclusions, grouped by reason.
• Settings > Crawl stats: host-level problems such as DNS failures, connection timeouts and robots.txt fetch failures.
• URL Inspection tool: single-URL live testing, the fastest way to confirm a fix before you request validation.

Almost everything people call a “crawl error” today lives in the Pages report. That is where we start.

How to read the Pages report before you fix anything

1. Open Indexing > Pages in the left sidebar.
2. Look at the two blocks at the top: Not indexed and Indexed. Click Not indexed.
3. Scroll to the table titled Why pages aren’t indexed. Each row is a reason, with a URL count and a trend arrow.
4. Sort by count, but do not blindly fix the biggest number first. Click into a reason and check the actual URLs. A 12,000-URL error caused by faceted filter parameters is a very different job from 12 broken product pages.
5. Export the URL list (the export button is top right) so you have a working file. The interface caps the displayed sample at 1,000 URLs per reason.

Key mental model: some of these labels are genuine errors that hurt you, some are expected behaviour that you deliberately configured, and some are Google’s judgement calls about page quality. Treat them differently.

The diagnostic table: label, meaning, action

Search Console label	What Googlebot actually saw	Severity	Your action
Not found (404)	Server returned 404 or 410	Low to medium	Redirect if a replacement exists, otherwise leave it and clean internal links
Server error (5xx)	500, 502, 503, 504 response or timeout	Critical	Check server logs, fix the root cause, then validate
Redirect error	Loop, chain too long, empty or bad redirect target	High	Flatten the chain to a single hop pointing at a 200 URL
Blocked by robots.txt	A Disallow rule stopped the fetch	Depends on intent	Remove the rule if the page should rank, otherwise ignore
Soft 404	200 status but the page looks empty or says “not found”	Medium	Return a real 404/410, or add real content, or redirect
Blocked due to access forbidden (403)	Server refused Googlebot	High if public page	Fix firewall / WAF / bot rules, whitelist verified Googlebot
Blocked due to unauthorized request (401)	Page behind login or HTTP auth	Depends on intent	Remove from sitemap, or open access if it should be public
Excluded by ‘noindex’ tag	Meta robots or X-Robots-Tag noindex found	Depends on intent	Remove the tag if the exclusion was accidental
Crawled – currently not indexed	Fetched fine, Google chose not to index	Quality signal	Improve depth, uniqueness and internal linking
Discovered – currently not indexed	Known URL, never crawled yet	Crawl budget signal	Speed up the server, prune thin URLs, strengthen internal links
Duplicate without user-selected canonical	Google picked a different canonical	Medium	Declare an explicit canonical and differentiate the content

Now the detail. Each section below follows the same shape: how to confirm the diagnosis, the fix, what not to do. quattr.com makes the same point with more data.

1. Not found (404)

How to confirm

Open a sample URL in the URL Inspection tool and click Test live URL. If the HTTP response is 404 or 410, the label is accurate. Also run a quick curl -I https://yoursite.com/the-url from a terminal to see the raw header.

The fix

404s are not automatically bad. Google treats them as a normal part of the web. What matters is why that URL was crawled in the first place. Click the URL in the report, then use Referring page data (or your own crawl of the site) to find the source.

Where the 404 comes from	Correct fix
Internal link or menu item	Edit the link. Do not paper over it with a redirect.
XML sitemap	Regenerate the sitemap. Sitemaps must only contain indexable 200 URLs.
External backlink to a deleted page	301 redirect to the closest equivalent page.
Old URL from a migration	301 redirect one to one, not to the homepage.
Deleted product or expired listing with no equivalent	Return 410 Gone and leave it. This drops out of the index faster than 404.
Garbage URL invented by a scraper or broken script	Ignore. 404 is the correct answer.

What not to do

• Do not mass-redirect every 404 to the homepage. Google reads bulk irrelevant redirects as soft 404s, so you trade one error label for another.
• Do not add 404 URLs to your sitemap to “get them recrawled”.
• Do not panic at a four-figure 404 count on a large ecommerce site. Check a sample of 20 URLs first. Discontinued SKUs are normal.

2. Server error (5xx)

This is the one to fix today. A 5xx tells Googlebot your server failed, and sustained 5xx responses cause Google to slow down crawling of your entire site. Pages already indexed can eventually drop out. Originally covered on https://pepperlandmarketing.com.

How to confirm

Run the live test in URL Inspection. If it returns 200 now, the error was transient. If it still fails, you have a live outage. Cross-check Settings > Crawl stats > By response to see whether 5xx responses are a spike or a steady baseline.

The fix, by status code

Code	Typical cause	Where to look
500	Application crash, PHP fatal error, bad plugin or theme update, database connection failure	PHP error log, WordPress debug log, recent deploys
502	Bad gateway between reverse proxy and backend	Nginx / PHP-FPM logs, upstream health, container restarts
503	Overload, maintenance mode left on, rate limiting hitting Googlebot	Resource graphs, maintenance plugins, CDN rate limit rules
504	Backend timeout, slow query, unindexed database table	Slow query log, TTFB on the failing template

Practical checklist

1. Search your server access log for Googlebot plus the failing path. You will usually see the exact timestamp and status.
2. Compare the 5xx spike date in Crawl stats with your deploy history. Nine times out of ten they line up.
3. If the errors only hit Googlebot and not human visitors, your CDN or security layer is rate limiting the crawler. Whitelist verified Googlebot by reverse DNS, never by user agent string alone.
4. If you are doing planned maintenance, serve 503 with a Retry-After header and keep it under a few hours. That is the one time 5xx is the right answer.

3. Redirect error

Search Console groups several distinct failures under one label. Googlebot follows a limited number of hops, so anything longer is abandoned.

The four causes

• Redirect loop: A points to B, B points back to A. Classic cause: an HTTP-to-HTTPS rule fighting a www-to-non-www rule, or a trailing slash rule fighting a CMS rewrite.
• Chain too long: five or more hops before a 200. Common after several site migrations stacked on top of each other.
• Bad or empty redirect URL: the Location header is malformed, relative when it should be absolute, or blank.
• Redirect target is itself an error: the chain ends on a 404 or 5xx.

The fix

1. Run each sample URL through a redirect trace tool or curl -ILk https://yoursite.com/url to see every hop.
2. Rewrite your rules so every redirect reaches its final destination in one hop.
3. Enforce a single canonical format sitewide and put those rules in a deliberate order: protocol first, then host, then trailing slash, then legacy path mappings.
4. Check that redirect plugins are not duplicating rules already handled at server or CDN level. Two layers each doing “add trailing slash” is a textbook loop.
5. Confirm the final URL returns 200 and is not itself blocked or noindexed.

Note: the separate label Page with redirect under Not indexed is not an error. It simply means the URL redirects, so Google indexed the target instead. That is working as intended.

4. Blocked by robots.txt

This one is only a problem if the block was unintentional. Half the URLs listed here are usually cart pages, admin paths and internal search results that you blocked on purpose.

How to confirm

1. Open https://yoursite.com/robots.txt directly in a browser.
2. Use the robots.txt report in Search Console (Settings > robots.txt) to see the fetched version, its status and the last fetch time.
3. Test a specific blocked URL with URL Inspection. The tool names the exact rule that is blocking it.

The fix

• Page should be indexed: remove or narrow the Disallow rule. Remember rules match by prefix, so Disallow: /p blocks /products/, /pricing/ and /blog/post-name.
• Page should not be indexed but keeps appearing in search results without a description: robots.txt is the wrong tool. Allow crawling and add a noindex tag instead. Google cannot see a noindex tag on a page it is forbidden to fetch.
• Page is genuinely private: leave the block, and remove those URLs from your XML sitemap so the conflict stops being reported.
• Blocked CSS or JS files: unblock them. Google needs them to render your pages properly.

Also check for the most damaging variant of all: a staging-site Disallow: / that got deployed to production. If your robots.txt file itself returns a 5xx, Google may stop crawling the whole site, so make sure it responds with 200 or 404 and nothing else.

5. Soft 404

A soft 404 means the URL returned HTTP 200 OK but the content looked like an error or an empty page to Google. It is a mismatch between what your server says and what the page shows.

Typical triggers

• A custom “Page not found” template that returns 200 instead of 404.
• Empty category, tag or search result pages with zero items.
• Out-of-stock product pages showing only “This item is unavailable”.
• JavaScript-rendered pages where the content never loads for the crawler, leaving a blank shell.
• Very thin pages: a heading, a breadcrumb and two lines of text.
• Bulk redirects sending unrelated URLs to the homepage.

The fix, in decision order

1. Should the page exist at all? If no, return a proper 404 or 410. Check your CMS or error template actually sends the right status header, not just a friendly message.
2. Does an equivalent page exist? If yes, 301 redirect to it, one to one and topically relevant.
3. Should the page exist but is currently empty? Add substance: for an out-of-stock product, keep the description, specs, reviews and link to alternatives rather than stripping the page.
4. Is content rendered client-side? Test with URL Inspection and read the rendered HTML tab. If the main content is missing there, move to server-side rendering or prerendering for that template.
5. Are these auto-generated empty archives? Noindex empty tag and filter pages, and stop linking to them internally.

6. Access errors: 403 and 401

These are increasingly common in 2026 because so many sites now run aggressive bot-blocking rules against AI crawlers, and legitimate Googlebot gets caught in the net.

The fix

• Run a live test in URL Inspection. If the live test succeeds but the crawl failed, you have an intermittent security rule.
• Ask your CDN or WAF provider for logs filtered on Googlebot. Look for challenge pages, JavaScript checks and geo blocks. Googlebot crawls mostly from US IP ranges, so geo-restricting to one country blocks it.
• Verify Googlebot properly with reverse DNS lookup, then allow it explicitly. Never rely on user agent matching only, because that is easy to spoof.
• If the page truly requires authentication, remove it from the sitemap and internal links so it stops being reported.

7. Crawled and Discovered – currently not indexed

These are not crawl errors, but they show up in the same report and cause the most confusion, so here is the short version.

Crawled – currently not indexed

Google fetched the page and decided it was not worth indexing. Technical fixes will not help here. What does help:

• Merge thin pages covering near-identical topics into one stronger page.
• Add original substance: data, examples, images, expert commentary. Templated text with a swapped city name will not get indexed.
• Link to the page from relevant, already-indexed pages. Orphan pages rarely earn indexing.
• Give it time. Newer pages on lower-authority sites can sit in this state for weeks.

Discovered – currently not indexed

Google knows the URL exists but has not crawled it yet, usually a crawl budget or server capacity decision. Improve server response times, cut the number of low-value URLs (parameters, filters, endless pagination), and tighten your internal link structure so priority pages are reachable within three clicks of the homepage.

Validate the fix and monitor the recrawl

Fixing the code is half the job. Search Console will not update on its own schedule fast enough for you to know whether you succeeded. This explainer is clearer than most.

Step by step

1. Test one URL live. URL Inspection, paste the URL, click Test live URL. Confirm the status code and rendered content are now correct.
2. Request indexing for that single URL if it is an important page. This is for spot checks only, not for hundreds of URLs.
3. Go back to the Pages report, click the error reason, and hit Validate fix. Google then recrawls a sample from the list.
4. Watch the validation states: Started, then Passed, Failed or Not started. If a single URL in the sample still fails, the whole validation fails, so make sure the fix is applied sitewide before you start.
5. Resubmit your sitemap if the fix involved URL changes. Go to Indexing > Sitemaps and check the Last read date updates.
6. Do not click Validate fix twice. Starting a new validation resets the process and costs you days.

Realistic timelines

Action	Expected wait
Live test reflects your change	Immediate
Single URL recrawled after Request indexing	Hours to a few days
Validation moves from Started to Passed	Typically 1 to 4 weeks
Pages report chart reflects the drop	Report data lags by a few days
Large site fully reprocessed	Several weeks to a few months

Monitoring after the fix

• Check Settings > Crawl stats monthly. Watch Average response time and the By response breakdown. Rising response times predict crawl problems before they become errors.
• Turn on Search Console email alerts and actually read them.
• Add a Pages report check to your post-deploy routine. Most new crawl errors are self-inflicted by a release.
• Keep an eye on the trend arrow next to each reason, not the absolute number. A flat 800 404s on a big catalogue is fine. A number climbing 30 percent week over week is not.

Five mistakes that keep crawl errors coming back

1. Redirecting everything to the homepage. Converts 404s into soft 404s and loses any link value.
2. Using robots.txt to hide pages from the index. Blocked pages can still appear in results. Use noindex and allow the crawl.
3. Leaving non-200 URLs in the sitemap. Your sitemap is a recommendation list. Filling it with errors damages trust in the whole file.
4. Fixing one URL then validating the whole reason. The validation samples multiple URLs and fails if any one still errors.
5. Treating every red label as urgent. Prioritise: 5xx and redirect loops first, then soft 404s and accidental blocks, then 404s, then indexing judgement calls.

FAQ

How long does it take Google to recrawl after I fix a crawl error?

A single URL submitted through URL Inspection is usually recrawled within a few days. A full validation across a reason group typically takes one to four weeks, and large sites can take longer. There is no way to force it faster, and repeatedly clicking Validate fix restarts the clock rather than speeding it up.

Do 404 errors hurt my rankings?

Not directly. Google has stated repeatedly that 404s are a normal part of the web and do not carry a ranking penalty. The indirect damage is real though: broken internal links waste crawl budget, users hit dead ends, and backlinks pointing at 404 pages pass no value. Fix the ones with links or traffic, ignore the rest.

How do I fix “Crawled – currently not indexed”?

Treat it as a content quality signal rather than a technical bug. Confirm the page renders fully, then improve it: add unique depth, merge near-duplicates, and link to it from established pages on your site. Requesting indexing on a thin page will not change Google’s mind.

What is the difference between a 404 and a soft 404?

A 404 returns the HTTP status code 404, telling Google clearly that the page is gone. A soft 404 returns 200 OK while showing error-like or empty content, which is contradictory. Google flags soft 404s because they waste crawl resources. The fix is to make the status code match reality.

Should I remove URLs with the Removals tool to clear crawl errors?

No. The Removals tool hides URLs from search results for about six months and does nothing to fix the underlying error. Use it only for urgent removals such as leaked private data, then apply the real fix, either noindex, a 404 or a redirect.

Why did my crawl errors spike suddenly with no changes on my side?

Check three things: your host or CDN may have changed security or rate limiting rules, an SSL certificate may have expired, or a plugin auto-update may have altered redirect behaviour. Compare the spike date in Crawl stats with your hosting and deploy timelines. In almost every case the correlation is obvious once you line up the dates.

Can I fix crawl errors without Search Console access?

You can find broken URLs with a third-party crawler and check status codes yourself, but you cannot see what Googlebot specifically experienced, and you cannot trigger validation or request indexing. If you own the site, verify it in Search Console first. Every other approach is guesswork.

Quick action plan

1. Open Indexing > Pages and export the Not indexed URL list.
2. Handle 5xx and redirect errors the same day.
3. Audit robots.txt for accidental blocks.
4. Sort soft 404s into: delete, redirect, or improve.
5. Clean internal links and sitemaps causing 404s.
6. Validate each reason once, then leave it alone.
7. Recheck Crawl stats in 30 days and confirm the trend is down.

Work through the table at the top, match your label, apply the fix, validate. That is the entire job.