401 Unauthorized: meaning and SEO impact

What is 401 Unauthorized

401 Unauthorized is an HTTP status from the 4xx group (client errors) that the server returns when a request requires authentication but none was provided or the credentials failed verification. Along with the status, the server typically sends a WWW-Authenticate header indicating which authentication method to use.

The name 'Unauthorized' is slightly misleading — it actually refers to a lack of authentication, not authorization. A more accurate name would be 'Unauthenticated', which is exactly the meaning codified in RFC 9110.

401 vs 403: the difference

Both codes indicate an access refusal, but for different reasons:

Impact on SEO and indexing

When Googlebot requests a URL and receives a 401 in response, it cannot read the page content. The result: the page is not indexed or gets dropped from the index. If 401 appears on public pages, it is a serious SEO problem:

• The page loses rankings and drops out of the index within a few weeks.
• Crawl budget is wasted: the crawler visits the URL but receives nothing.
• Link equity pointing to blocked pages is not passed further.
• Errors will appear in Google Search Console under Coverage → Excluded pages.

Common causes of 401 on public pages

1. HTTP Basic Auth on the entire site or section. Often set up on staging and then forgotten before launch, or accidentally applied to the production environment.
2. Misconfigured CDN or reverse proxy. The intermediate layer requires a token or signature that Googlebot does not have.
3. Expired API tokens. Pages generated via an API with an expired key return 401 instead of content.
4. Application code error. A middleware or guard accidentally requires authentication where it should not.
5. WordPress or other CMS after an update. Content-protection plugins may change settings during an update.

How to fix 401 on public pages

Diagnosis and resolution workflow:

1. Find problematic URLs via Google Search Console → Coverage or a crawler (Screaming Frog).
2. Check the response code with curl — confirm the problem is reproducible.
3. Identify where authentication is configured: server (Apache/Nginx), CDN, or application middleware.
4. Remove the authentication requirement from public URLs or add an exception for them.
5. If staging needs protection — add robots.txt with Disallow: / or restrict access by IP.

# Check response code
curl -o /dev/null -s -w "%{http_code}\n" https://example.com/page
# Expected: 200 (or 301/302)
# If you see 401 — there is a problem

# Check with Basic Auth
curl -u login:password -o /dev/null -s -w "%{http_code}\n" https://example.com/page

# Apache — protect a directory while keeping public pages open
<Directory /var/www/html/admin>
    AuthType Basic
    AuthName "Admin Area"
    AuthUserFile /etc/.htpasswd
    Require valid-user
</Directory>

# Public pages are NOT touched — accessible without authentication

# Nginx — Basic Auth for /admin only, public pages remain open
server {
    # Public section
    location / {
        try_files $uri $uri/ =404;
    }

    # Protected section
    location /admin/ {
        auth_basic "Admin Area";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

// Next.js — API route with correct 401
import { NextRequest, NextResponse } from 'next/server';

export async function GET(request: NextRequest) {
  const token = request.headers.get('authorization');

  if (!token || !isValidToken(token)) {
    return NextResponse.json(
      { error: 'Authentication required' },
      {
        status: 401,
        headers: { 'WWW-Authenticate': 'Bearer realm="API"' },
      }
    );
  }

  return NextResponse.json({ data: 'protected content' });
}