← Back to blog

Technical SEO · July 17, 2026 · 7 min read

WordPress Crawl Budget: How Tag, Author, and Attachment Pages Bloat Your Index

Learn how WordPress tag archives, author pages, and attachment URLs drain your crawl budget — and the exact fixes to reclaim it for content that ranks.

By FluxWriter Team

WordPress crawl budget is one of those technical SEO concepts that sounds abstract until you see Googlebot spending its daily allowance on /tag/uncategorized/ and /author/admin/page/3/ instead of your actual content. For most WordPress sites, a significant slice of the crawlable URL surface is auto-generated archive and attachment pages that serve no audience and carry no ranking potential. Fixing this is less about grand strategy and more about knowing exactly which switches to flip.

What Crawl Budget Actually Means in Practice

Google assigns each site a crawl rate limit — a rough ceiling on how many requests Googlebot will make in a given period without slowing your server down. Separately, it tracks crawl demand: pages it considers valuable enough to revisit regularly. Together, these define your effective crawl budget.

For a 500-post WordPress blog, a conservative audit typically surfaces somewhere between 1,500 and 4,000 additional URLs generated by WordPress's default behavior — before you've added a single plugin or uploaded a single image. That's the problem.

Where the Bloat Comes From

WordPress auto-generates URL patterns most site owners never intentionally create:

A site with 300 posts, 50 tags, and 600 uploaded images can easily have more attachment and archive URLs than content URLs. Googlebot crawls what it finds — links in your HTML, sitemap entries, references in RSS — and none of these patterns are hidden by default.

Tag Pages: High Count, Low Value

Tag archives make taxonomic sense in theory. In practice, most WordPress installs accumulate tags inconsistently. A post about "Google Analytics 4" might be tagged analytics, GA4, google analytics, and tracking — each generating its own paginated archive with a single post in it.

Single-entry tag archives are near-zero value for users and represent duplicate thin content for crawlers. Googlebot doesn't know your analytics archive is curated and your ga4 archive is an accident.

The Fix for Tags

Option 1 — Consolidate then noindex stragglers. Merge tags with fewer than 3–5 posts into broader ones using a redirect. Then noindex surviving thin archives.

Option 2 — Noindex all tag archives. If tags provide no navigational value to users (many content-heavy blogs fall into this category), add to your functions.php or use Yoast/RankMath to set:

Robots: noindex, follow

Note follow — you want Googlebot to still crawl links on tag pages to discover posts, just not index the archive URL itself. Noindexing does remove URLs from the index eventually, but Googlebot still crawls them until signals stabilize. For immediate crawl savings, combine noindex with removal from your XML sitemap.

Author Archives: The Single-Author Trap

Author archives were designed for multi-author publications. On a solo blog or a company site where all posts appear under one account, /author/admin/ is a near-duplicate of your homepage — same posts, different wrapper.

The risks are compounded when:

• The author slug is admin (exposes your login username to brute-force attempts)
• You have multiple editor-level WordPress accounts that have never published a post (each still gets an archive URL at /author/their-slug/)
• You've changed display names but the archive slug still reflects an old identity

The Fix for Author Archives

For single-author sites, redirect the author archive to the homepage and noindex it. For multi-author publications, keep the archive but ensure each author has a genuine bio, a canonical URL set, and enough posts to justify a page.

To catch empty author archives — users who exist in WordPress but have zero published posts — run:

SELECT user_login, user_nicename
FROM wp_users u
LEFT JOIN wp_posts p ON p.post_author = u.ID AND p.post_status = 'publish'
WHERE p.ID IS NULL;

Any result from this query is an author archive URL that currently returns an empty page. Either delete the account or noindex the archive.

Attachment Pages: WordPress's Quiet Crawl Drain

This one surprises most site owners. When you upload an image in WordPress, the CMS creates a standalone page for that image at a URL like /2024/06/screenshot-homepage/. The page typically contains the image, the image title (often a filename like screenshot-2024-06-homepage-v2-final-FINAL), and nothing else.

Attachment pages have almost no SEO value. Google has been explicit about this: thin attachment pages are a quality signal problem. Yet on media-heavy sites — those with product photos, portfolio work, or regular screenshot-based tutorials — attachment pages can outnumber published posts by a factor of five or more.

The Fix for Attachment Pages

Since WordPress 6.4, new installs redirect attachment pages to the parent post by default. If your site predates this or you haven't updated your settings:

1. In Yoast SEO: SEO → Search Appearance → Media → set "Redirect attachment URLs to the attachment itself" to Yes.
2. In RankMath: Titles & Meta → Media Attachment → "Redirect Attachment Pages."
3. Manual: Add a redirect rule in your theme's functions.php or via a dedicated redirect plugin that sends all attachment post type URLs to the parent post permalink.

Verify the redirect is working with:

curl -I https://yoursite.com/2024/06/your-image-filename/

You want a 301 response with a Location header pointing to the parent post. A 200 means the attachment page is still live.

Date Archives and Feed Multiplication

Date archives (/2024/, /2024/06/) are similar to author archives — occasionally useful on high-velocity news sites, almost never useful on typical company or personal blogs.

More problematic is how feeds multiply your archive count. Every taxonomy archive in WordPress generates a corresponding RSS feed: /tag/seo/feed/, /author/admin/feed/, /category/news/feed/. These are crawlable, indexable (though Google rarely indexes them), and they consume crawl budget. Unless you're running a content business where subscribers follow per-tag feeds, disallow these in robots.txt:

Disallow: /tag/*/feed/
Disallow: /author/*/feed/
Disallow: /category/*/feed/

Keep your main /feed/ accessible — it's legitimately useful for RSS subscribers and news aggregators.

Validating Your Cleanup

After making changes, don't wait months for organic signal improvement. Use these checkpoints:

Google Search Console → Settings → Crawl Stats. Look at the "By response" breakdown. A successful cleanup shifts traffic from 200 OK toward 301 Moved (for redirected attachments) and reduces total daily crawled URLs within a few weeks.

Log file analysis. If you have server log access, filter for Googlebot hits on paths matching /tag/, /author/, and any year-based path (/2022/, /2023/). A pre/post comparison two weeks after deployment is more reliable than waiting for GSC to update.

XML sitemap audit. Run your sitemap through Screaming Frog or a similar crawler and filter for URLs that return noindex. Any noindexed URL still in your sitemap is a configuration error — remove it. Your sitemap should contain only canonically indexable URLs.

FAQ

Does noindexing tag and author pages hurt my site's authority?

Not meaningfully. These pages rarely attract external links, and the posts they link to are already indexed independently. You're not destroying equity — you're stopping Googlebot from wasting time on URLs that were never ranking candidates.

How long does it take for WordPress crawl budget improvements to show up in GSC?

Crawl stats in GSC typically reflect the prior 90 days in aggregate, but the rolling 7-day view updates more quickly. Most sites see crawl count reductions within 2–4 weeks of deploying noindex and redirects at scale.

Should I also block these paths in robots.txt instead of using noindex?

Using Disallow in robots.txt blocks crawling but doesn't remove URLs from the index if they're already indexed or if other sites link to them. For URL types you want completely excluded from Google's index, noindex is more reliable than robots.txt for pages already known to Google. Use robots.txt disallow for feed paths and pagination you're certain have no indexed footprint.

The practical upshot: audit your WordPress URL surface with a crawl tool, identify the ratio of archive and attachment URLs to actual content URLs, and bring that ratio down to something defensible. On most mid-sized WordPress sites, getting this right recovers hundreds to thousands of crawl requests per week that were previously wasted on auto-generated noise.

If you use FluxWriter to plan and draft your technical content, the SEO metadata fields prompt you to verify canonical URLs and indexability settings before publishing — a useful forcing function when working through a cleanup like this one.

← All posts