Technical SEO

Crawl Waste Optimizer

A technical report that turns redirects, sitemap leaks, non-200s, robots blocks, noindex waste, duplicate canonicals, and skipped crawl jobs into a prioritized cleanup queue.

Best for: Technical SEOs, migration teams, developers, and SEO operations leads.

Try this report API docs All examples

Crawl Waste Optimizer Technical SEO

34Waste Score

12Wasted URLs

3Redirects

4Sitemap Leaks

Item	Status	Evidence
/old-service	Redirect	Redirecting URL remains in sitemap and watchlist.
/noindex-page	Leak	Noindex URL included in XML sitemap.
/blog/root-canal-guide	Canonical	Duplicate canonical cluster should be consolidated.

What It Includes

Crawl waste score with wasted URL count and severity split
Redirect chains, sitemap leak candidates, canonical clusters, and crawl errors
Fix recommendations grouped by blocker type and business priority
Developer-ready evidence for migration cleanup or sprint tickets

How It Is Generated

HTML crawl results and crawl queue status
Sitemap inventory and sitemap membership
Robots, noindex, canonical, redirect, and HTTP status evidence
Indexation snapshots and watchlist priority

API Inputs

/v1/crawl/waste/v1/crawl/errors/v1/crawl/soft-404/v1/redirects/chains/v1/sitemaps/leaks/v1/canonicals/clusters

MCP Uses

Ask for the highest-impact crawl waste fixes before a migration.
Generate a developer ticket from sitemap leak evidence.
Explain why a URL is wasting crawl attention.

Ways People Use It

Migration QA Technical SEO lead

Run the report after launch to catch redirects, sitemap leaks, and non-canonical URLs still being submitted.

Cleanup work becomes a prioritized issue list instead of a raw crawl export.

Developer Ticket Engineering manager

Turn a redirect chain or noindex sitemap leak into a ticket with URL, evidence, severity, and recommended fix.

Developers receive the exact URL and why it matters.

Portal Widget SEO platform builder

Embed crawl waste score and top issue types in a client-facing dashboard.

Customers can monitor cleanup progress without reading crawler spreadsheets.

Evidence Contract

The live API returns the issue plus the evidence around it: what happened, where it happened, when it was observed, why it matters, and how to act.

Question	Where It Comes From	Example
What	Issue ID, severity, message, schema, crawl, or GSC state	`jsonld_invalid`, `schema_missing`, `noindex_detected`
Where	URL, site, client, final URL, JSON-LD block index, or endpoint row	`/pricing`, block `0`
When	`generated_at`, `fetched_at`, event timestamps, and history metrics	Last observed during the current refresh.
Why	Evidence fields, parse errors, HTTP status, GSC verdict, and crawl facts	JSON parse error, noindex header, sitemap leak, or canonical conflict.
How	Recommended actions and MCP or report prompts	Fix malformed JSON-LD, remove noindex, update sitemap, or consolidate canonical.

For malformed JSON-LD, the API records the affected page, fetch time, invalid block index, parser error, issue severity, and recommended action. It intentionally avoids storing raw page payloads unless a future redacted evidence store is enabled.

Example API Flow

curl.exe -H "Authorization: Bearer YOUR_API_TOKEN" https://indexpulseapi.com/v1/crawl/waste

MCP prompt example:
"Ask for the highest-impact crawl waste fixes before a migration."