What is DeepCrawl? Full-Site Crawling for RAG and Knowledge Bases

What is DeepCrawl?

DeepCrawl is an asynchronous full-site crawling API that discovers all pages on a website, fetches their content, and returns it as a structured dataset. Unlike a single-page crawl API that processes one URL at a time, DeepCrawl follows links across an entire site, collects clean content from every page, and delivers it in a format ready for RAG pipelines, search indexes, or knowledge bases.

What is deep crawling?

Deep crawling is the process of recursively following links to map and extract content from an entire website, not just the entry page. A deep crawler starts at a root URL, discovers links on that page, follows them to discover more pages, and continues until it has mapped the whole site. At each page, it extracts the main content, title, metadata, and links.

This is different from:

• Single-page crawl: fetches one URL and returns its content
• Sitemap parsing: reads sitemap.xml but does not fetch page content
• Search indexing: discovers pages via search engines, not by crawling

How DeepCrawl works

1. Start a task: Submit a root URL and choose a discovery mode (sitemap or all-links)
2. Background processing: DeepCrawl discovers pages, fetches content, and tracks progress
3. Poll status: Check the task status endpoint to see how many pages have been crawled
4. Collect results: When the task completes, retrieve clean content from every page

Sitemap mode vs. all-links mode

DeepCrawl offers two discovery modes:

Sitemap mode: Reads the site's sitemap.xml to find pages. Best for well-structured sites with complete sitemaps. Faster and more predictable.

All-links mode: Follows links from the root page to discover pages recursively. Best for sites with missing or incomplete sitemaps. Slower but more thorough.

When to use DeepCrawl

DeepCrawl is ideal when you need content from an entire website, not just individual pages:

• RAG knowledge bases: Crawl a documentation site, help center, or blog and feed the content into your RAG pipeline so your AI assistant can answer questions about the whole site.
• Search index refresh: Crawl a site and update your internal search index with fresh content.
• Content migration: Export all content from a site in a structured format for migration or backup.
• Competitive analysis: Crawl a competitor's site to analyze their content structure, topics, and metadata.
• Compliance archiving: Capture a complete snapshot of a site for regulatory or archival purposes.

DeepCrawl vs. Crawl API: which to choose?

• Scope
  • Crawl API: One URL or small batch
  • DeepCrawl: Entire website
• Execution
  • Crawl API: Synchronous (immediate response)
  • DeepCrawl: Asynchronous (background task)
• Use case
  • Crawl API: Read a specific page
  • DeepCrawl: Build a full-site corpus
• Credits
  • Crawl API: 1 per request
  • DeepCrawl: 20 per task
• Output
  • Crawl API: Clean content per URL
  • DeepCrawl: Clean content for all discovered pages
• Best for
  • Crawl API: Agent workflows, RAG ingestion
  • DeepCrawl: Knowledge base building, site migration

Use Crawl API when you need content from a specific page right now. Use DeepCrawl when you need the whole site and can wait for a background task to complete.

Getting started with DeepCrawl

1. Sign up at Search1API — get 100 free credits
2. Start a DeepCrawl task with your target URL
3. Poll the status endpoint until the task completes
4. Retrieve clean content from all crawled pages
5. Feed the content into your RAG pipeline, search index, or storage

curl -X POST https://api.search1api.com/deepcrawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "mode": "sitemap"
  }'

FAQ

How long does a DeepCrawl task take?

It depends on the site size. A small documentation site (50-100 pages) typically completes in under a minute. Larger sites (500+ pages) may take several minutes.

How many pages can DeepCrawl handle?

DeepCrawl is designed for documentation sites, help centers, and blogs (typically 50-500 pages). For very large sites (10,000+ pages), contact us about enterprise options.

What does DeepCrawl cost?

Starting a DeepCrawl task costs 20 credits. You get 100 free credits on signup, so you can start 5 DeepCrawl tasks for free. After that, credits start at $8.90 for 1,000 credits.

Does DeepCrawl respect robots.txt?

Yes. DeepCrawl is robots.txt-aware and will not crawl pages disallowed by the site's robots.txt.