Sitemap API: Fast Website Structure Discovery

Introduction

Search1API's Sitemap endpoint provides developers with a powerful tool to discover and extract all related links from any given URL. This API is essential for site mapping, content discovery, and building comprehensive web crawlers.

Authentication

All Search1API endpoints require authentication using Bearer token. Include your API key in the Authorization header:

Authorization: Bearer your_api_key_here

Basic Usage

Making a Sitemap Request

The sitemap endpoint is straightforward - just provide a URL:

POST https://api.search1api.com/sitemap
 
{
    "url": "https://example.com",
    "type": "all"
}

The API will respond with a list of all discovered links:

{
    "links": [
        "https://example.com/about",
        "https://example.com/products",
        "https://example.com/blog",
        "https://example.com/contact",
        // Additional links...
    ]
}

Discovery Modes

The Sitemap API offers two flexible modes to match your specific needs:

• Sitemap mode (default): Processes only links defined in the website's sitemap.xm l
• All mode: Discovers and extracts all findable links throughout the page, including navigation, footer links, and content references. Select your mode by setting the "type" parameter:

Key Features

1. Comprehensive Link Discovery
   • Finds all related links on a webpage
   • Includes internal and subpath links
   • Discovers hidden or dynamically loaded links
2. Smart Processing
   • Automatically handles relative URLs
   • Resolves redirects
   • Filters out invalid or malformed links
3. Flexible Discovery Strategies
   • Sitemap mode for official, published content paths
   • All mode for complete site structure analysis
   • Tailor discovery to match your use case

Use Cases

1. Website Mapping
   • Create site structure visualizations
   • Build content inventories
   • Track website changes
2. Content Discovery
   • Find all available content
   • Discover new sections and pages
   • Map content relationships
3. SEO Tools
   • Audit website structure
   • Find broken links
   • Analyze site hierarchy
4. Integration Scenarios
   • Combine with Crawl API for full content extraction
   • Use with Search API for targeted content discovery
   • Build automated content indexing systems

Best Practices

• Choosing the Right Discovery Mode
  • When to use Sitemap Mode:
    • For official, published content only
    • When focusing on primary content paths
    • For faster, more efficient discovery
    • When targeting specific sections defined in the sitemap
  • When to use All Mode:
  • For complete site structure analysis
  • When sitemap.xml is missing or incomplete
  • For finding hidden or unlisted content
  • When comprehensive coverage is critical
• Performance Optimization
  • Cache results when appropriate
  • Implement rate limiting for large sites
  • Use incremental discovery for huge websites
• Error Handling
  • Handle timeouts gracefully
  • Implement retry logic
  • Validate discovered URLs

Common Integration Patterns

1. Full Site Mapping

import requests
 
headers = {
    'Authorization': 'Bearer your_api_key_here',
    'Content-Type': 'application/json'
}
 
def get_site_map(url, discovery_type="all"):
    data = {
        'url': url,
        'type': discovery_type
    }
    response = requests.post(
        'https://api.search1api.com/sitemap',
        headers=headers,
        json=data
    )
    return response.json()['links']

1. Combined with Crawl API

# First get all links
sitemap_result = get_site_map('https://example.com')
 
# Then crawl each important page
for link in sitemap_result:
    if is_important_page(link):
        crawl_data = {
            'url': link
        }
        crawl_response = requests.post(
            'https://api.search1api.com/crawl',
            headers=headers,
            json=crawl_data
        )

Why Choose Our Sitemap API?

• Flexible: Two discovery modes for different needs
• Fast: Quick discovery of all related links
• Comprehensive: Finds both visible and hidden links
• Reliable: Handles complex website structures
• Economic: Starting From Free
• Simple: Easy to integrate and use

Common Use Patterns

1. Content Discovery
   • Map entire websites quickly
   • Find new content automatically
   • Track site updates
2. SEO Analysis
   • Analyze site structure
   • Check internal linking
   • Monitor site changes
3. Content Aggregation
   • Discover all available content
   • Build content databases
   • Create site archives

Get Started

Visit our API documentation to start using Search1API's Sitemap endpoint today. Transform your website discovery and mapping capabilities with our powerful API!