Python Sitemap Generator Project

Creating a reliable sitemap is a cornerstone of effective SEO, and Python makes the process both flexible and scalable. In this guide we’ll walk through a complete Python sitemap generator project—from setting up the environment to deploying a ready‑to‑use script that crawls your site, respects robots.txt, and outputs a standards‑compliant XML file. Whether you’re a solo developer, a small business owner, or part of a larger development team, this tutorial equips you with the code, best practices, and optimization tips you need to boost your site’s visibility on search engines.

Why a Custom Python Sitemap Generator?

While many CMS platforms offer built‑in sitemap tools, a custom solution gives you:

• Full control over which URLs are included, their priority, and change frequency.
• The ability to integrate with existing pipelines (CI/CD, static site generators, etc.).
• Scalability for large sites with thousands of pages.
• Compliance with the Sitemap XML protocol and easy updates when the spec changes.

Project Overview

Our Python sitemap generator will consist of three main components:

1. Crawler: Recursively discovers internal links while respecting robots.txt.
2. URL Processor: Normalizes URLs, removes duplicates, and assigns priority and changefreq values.
3. XML Builder: Generates a valid sitemap.xml file ready for submission to Google Search Console, Bing Webmaster Tools, and other search engines.

Setting Up the Development Environment

Required Packages

We’ll rely on a few well‑maintained libraries to keep the code concise and robust:

pip install requests beautifulsoup4 lxml tqdm

Explanation of each package:

• requests – Handles HTTP requests with automatic redirects and timeout handling.
• beautifulsoup4 – Parses HTML and extracts <a> tags efficiently.
• lxml – Provides fast XML generation and validation.
• tqdm – Adds a progress bar, making long crawls more user‑friendly.

Project Structure

python-sitemap-generator/
│
├─ main.py            # Entry point
├─ crawler.py         # Crawl logic
├─ processor.py       # URL normalization & priority logic
├─ sitemap_builder.py # XML creation
└─ requirements.txt   # Pin versions for reproducibility

Building the Crawler

Respecting robots.txt

Before we start fetching pages, it’s essential to obey the site’s robots.txt. The following helper function reads the file and determines if a URL is allowed:

import urllib.robotparser as robotparser

def is_allowed(base_url, path, user_agent='*'):
    rp = robotparser.RobotFileParser()
    rp.set_url(f"{base_url.rstrip('/')}/robots.txt")
    rp.read()
    return rp.can_fetch(user_agent, path)

Recursive Link Extraction

The core crawling routine uses a queue to manage discovered URLs and a set to avoid repeats. It also limits depth to prevent infinite loops on large sites.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from tqdm import tqdm

def crawl(start_url, max_depth=3):
    visited = set()
    queue = [(start_url, 0)]
    base = f"{urlparse(start_url).scheme}://{urlparse(start_url).netloc}"

    while queue:
        current_url, depth = queue.pop(0)
        if depth > max_depth or current_url in visited:
            continue

        # Respect robots.txt
        if not is_allowed(base, urlparse(current_url).path):
            continue

        try:
            response = requests.get(current_url, timeout=10)
            response.raise_for_status()
        except requests.RequestException:
            continue

        visited.add(current_url)
        soup = BeautifulSoup(response.text, 'html.parser')
        for link in soup.find_all('a', href=True):
            href = link['href']
            # Resolve relative URLs
            absolute = urljoin(base, href)
            # Keep only internal links
            if urlparse(absolute).netloc == urlparse(base).netloc:
                queue.append((absolute, depth + 1))

    return visited

Processing URLs for the Sitemap

Normalizing and De‑duplicating

After crawling, we must ensure every URL is in a canonical form—trailing slashes, URL‑encoded characters, and duplicate query strings can cause search engines to treat the same page as multiple entries.

from urllib.parse import urlunparse, urlparse, parse_qsl, urlencode

def normalize(url):
    parsed = urlparse(url)
    # Remove fragment, sort query parameters
    query = urlencode(sorted(parse_qsl(parsed.query)), doseq=True)
    path = parsed.path.rstrip('/') or '/'
    return urlunparse((parsed.scheme, parsed.netloc, path, '', query, ''))

Assigning Priority and Change Frequency

Google recommends setting priority values between 0.0 and 1.0 to indicate the relative importance of pages. A simple heuristic is to base priority on URL depth.

def assign_meta(url, base_url):
    depth = urlparse(url).path.count('/')
    # Shallow pages get higher priority
    priority = max(0.1, round(1.0 - (depth * 0.1), 1))
    # Assume static pages change rarely
    changefreq = 'weekly' if depth > 2 else 'daily'
    return priority, changefreq

Generating the XML Sitemap

Using lxml for Standards‑Compliant Output

The lxml library lets us build XML with proper namespaces and pretty‑print the result.

from lxml import etree
from datetime import datetime

def build_sitemap(urls, base_url):
    NSMAP = {None: "http://www.sitemaps.org/schemas/sitemap/0.9"}
    urlset = etree.Element('urlset', nsmap=NSMAP)

    for raw_url in urls:
        url = normalize(raw_url)
        priority, changefreq = assign_meta(url, base_url)

        url_el = etree.SubElement(urlset, 'url')
        etree.SubElement(url_el, 'loc').text = url
        etree.SubElement(url_el, 'lastmod').text = datetime.utcnow().date().isoformat()
        etree.SubElement(url_el, 'changefreq').text = changefreq
        etree.SubElement(url_el, 'priority').text = str(priority)

    return etree.tostring(urlset, pretty_print=True, xml_declaration=True, encoding='UTF-8')

Saving the File

Finally, write the generated XML to sitemap.xml at the root of your website.

def save_sitemap(xml_bytes, output_path='sitemap.xml'):
    with open(output_path, 'wb') as f:
        f.write(xml_bytes)

Putting It All Together – main.py

from crawler import crawl
from sitemap_builder import build_sitemap, save_sitemap

if __name__ == '__main__':
    START_URL = 'https://example.com'   # Replace with your domain
    print('Starting crawl…')
    discovered = crawl(START_URL, max_depth=4)
    print(f'Found {len(discovered)} unique URLs.')

    print('Building sitemap XML…')
    sitemap_xml = build_sitemap(discovered, START_URL)

    print('Saving sitemap.xml')
    save_sitemap(sitemap_xml)
    print('Sitemap generation complete!')

SEO Best Practices for Your Generated Sitemap

• Limit to 50,000 URLs per file (or 50 MB uncompressed). Split larger sites into multiple sitemap files and reference them in a sitemap_index.xml.
• Include only canonical URLs. Duplicate content can dilute ranking signals.
• Update lastmod whenever a page changes; many CMS platforms can trigger a rebuild automatically.
• Submit the sitemap through Google Search Console and Bing Webmaster Tools to accelerate indexing.
• Monitor crawl errors after submission; fix 404s or redirects that appear in the console.

Advanced Enhancements

Parallel Crawling

For very large sites, consider using concurrent.futures.ThreadPoolExecutor or an async framework like aiohttp to fetch pages concurrently, dramatically reducing total crawl time.

Dynamic Content Detection

If your site heavily relies on JavaScript, integrate a headless browser (e.g., Playwright or Selenium) to render pages before extracting links. This ensures URLs generated client‑side are not missed.

Integration with CI/CD

Automate sitemap generation as part of your deployment pipeline:

• Run the script after a successful build.
• Upload the resulting sitemap.xml to your static‑site CDN (Netlify, Vercel, Cloudflare Pages).
• Trigger a ping to search engines using https://www.google.com/ping?sitemap=URL to notify them of the update.

Testing and Validation

Before submitting, validate the XML against the official schema:

import subprocess

def validate_sitemap(path):

    schema_url = 'http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd'

    result = subprocess.run(['xmllint