Python Web Broken Link Checker Utility

Keeping a website free of dead URLs is essential for user experience, SEO rankings, and brand credibility. Yet, as pages evolve and external resources change, broken links silently creep in, leading to 404 errors and lost traffic. In this guide we’ll walk you through building a robust Python web broken link checker utility that can scan single pages or entire sites, report problematic URLs, and integrate seamlessly into your development workflow. Whether you’re a solo developer, a DevOps engineer, or part of an SEO team, this utility will empower you to maintain a healthy link ecosystem with minimal effort.

Why a Broken Link Checker Matters for SEO

Search engines like Google treat broken links as a signal of poor site maintenance. A high 404 rate can:

• Reduce crawl efficiency, causing important pages to be skipped.
• Lower domain authority because external sites may remove backlinks.
• Increase bounce rates as users encounter dead ends.

Automating the detection process with a Python script not only saves time but also provides actionable data that can be fed into continuous integration pipelines, ensuring that every deployment meets a baseline quality standard.

Choosing the Right Python Libraries

Requests vs. HTTPX vs. aiohttp

For synchronous checks, requests offers a simple API and excellent error handling. For high‑performance, concurrent crawling, aiohttp combined with asyncio can fetch hundreds of URLs in parallel. HTTPX bridges the gap, supporting both sync and async modes with a familiar interface.

Parsing HTML with BeautifulSoup

Extracting href and src attributes from <a>, <img>, <script>, and <link> tags is straightforward using BeautifulSoup. Its robust parser handles malformed markup, which is common on the web.

Designing the Core Checker Logic

Step‑by‑step Workflow

1. Fetch the target page – Use a reliable HTTP client with timeout and retry settings.
2. Parse the HTML – Identify all candidate URLs (internal and external).
3. Normalize URLs – Resolve relative paths against the base URL.
4. Validate each link – Issue a HEAD request (fallback to GET if HEAD is unsupported).
5. Collect results – Store status code, response time, and any redirect chain.
6. Report – Output to console, CSV, JSON, or an HTML summary.

Sample Synchronous Implementation

import requests
from urllib.parse import urljoin, urlparse
from bs4 import BeautifulSoup

def get_links(page_url):
    resp = requests.get(page_url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, 'html.parser')
    tags = soup.find_all(['a', 'img', 'script', 'link'])
    links = set()
    for tag in tags:
        attr = 'href' if tag.name != 'img' else 'src'
        url = tag.get(attr)
        if url:
            full_url = urljoin(page_url, url)
            links.add(full_url)
    return links

def check_link(url):
    try:
        head = requests.head(url, allow_redirects=True, timeout=5)
        return head.status_code
    except requests.RequestException:
        # Fallback to GET for sites that block HEAD
        try:
            get = requests.get(url, allow_redirects=True, timeout=5)
            return get.status_code
        except requests.RequestException:
            return None

def main(start_url):
    broken = []
    for link in get_links(start_url):
        code = check_link(link)
        if not code or code >= 400:
            broken.append((link, code))
    for url, status in broken:
        print(f'Broken: {url} → {status}')

if __name__ == '__main__':
    main('https://example.com')

This script provides a quick way to scan a single page. However, for large sites you’ll want an asynchronous version to reduce total runtime.

Scaling Up with Asynchronous Crawling

Async Architecture Overview

Using aiohttp, we can launch thousands of concurrent requests while respecting a configurable semaphore to avoid overwhelming the target server. The following pattern is widely adopted:

• Queue URLs in an asyncio.Queue.
• Spawn a pool of worker coroutines that dequeue, fetch, and validate links.
• Persist results to a thread‑safe collection.

Async Checker Example

import asyncio
import aiohttp
from bs4 import BeautifulSoup
from urllib.parse import urljoin

CONCURRENCY = 20
TIMEOUT = aiohttp.ClientTimeout(total=10)

async def fetch(session, url):
    async with session.get(url, timeout=TIMEOUT) as resp:
        return await resp.text()

async def extract_links(session, base_url):
    html = await fetch(session, base_url)
    soup = BeautifulSoup(html, 'html.parser')
    links = set()
    for tag in soup.find_all(['a', 'img', 'script', 'link']):
        attr = 'href' if tag.name != 'img' else 'src'
        raw = tag.get(attr)
        if raw:
            links.add(urljoin(base_url, raw))
    return links

async def validate_link(session, url):
    try:
        async with session.head(url, allow_redirects=True) as resp:
            return resp.status
    except aiohttp.ClientError:
        # fallback to GET if HEAD fails
        try:
            async with session.get(url, allow_redirects=True) as resp:
                return resp.status
        except aiohttp.ClientError:
            return None

async def worker(name, queue, session, results):
    while not queue.empty():
        url = await queue.get()
        status = await validate_link(session, url)
        if not status or status >= 400:
            results.append((url, status))
        queue.task_done()

async def main(start_url):
    queue = asyncio.Queue()
    async with aiohttp.ClientSession() as session:
        links = await extract_links(session, start_url)
        for link in links:
            await queue.put(link)

        results = []
        tasks = [
            asyncio.create_task(worker(f'worker-{i}', queue, session, results))
            for i in range(CONCURRENCY)
        ]
        await queue.join()
        for t in tasks:
            t.cancel()
        for url, code in results:
            print(f'Broken: {url} → {code}')

if __name__ == '__main__':
    asyncio.run(main('https://example.com'))

The async version can process a medium‑sized website (hundreds of pages) in under a minute, making it ideal for CI pipelines or scheduled health checks.

Output Formats and Reporting

Different stakeholders prefer different data representations. Below are three common formats you can generate with a few extra lines of code:

• Console table – Quick visual feedback during development.
• CSV file – Easy import into Excel or Google Sheets for bulk analysis.
• HTML report – Shareable dashboard with clickable links and status colors.

Generating a CSV Report

import csv

def write_csv(results, filename='broken_links.csv'):
    with open(filename, 'w', newline='', encoding='utf-8') as f:
        writer = csv.writer(f)
        writer.writerow(['URL', 'Status Code'])
        for url, code in results:
            writer.writerow([url, code or 'Error'])

Creating a Simple HTML Summary

def write_html(results, filename='report.html'):
    rows = ''
    for url, code in results:
        color = 'red' if not code or code >= 400 else 'green'
        rows += f'{url}{code or "Error"}'
    html = f'''
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Broken Link Report</title>
        <style>
            table {{border-collapse: collapse; width: 100%;}}
            th, td {{border: 1px solid #ddd; padding: 8px;}}
            th {{background-color: #f2f2f2;}}
        </style>
    </head>
    <body>
        <h2>Broken Link Report</h2>
        <table>
            <tr><th>URL</th><th>Status</th></tr>
            {rows}
        </table>
    </body>
    </html>
    '''
    with open(filename, 'w', encoding='utf-8') as f:
        f.write(html)

These helper functions can be called after the crawling phase, allowing you to tailor the output to the audience—developers get raw CSV, marketers receive a polished HTML page.

Best Practices for Reliable Link Checking

• Respect robots.txt – Before crawling, parse /robots.txt and obey User-agent: * rules to avoid legal issues.
• Throttle requests – Use a delay