Author: 69434e

  • Python Document Scanner App Opencv

    Scanning documents with a smartphone or webcam has become a daily routine for students, freelancers, and small businesses. While commercial apps offer polished interfaces, building your own Python document scanner app with OpenCV gives you full control over privacy, customization, and learning opportunities. In this guide we’ll walk through every step—from setting up the development environment to writing clean, production‑ready code—so you can create a fast, reliable scanner that automatically detects page edges, corrects perspective, and produces a crisp black‑and‑white PDF.

    Why Choose OpenCV for a Document Scanner?

    OpenCV (Open Source Computer Vision Library) is the de‑facto standard for image processing in Python. Its strengths for a document scanner app include:

    • Real‑time performance: Optimized C++ core with Python bindings makes processing frames from a webcam or mobile camera lightning fast.
    • Rich set of algorithms: Edge detection, contour finding, perspective transforms, and adaptive thresholding are all built‑in.
    • Cross‑platform support: Write once, run on Windows, macOS, Linux, or even Android via Kivy or PySide.
    • Active community: Thousands of tutorials, forums, and GitHub projects can help you troubleshoot quickly.

    Prerequisites and Installation

    Before diving into code, make sure you have the following installed on your development machine:

    1. Python 3.8+ (preferably from python.org)
    2. OpenCV‑Python package
    3. NumPy for array manipulation
    4. PyMuPDF or ReportLab for PDF generation (optional but recommended)

    Run the commands below in your terminal to set up a virtual environment and install the required libraries:

    python -m venv scanner-env
    source scanner-env/bin/activate   # On Windows use `scanner-env\Scripts\activate`
    pip install opencv-python numpy pymupdf
    

    Understanding the Core Workflow

    A robust document scanner follows a predictable pipeline. Breaking it down helps you debug each stage and swap out components later.

    1. Capture the Image

    Whether you read from a static file or a live camera feed, the first step is to obtain a high‑resolution color image. OpenCV’s cv2.VideoCapture class makes webcam integration trivial.

    2. Detect the Document Contour

    The goal is to locate the four corners of the page. The typical approach combines:

    • Grayscale conversion
    • Gaussian blur to reduce noise
    • Canny edge detection
    • Contour extraction and approximation

    We then select the largest quadrilateral contour assuming it’s the document.

    3. Apply Perspective Transform

    Once we have the corner points, cv2.getPerspectiveTransform and cv2.warpPerspective map the skewed image to a flat, top‑down view. This step is crucial for readability.

    4. Enhance Readability with Thresholding

    Document images look best in black‑and‑white. Adaptive thresholding (or Otsu’s method) converts the warped color image into a clean binary image that mimics a scanned page.

    5. Export to PDF (Optional)

    After processing, you can save the result as a high‑resolution JPEG/PNG or embed it in a PDF using PyMuPDF. PDFs are ideal for sharing and archiving.

    Full Code Walkthrough

    Below is a complete, well‑commented Python script that implements the workflow described above. Feel free to copy, modify, and integrate it into a larger GUI or mobile app.

    import cv2
    import numpy as np
    import fitz  # PyMuPDF for PDF creation
    
    def order_points(pts):
        # Ensure points are in order: top‑left, top‑right, bottom‑right, bottom‑left
        rect = np.zeros((4, 2), dtype="float32")
        s = pts.sum(axis=1)
        rect[0] = pts[np.argmin(s)]      # top‑left
        rect[2] = pts[np.argmax(s)]      # bottom‑right
        diff = np.diff(pts, axis=1)
        rect[1] = pts[np.argmin(diff)]   # top‑right
        rect[3] = pts[np.argmax(diff)]   # bottom‑left
        return rect
    
    def four_point_transform(image, pts):
        rect = order_points(pts)
        (tl, tr, br, bl) = rect
    
        # Compute width of new image
        widthA = np.linalg.norm(br - bl)
        widthB = np.linalg.norm(tr - tl)
        maxWidth = max(int(widthA), int(widthB))
    
        # Compute height of new image
        heightA = np.linalg.norm(tr - br)
        heightB = np.linalg.norm(tl - bl)
        maxHeight = max(int(heightA), int(heightB))
    
        # Destination points for "birds‑eye view"
        dst = np.array([
            [0, 0],
            [maxWidth - 1, 0],
            [maxWidth - 1, maxHeight - 1],
            [0, maxHeight - 1]], dtype="float32")
    
        # Perspective transform matrix and warped image
        M = cv2.getPerspectiveTransform(rect, dst)
        warped = cv2.warpPerspective(image, M, (maxWidth, maxHeight))
        return warped
    
    def scan_document(image_path):
        # 1️⃣ Load image and resize for faster processing
        image = cv2.imread(image_path)
        ratio = image.shape[0] / 500.0
        orig = image.copy()
        image = cv2.resize(image, (int(image.shape[1] / ratio), 500))
    
        # 2️⃣ Convert to grayscale, blur, and detect edges
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        gray = cv2.GaussianBlur(gray, (5, 5), 0)
        edged = cv2.Canny(gray, 75, 200)
    
        # 3️⃣ Find contours and keep the largest quadrilateral
        cnts, _ = cv2.findContours(edged.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
        cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:5]
    
        screenCnt = None
        for c in cnts:
            peri = cv2.arcLength(c, True)
            approx = cv2.approxPolyDP(c, 0.02 * peri, True)
    
            if len(approx) == 4:
                screenCnt = approx
                break
    
        if screenCnt is None:
            raise ValueError("Could not find document contour.")
    
        # 4️⃣ Apply perspective transform on the original high‑resolution image
        warped = four_point_transform(orig, screenCnt.reshape(4, 2) * ratio)
    
        # 5️⃣ Convert to grayscale and apply adaptive threshold
        warped_gray = cv2.cvtColor(warped, cv2.COLOR_BGR2GRAY)
        scanned = cv2.adaptiveThreshold(
            warped_gray, 255,
            cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
            cv2.THRESH_BINARY,
            11, 2)
    
        return scanned
    
    def save_as_pdf(image, pdf_path):
        # Convert NumPy image to PDF page
        height, width = image.shape
        pdf_doc = fitz.open()
        pdf_page = pdf_doc.new_page(width=width, height=height)
        # PyMuPDF expects PNG bytes
        png_bytes = cv2.imencode('.png', image)[1].tobytes()
        pdf_page.insert_image(pdf_page.rect, stream=png_bytes)
        pdf_doc.save(pdf_path)
        pdf_doc.close()
    
    if __name__ == "__main__":
        import argparse, os
        parser = argparse.ArgumentParser(description="Simple Python document scanner using OpenCV")
        parser.add_argument("input", help="Path to input image (photo of a document)")
        parser.add_argument("-o", "--output", default="scanned.pdf", help="Output PDF filename")
        args = parser.parse_args()
    
        scanned_img = scan_document(args.input)
        # Save intermediate JPEG for debugging
        cv2.imwrite("scanned.jpg", scanned_img)
    
        save_as_pdf(scanned_img, args.output)
        print(f"✅ Scanned PDF saved to {os.path.abspath(args.output)}")
    

    Key takeaways from the script:

    • Resizing for speed: The image is downscaled to 500 px height for contour detection, then the original resolution is used for the final transform.
    • Robust corner ordering: The order_points helper guarantees a consistent point order, preventing upside‑down results.
    • Adaptive thresholding: Works well under varying lighting conditions, producing a clean black‑and‑white output.
    • PDF generation: Using PyMuPDF keeps the workflow pure Python without external binaries.

    Enhancing the Scanner: Features You Can Add

    Once the basic pipeline works, consider these extensions to turn a simple script

  • Python Diff Checker File Comparator

    When you need to spot differences between two text files, source code versions, or configuration scripts, a reliable Python diff checker can save you countless hours. In this guide we’ll explore why file comparison matters, walk through the most popular Python libraries for diffing, and build a simple yet powerful file comparator from scratch. Whether you’re a seasoned developer, a sysadmin, or a data analyst, you’ll walk away with a ready‑to‑use script and a deeper understanding of how Python tackles the diff problem.

    Why a File Comparator Is Essential in Modern Development

    In today’s fast‑paced development cycles, files change constantly. Here are the top scenarios where a file comparator becomes indispensable:

    • Code reviews: Quickly see what lines were added, removed, or modified between commits.
    • Configuration drift: Detect unintended changes in server or application config files.
    • Data validation: Compare exported CSV or JSON reports to ensure data integrity.
    • Backup verification: Confirm that backup copies match the original files.

    Choosing the Right Python Diff Library

    Python offers several built‑in and third‑party modules for computing differences. The table below highlights the most common options and their ideal use cases.

    Library Key Features Best For
    difflib SequenceMatcher, unified and context diff output, HTML diff generation Simple text diff, quick scripts
    filecmp Binary file comparison, shallow vs deep compare, directory tree comparison Fast equality checks, directory sync
    deepdiff Nested data structures, detailed diff reports, JSON/YAML support Complex data, API responses
    python‑difflib2 Extended diff formats, better performance on large files Large logs, big source files

    For most everyday tasks, the standard library’s difflib provides everything you need without adding external dependencies. Below we’ll build a diff checker using difflib and enhance it with optional features like HTML output and command‑line arguments.

    Building a Basic Python Diff Checker

    Step 1: Set Up the Project Structure

    my_diff_tool/
    ├── diff_checker.py
    └── README.md
    

    Step 2: Write the Core Comparison Logic

    The following script reads two files, computes a unified diff, and prints the result to the console. It also includes a --html flag to generate a styled HTML report.

    #!/usr/bin/env python3
    import argparse
    import difflib
    import sys
    from pathlib import Path
    
    def read_file(path: Path) -> list[str]:
        """Return the file contents as a list of lines, preserving line endings."""
        try:
            return path.read_text(encoding='utf-8').splitlines(keepends=True)
        except Exception as e:
            sys.exit(f"Error reading {path}: {e}")
    
    def generate_unified_diff(a: list[str], b: list[str], a_name: str, b_name: str) -> str:
        """Create a unified diff string similar to the `diff -u` command."""
        return ''.join(difflib.unified_diff(
            a, b,
            fromfile=a_name,
            tofile=b_name,
            lineterm=''
        ))
    
    def generate_html_diff(a: list[str], b: list[str], a_name: str, b_name: str) -> str:
        """Return an HTML diff using difflib.HtmlDiff."""
        html = difflib.HtmlDiff(wrapcolumn=80).make_file(
            a, b, fromdesc=a_name, todesc=b_name, context=True, numlines=3
        )
        return html
    
    def main():
        parser = argparse.ArgumentParser(
            description="Simple Python diff checker – compare two text files."
        )
        parser.add_argument('file_a', type=Path, help='First file to compare')
        parser.add_argument('file_b', type=Path, help='Second file to compare')
        parser.add_argument(
            '--html', action='store_true',
            help='Generate an HTML diff report instead of plain text'
        )
        parser.add_argument(
            '--output', type=Path,
            help='Write the diff to a file (default: stdout)'
        )
        args = parser.parse_args()
    
        a_lines = read_file(args.file_a)
        b_lines = read_file(args.file_b)
    
        if args.html:
            diff = generate_html_diff(a_lines, b_lines, str(args.file_a), str(args.file_b))
        else:
            diff = generate_unified_diff(a_lines, b_lines, str(args.file_a), str(args.file_b))
    
        if args.output:
            args.output.write_text(diff, encoding='utf-8')
            print(f"Diff written to {args.output}")
        else:
            print(diff)
    
    if __name__ == '__main__':
        main()
    

    Step 3: Run the Tool

    Open a terminal and execute:

    # Plain text diff
    python diff_checker.py old_version.py new_version.py
    
    # HTML diff saved to a file
    python diff_checker.py old_version.py new_version.py --html --output diff.html
    

    Enhancing the Diff Checker for Real‑World Use

    While the basic script works well for small files, production environments often demand extra features. Below are three upgrades you can add without rewriting the core logic.

    1. Recursive Directory Comparison

    • Use filecmp.dircmp to walk two directories and report added, removed, or changed files.
    • Combine it with the existing file diff logic to automatically generate diffs for every modified file.
    import filecmp
    from pathlib import Path
    
    def compare_directories(dir_a: Path, dir_b: Path):
        comparison = filecmp.dircmp(dir_a, dir_b)
        # Files only in dir_a (deleted)
        for f in comparison.left_only:
            print(f"Deleted: {dir_a / f}")
        # Files only in dir_b (added)
        for f in comparison.right_only:
            print(f"Added: {dir_b / f}")
        # Files present in both but differing
        for f in comparison.diff_files:
            print(f"Modified: {f}")
            # Call our diff function for each changed file
            a_path = dir_a / f
            b_path = dir_b / f
            a_lines = read_file(a_path)
            b_lines = read_file(b_path)
            print(generate_unified_diff(a_lines, b_lines, str(a_path), str(b_path)))
    

    2. Ignoring Whitespace and Comments

    Sometimes you only care about functional changes. You can filter out blank lines or comment lines before diffing:

    def filter_lines(lines: list[str], ignore_blank=True, ignore_comments=True) -> list[str]:
        filtered = []
        for line in lines:
            stripped = line.strip()
            if ignore_blank and not stripped:
                continue
            if ignore_comments and stripped.startswith('#'):
                continue
            filtered.append(line)
        return filtered
    

    3. Colorized Terminal Output

    Adding ANSI colors makes diffs easier to read. The termcolor or colorama packages can wrap added lines in green and removed lines in red.

    from termcolor import colored
    
    def colorize_unified(diff: str) -> str:
        colored_lines = []
        for line in diff.splitlines():
            if line.startswith('+') and not line.startswith('+++'):
                colored_lines.append(colored(line, 'green'))
            elif line.startswith('-') and not line.startswith('---'):
                colored_lines.append(colored(line, 'red'))
            else:
                colored_lines.append(line)
        return '\n'.join(colored_lines)
    

    Performance Tips for Large Files

    When comparing gigabyte‑scale logs or massive data dumps, keep these optimizations in mind:

    • Read lazily: Process files line‑by‑line using generators instead of loading the whole file into memory.
    • Chunked hashing: Compute a quick MD5 or SHA‑256 hash of each chunk; if hashes match, skip detailed diff for that chunk.
    • Parallel processing: Split the workload across CPU cores with concurrent.futures for multi‑file directory diffs.

    SEO Keywords Integrated Naturally

    Throughout this article we’ve woven in high‑traffic search terms such as Python diff checker, file comparator, compare files in Python, difflib example, and generate HTML diff report. By using these phrases in headings, bold text, and code comments, the post aligns with the queries developers type into search engines, boosting discoverability.

    Testing Your Diff Checker

    Before deploying the script, run a few automated tests to verify correctness. Below is a minimal pytest suite.

    import pytest
    from pathlib import Path
    from diff_checker import read_file, generate_unified_diff

    def test_read_file(tmp_path):
    f = tmp_path / "sample.txt"
    f.write_text("line1\\nline2\\n")
    assert read_file(f) == ["line1\\n", "line2\\n"]

    def test_unified_diff():
    a = ["a\\n", "b\\n", "c\\n"]
    b = ["a\\n", "x\\n", "c\\n"]
    diff = generate_unified_diff(a, b, "a.txt", "b.txt")
    assert "-b\\n" in diff
    assert "+x\\n" in diff

  • Python Markdown Previewer Desktop Application

    Looking for a fast, lightweight way to write and preview Markdown without leaving your desktop? A Python‑based Markdown previewer combines the simplicity of plain‑text editing with the power of live rendering, giving writers, developers, and educators a seamless workflow that works on Windows, macOS, and Linux. In this guide we’ll explore why a dedicated desktop previewer matters, walk through the core features you should expect, and even provide a step‑by‑step code example that you can customize and ship as a standalone app.

    Why a Desktop Markdown Previewer Is Still Relevant in 2026

    Even with browser‑based editors and cloud services, many users prefer a local application for privacy, speed, and offline access. A desktop previewer offers:

    • Zero latency: Render Markdown instantly as you type, without network delays.
    • Full control over extensions: Add custom syntax, math support, or task‑list rendering that online editors may block.
    • Secure workflow: Your notes never leave your machine, which is crucial for confidential documentation.
    • Cross‑platform consistency: The same UI and rendering engine works on every OS you support.

    Core Features Every Python Markdown Previewer Should Have

    Live Rendering

    The heart of the app is a split‑view layout where the left pane is a plain‑text editor and the right pane shows the rendered HTML. Real‑time updates keep the experience fluid.

    Syntax Highlighting and Theme Support

    Using libraries like Pygments or QtWebEngine, you can highlight code blocks and let users switch between light and dark themes.

    Export Options

    Most users need to share their work. Provide one‑click export to HTML, PDF (via WeasyPrint or pdfkit), and even plain‑text or GitHub‑flavored Markdown.

    Custom Extensions

    Python’s markdown package supports extensions for tables, footnotes, and math. A well‑designed previewer lets power users enable or disable them on the fly.

    Choosing the Right GUI Toolkit for Your Previewer

    Python offers several mature GUI frameworks. Your choice will affect development speed, look‑and‑feel, and packaging complexity.

    • Tkinter: Built‑in, lightweight, but limited styling options.
    • PyQt5 / PySide2: Rich widget set, native look on all platforms, and excellent WebEngine support for HTML rendering.
    • wxPython: Native controls and good performance, though documentation can be sparse.
    • Kivy: Ideal for touch‑enabled devices, but overkill for a simple desktop previewer.

    For most developers, PyQt5 strikes the best balance between power and ease of packaging.

    Step‑by‑Step: Building a Minimal Markdown Previewer with PyQt5

    1. Set Up Your Development Environment

    python -m venv mdpreview-env
    source mdpreview-env/bin/activate  # On Windows use `mdpreview-env\Scripts\activate`
    pip install pyqt5 markdown
    

    2. Create the Main Window Layout

    The UI consists of a QTextEdit for input and a QWebEngineView for output, placed side by side using a QSplitter.

    import sys
    from PyQt5.QtWidgets import QApplication, QMainWindow, QTextEdit, QSplitter
    from PyQt5.QtWebEngineWidgets import QWebEngineView
    from PyQt5.QtCore import Qt
    import markdown
    
    class MarkdownPreviewer(QMainWindow):
        def __init__(self):
            super().__init__()
            self.setWindowTitle("Python Markdown Previewer")
            splitter = QSplitter(Qt.Horizontal)
    
            self.editor = QTextEdit()
            self.editor.textChanged.connect(self.update_preview)
            splitter.addWidget(self.editor)
    
            self.viewer = QWebEngineView()
            splitter.addWidget(self.viewer)
    
            self.setCentralWidget(splitter)
            self.resize(900, 600)
    
        def update_preview(self):
            raw_md = self.editor.toPlainText()
            html = markdown.markdown(raw_md, extensions=['fenced_code', 'tables'])
            self.viewer.setHtml(html)
    
    if __name__ == "__main__":
        app = QApplication(sys.argv)
        window = MarkdownPreviewer()
        window.show()
        sys.exit(app.exec_())
    

    3. Add Theme Switching (Optional)

    Inject a small CSS block into the generated HTML to toggle between light and dark modes.

    def apply_theme(self, dark=False):
        css = """
        body { font-family: Arial, sans-serif; margin: 20px; }
        pre { background:#f4f4f4; padding:10px; }
        """ if not dark else """
        body { background:#2e2e2e; color:#e0e0e0; font-family:Arial, sans-serif; margin:20px; }
        pre { background:#3e3e3e; color:#e0e0e0; padding:10px; }
        """
        self.viewer.page().runJavaScript(f"document.body.style = `{css}`;")
    

    Enhancing the Previewer: Real‑World Features

    Live Reload with Debounce

    Typing fast can trigger dozens of renders per second. Use a QTimer to debounce updates.

    from PyQt5.QtCore import QTimer
    
    self.timer = QTimer()
    self.timer.setInterval(300)  # milliseconds
    self.timer.setSingleShot(True)
    self.timer.timeout.connect(self.update_preview)
    
    self.editor.textChanged.connect(self.timer.start)
    

    Export to PDF

    Leverage the built‑in printing support of QWebEngineView:

    def export_pdf(self, path):
        printer = QPrinter()
        printer.setOutputFormat(QPrinter.PdfFormat)
        printer.setOutputFileName(path)
        self.viewer.page().print(printer, lambda ok: print("PDF saved" if ok else "Failed"))
    

    Adding Custom Markdown Extensions

    For example, enable GitHub‑style task lists:

    html = markdown.markdown(
        raw_md,
        extensions=['fenced_code', 'tables', 'toc', 'mdx_gfm']
    )
    

    Packaging the Application for End Users

    After you’re happy with the feature set, turn the script into a distributable executable.

    1. PyInstaller: The most popular choice. Run pyinstaller --onefile --windowed your_app.py to generate a single .exe (Windows) or .app (macOS) bundle.
    2. cx_Freeze: Works well for cross‑platform builds but requires a bit more configuration.
    3. Briefcase (BeeWare): Creates native installers and can even target mobile, though it’s heavier.

    Remember to include the QtWebEngineProcess binaries and any custom CSS or icon files in the --add-data argument, otherwise the preview pane will appear blank.

    SEO Tips for Your Markdown Previewer Project

    If you plan to share the project on GitHub, a personal site, or the Python Package Index (PyPI), follow these SEO best practices to attract organic traffic:

    • Keyword‑rich README: Use phrases like “Python markdown previewer”, “desktop Markdown editor”, and “live Markdown rendering”.
    • Meta tags in the project website: Include <meta name="description" content="Cross‑platform Python markdown previewer with live rendering and PDF export.">.
    • Blog posts and tutorials: Repurpose the content above into a series of short articles, each targeting a specific long‑tail keyword.
    • Video demo: Upload a quick walkthrough to YouTube; the video description can reinforce the same keywords.

    Common Pitfalls and How to Avoid Them

    Missing QtWebEngine Dependencies

    On Linux, the qtwebengine5-dev package may be required. Test the executable on a clean VM before publishing.

    Performance Bottlenecks with Large Files

    Rendering a 10,000‑line Markdown file can freeze the UI. Mitigate this by:

    • Using the debounce timer mentioned earlier.
    • Only re‑render the changed portion if you integrate a diff algorithm.
    • Offering a “Pause Live Preview” toggle for power users.

    Cross‑Platform Font Inconsistencies

  • Python Automated Api Endpoint Tester

    Testing API endpoints manually is a time‑consuming nightmare that often leads to missed edge cases, flaky releases, and frustrated developers. Fortunately, Python makes it easy to automate the entire testing workflow, turning repetitive curl commands into a robust, maintainable suite that can run on every pull request. In this guide you’ll learn how to build a Python automated API endpoint tester from scratch, integrate it with pytest and GitHub Actions, and scale it for large micro‑service architectures—all while keeping the code clean, readable, and SEO‑friendly.

    Why Automate API Testing with Python?

    Python’s simplicity, extensive ecosystem, and strong community support make it the go‑to language for API testing. Here are the top reasons you should consider a Python automated API endpoint tester for your next project:

    • Readability: Python’s syntax reads like plain English, reducing the learning curve for new team members.
    • Rich libraries: Packages such as requests, httpx, and pytest provide all the building blocks you need.
    • CI/CD ready: Python scripts can be executed in any CI environment, enabling continuous verification of API contracts.
    • Extensibility: Plug‑in support for authentication, data generation, and reporting lets you grow the tester alongside your API.

    Core Components of a Python API Tester

    A well‑structured tester typically consists of four layers:

    1. Configuration layer – stores base URLs, environment variables, and authentication tokens.
    2. Request layer – builds and sends HTTP requests using a library like requests or httpx.
    3. Validation layer – asserts response status, schema, and business logic.
    4. Reporting layer – aggregates results and formats them for CI pipelines.

    1. Setting Up the Project Structure

    Start with a clean directory layout that separates concerns and encourages reuse:

    api_tester/
    │
    ├─ tests/                # pytest test modules
    │   ├─ __init__.py
    │   └─ test_users.py
    │
    ├─ utils/                # helper functions (request builder, auth, etc.)
    │   ├─ __init__.py
    │   └─ client.py
    │
    ├─ config.yaml           # environment‑specific settings
    ├─ requirements.txt
    └─ pytest.ini            # pytest configuration
    

    2. Managing Configuration with YAML

    Store environment‑specific data in config.yaml. This keeps secrets out of source code and makes it easy to switch between dev, staging, and production.

    # config.yaml
    default:
      base_url: https://api.example.com/v1
      timeout: 10
    
    staging:
      base_url: https://staging.api.example.com/v1
      auth_token: ${STAGING_TOKEN}
    
    production:
      base_url: https://api.example.com/v1
      auth_token: ${PROD_TOKEN}
    

    Load the configuration at runtime using PyYAML:

    import os
    import yaml
    
    def load_config(env: str = "default"):
        with open("config.yaml") as f:
            cfg = yaml.safe_load(f)
        # Merge default with selected env
        merged = {**cfg["default"], **cfg.get(env, {})}
        # Resolve environment variables
        for key, value in merged.items():
            if isinstance(value, str) and value.startswith("${") and value.endswith("}"):
                env_key = value[2:-1]
                merged[key] = os.getenv(env_key)
        return merged
    

    3. Building a Reusable HTTP Client

    The client.py module abstracts away the low‑level request details. By using a session object you gain connection pooling and automatic header handling.

    import requests
    from .config import load_config
    
    class APIClient:
        def __init__(self, env="default"):
            self.cfg = load_config(env)
            self.session = requests.Session()
            self.session.headers.update({
                "Accept": "application/json",
                "User-Agent": "PythonAPItester/1.0"
            })
            token = self.cfg.get("auth_token")
            if token:
                self.session.headers["Authorization"] = f"Bearer {token}"
    
        def request(self, method: str, endpoint: str, **kwargs):
            url = f"{self.cfg['base_url'].rstrip('/')}/{endpoint.lstrip('/')}"
            response = self.session.request(method, url, timeout=self.cfg["timeout"], **kwargs)
            response.raise_for_status()  # raise for 4xx/5xx automatically
            return response
    

    4. Writing Tests with pytest

    With the client ready, writing a test is as simple as describing the expected behavior. Below is a sample test for a /users endpoint that returns a list of user objects.

    import pytest
    from utils.client import APIClient
    
    @pytest.fixture(scope="module")
    def api():
        return APIClient(env="staging")
    
    def test_get_users_returns_200(api):
        resp = api.request("GET", "/users")
        assert resp.status_code == 200
        data = resp.json()
        assert isinstance(data, list)
        # Basic schema check for the first item
        if data:
            user = data[0]
            assert "id" in user
            assert "email" in user
            assert isinstance(user["id"], int)
            assert isinstance(user["email"], str)
    

    5. Adding JSON Schema Validation

    For stricter contracts, integrate jsonschema. Define a reusable schema file (schemas/user.json) and validate each response against it.

    # schemas/user.json
    {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "type": "object",
      "required": ["id", "email", "created_at"],
      "properties": {
        "id": {"type": "integer"},
        "email": {"type": "string", "format": "email"},
        "created_at": {"type": "string", "format": "date-time"},
        "name": {"type": ["string", "null"]},
        "is_active": {"type": "boolean"}
      }
    }
    
    from jsonschema import validate, ValidationError
    import json
    import pathlib
    
    SCHEMA_PATH = pathlib.Path(__file__).parent.parent / "schemas" / "user.json"
    USER_SCHEMA = json.loads(SCHEMA_PATH.read_text())
    
    def test_user_schema(api):
        resp = api.request("GET", "/users/1")
        data = resp.json()
        try:
            validate(instance=data, schema=USER_SCHEMA)
        except ValidationError as e:
            pytest.fail(f"JSON schema validation failed: {e.message}")
    

    Running Tests Locally and in CI

    Once your test suite is ready, you can run it with a single command:

    pytest -v tests/

    To integrate with a CI system like GitHub Actions, add a workflow file (.github/workflows/api-test.yml) that installs dependencies, sets environment variables, and executes the test suite on each push.

    # .github/workflows/api-test.yml
    name: API Endpoint Tests
    
    on:
      push:
        branches: [main, develop]
      pull_request:
    
    jobs:
      test:
        runs-on: ubuntu-latest
        env:
          STAGING_TOKEN: ${{ secrets.STAGING_TOKEN }}
          PROD_TOKEN: ${{ secrets.PROD_TOKEN }}
        steps:
          - uses: actions/checkout@v4
          - name: Set up Python
            uses: actions/setup-python@v5
            with:
              python-version: "3.12"
          - name: Install dependencies
            run: |
              python -m pip install --upgrade pip
              pip install -r requirements.txt
          - name: Run pytest
            run: pytest -v tests/
    

    Best Practices for a Scalable Tester

    • Parameterize tests: Use @pytest.mark.parametrize to cover multiple input combinations without duplicating code.
    • Data‑driven testing: Store request payloads and expected responses in JSON or CSV files and load them at runtime.
    • Parallel execution: Leverage pytest-xdist to run tests concurrently, reducing total execution time.
    • Mock external services: When testing integration points, use responses or pytest-mock to simulate third‑party APIs.
    • Version control schemas: Keep JSON schema files under Git to track contract changes over time.

    Extending the Tester with Advanced Features

    1. Automatic Test Generation from OpenAPI Specs

    If your API already has an OpenAPI (Swagger) definition, you can generate test cases automatically with tools like schemathesis. This eliminates the manual effort of writing repetitive endpoint tests.

    # Install schemathesis
    pip install schemathesis
    
    # Run generated tests against the staging environment
    schemathesis run https://staging.api.example.com/openapi.json \
        --base-url https://staging.api.example.com/v1 \
        --hypothesis-max-examples 50
    

    2. Performance and Load Checks

    Combine functional testing with simple performance assertions using time or locust. For example, ensure that response time stays under 300 ms:

    def test_response_time(api):
        import time
        start = time.perf_counter()
        resp = api.request("GET", "/orders")
        elapsed = (time.perf_counter() - start) * 1000  # ms
        assert elapsed < 300, f"Response took {elapsed:.1f} ms"
    

    3. Generating HTML Test Reports

  • Python Calendar Event Auto Scheduler

    Imagine never having to manually type a meeting, reminder, or birthday again. With a Python calendar event auto scheduler, you can let code handle the heavy lifting—creating, updating, and syncing events across multiple platforms with just a few lines of script. In this guide we’ll explore why automated scheduling matters, walk through the core Python libraries you’ll need, and build a practical, extensible solution that works with Google Calendar, Outlook, and even local iCal files. Ready to turn your calendar into a smart assistant? Let’s dive in.

    Why Automate Calendar Events?

    Manual entry is not only time‑consuming, it’s also error‑prone. Here are the top reasons businesses and developers choose to automate their calendars:

    • Consistency: Guarantees every event follows the same naming, tagging, and reminder conventions.
    • Scalability: Create hundreds of recurring meetings or task reminders with a single script.
    • Integration: Sync events across Google Calendar, Microsoft Outlook, Apple iCal, and internal tools.
    • Productivity: Free up mental bandwidth for strategic work instead of repetitive data entry.

    Key Python Libraries for Calendar Automation

    Built‑in Modules

    • datetime – Handles dates, times, and time‑zone conversions.
    • calendar – Generates month calendars, calculates weekdays, and supports locale‑specific data.

    Third‑Party Packages

    • icalendar – Read and write iCalendar (.ics) files, perfect for local calendar exports.
    • google-api-python-client & google-auth – Interact with Google Calendar via its REST API.
    • exchangelib – Connect to Microsoft Exchange and Outlook calendars.
    • pytz or zoneinfo (Python 3.9+) – Manage time‑zone aware datetime objects.

    Setting Up Your Development Environment

    1. Install Python 3.10+ (recommended for zoneinfo support).
    2. Create a virtual environment:
      python -m venv venv
      source venv/bin/activate  # On Windows: venv\Scripts\activate
    3. Install required packages:
      pip install icalendar google-api-python-client google-auth exchangelib pytz
    4. Obtain API credentials:
      • Google: Create a project in Google Cloud Console, enable Calendar API, and download credentials.json.
      • Outlook: Register an app in Azure AD and note the client ID, secret, and tenant ID.

    Designing the Auto Scheduler Architecture

    A clean architecture keeps your code maintainable and allows you to swap out calendar providers without rewriting the core logic. Below is a simple layered approach:

    • Input Layer – Reads event definitions from CSV, JSON, or a database.
    • Business Logic Layer – Calculates recurrence rules, resolves time‑zones, and validates data.
    • Adapter Layer – Contains provider‑specific classes (GoogleCalendarAdapter, OutlookAdapter, IcsAdapter).
    • Output Layer – Sends events to the chosen calendar service.

    Sample Event Definition (JSON)

    {
      "title": "Weekly Team Sync",
      "description": "Discuss project updates and blockers.",
      "start": "2026-10-20T10:00:00",
      "duration_minutes": 60,
      "timezone": "America/New_York",
      "recurrence": "RRULE:FREQ=WEEKLY;COUNT=10",
      "reminders": [15, 5],
      "attendees": ["alice@example.com", "bob@example.com"]
    }

    Implementing the Core Scheduler in Python

    1. Parsing and Normalizing Event Data

    import json
    from datetime import datetime, timedelta
    from zoneinfo import ZoneInfo
    
    def load_event(path: str) -> dict:
        with open(path, 'r', encoding='utf-8') as f:
            data = json.load(f)
    
        # Convert start string to aware datetime
        tz = ZoneInfo(data['timezone'])
        data['start_dt'] = datetime.fromisoformat(data['start']).replace(tzinfo=tz)
    
        # Compute end time
        data['end_dt'] = data['start_dt'] + timedelta(minutes=data['duration_minutes'])
        return data
    

    2. Building an iCalendar Object (Portable Export)

    from icalendar import Calendar, Event, vRecur
    
    def build_ics(event: dict) -> Calendar:
        cal = Calendar()
        cal.add('prodid', '-//Python Auto Scheduler//')
        cal.add('version', '2.0')
    
        vevent = Event()
        vevent.add('summary', event['title'])
        vevent.add('dtstart', event['start_dt'])
        vevent.add('dtend', event['end_dt'])
        vevent.add('description', event['description'])
        vevent.add('uid', f"{event['title']}-{event['start_dt'].timestamp()}@example.com")
        vevent.add('rrule', vRecur.from_ical(event['recurrence']))
    
        # Add reminders as VALARM components
        for minutes in event['reminders']:
            alarm = Calendar()
            alarm.add('action', 'DISPLAY')
            alarm.add('description', f"Reminder: {event['title']}")
            alarm.add('trigger', timedelta(minutes=-minutes))
            vevent.add_component(alarm)
    
        cal.add_component(vevent)
        return cal
    

    3. Google Calendar Adapter

    from google.oauth2 import service_account
    from googleapiclient.discovery import build
    
    SCOPES = ['https://www.googleapis.com/auth/calendar']
    SERVICE_ACCOUNT_FILE = 'service-account.json'
    
    class GoogleCalendarAdapter:
        def __init__(self):
            credentials = service_account.Credentials.from_service_account_file(
                SERVICE_ACCOUNT_FILE, scopes=SCOPES)
            self.service = build('calendar', 'v3', credentials=credentials)
    
        def create_event(self, event: dict, calendar_id='primary'):
            body = {
                'summary': event['title'],
                'description': event['description'],
                'start': {
                    'dateTime': event['start_dt'].isoformat(),
                    'timeZone': event['timezone'],
                },
                'end': {
                    'dateTime': event['end_dt'].isoformat(),
                    'timeZone': event['timezone'],
                },
                'recurrence': [event['recurrence']],
                'reminders': {
                    'useDefault': False,
                    'overrides': [{'method': 'popup',
                                   'minutes': m} for m in event['reminders']],
                },
                'attendees': [{'email': a} for a in event['attendees']],
            }
            created = self.service.events().insert(calendarId=calendar_id, body=body).execute()
            return created.get('htmlLink')
    

    4. Putting It All Together

    def main():
        event = load_event('weekly_sync.json')
        
        # Export to .ics for local use
        cal = build_ics(event)
        with open('weekly_sync.ics', 'wb') as f:
            f.write(cal.to_ical())
        
        # Push to Google Calendar
        google = GoogleCalendarAdapter()
        link = google.create_event(event)
        print(f'Event created: {link}')
    
    if __name__ == '__main__':
        main()
    

    The script above demonstrates a full workflow: read a JSON definition, generate a portable iCal file, and push the event to Google Calendar. By swapping GoogleCalendarAdapter with an OutlookAdapter that uses exchangelib, you instantly support multiple platforms without altering the core logic.

    Handling Recurring Events and Edge Cases

    Recurring events often cause headaches because different providers interpret RRULE syntax slightly differently. Here are best practices to keep your scheduler robust:

    • Validate RRULE strings. Use dateutil.rrule to parse and test recurrence before sending it to an API.
    • Limit the count. Avoid infinite recurrences; set a reasonable COUNT or UNTIL date.
    • Time‑zone consistency. Store all datetimes as UTC internally, then convert to the user’s zone when building API payloads.
    • Conflict detection. Pull existing events for the target window and compare start/end times to prevent double‑booking.

    Scheduling Jobs: Running the Auto Scheduler Automatically

    Once your script works, you’ll want it to run on a schedule—daily, hourly, or triggered by a webhook. Choose the method that matches your deployment environment:

    Local or Server‑Based Cron Jobs

    # Example crontab entry – run at 02:00 UTC every day
    0 2 * * * /path/to/venv/bin/python /path/to/scheduler.py >> /var/log/scheduler.log 2>&1

    Cloud Functions / Server

  • Python Exif Metadata Stripper Tool

    In an age where every photo you share can unintentionally reveal your location, device model, and even personal habits, protecting your privacy has never been more critical. One of the most effective ways to safeguard that information is by stripping EXIF metadata from images before they go public. This article walks you through everything you need to know about building a reliable Python EXIF metadata stripper tool—from the basics of EXIF data to a fully functional, command‑line script you can customize for any workflow.

    What Is EXIF Metadata and Why Remove It?

    EXIF (Exchangeable Image File Format) is a set of tags embedded directly into JPEG, TIFF, and many RAW image files. These tags store a wealth of information, such as:

    • Camera make and model
    • GPS coordinates (latitude, longitude, altitude)
    • Date and time the photo was taken
    • Exposure settings (shutter speed, aperture, ISO)
    • Software used for editing

    While this data is useful for photographers and developers, it can also expose sensitive details to anyone who accesses the image. For businesses, photographers, journalists, or everyday social‑media users, removing EXIF can prevent:

    • Unwanted location tracking
    • Device fingerprinting
    • Legal liabilities when sharing images that contain personal data
    • Unnecessary file size bloat (some tags add a few kilobytes)

    Choosing the Right Python Libraries

    Python offers several libraries that make EXIF manipulation straightforward. The two most popular choices are:

    1. Pillow (PIL Fork)

    Pillow is the de‑facto imaging library for Python. It can read and write image files and provides a simple Image.getexif() method for accessing EXIF data. However, Pillow’s EXIF handling is limited to reading; it cannot easily delete or modify tags without re‑encoding the image.

    2. piexif

    piexif is a lightweight library specifically designed for EXIF operations. It allows you to:

    • Read EXIF metadata as a dictionary
    • Delete specific tags or the entire EXIF block
    • Insert new EXIF data if needed

    For a dedicated EXIF stripper, piexif is the preferred choice because it manipulates the metadata without altering the pixel data, preserving image quality.

    Setting Up Your Development Environment

    Before writing any code, ensure you have a clean Python environment. Follow these steps:

    # 1. Create a virtual environment (recommended)
    python -m venv exif-stripper-env
    # 2. Activate the environment
    # Windows
    exif-stripper-env\Scripts\activate
    # macOS / Linux
    source exif-stripper-env/bin/activate
    
    # 3. Install required packages
    pip install pillow piexif tqdm
    

    The tqdm package adds a progress bar, which is handy when processing large batches of images.

    Building the Core Stripper Function

    Below is a concise, well‑documented function that removes all EXIF data from a single image file while preserving its original format and quality.

    import os
    import piexif
    from PIL import Image
    
    def strip_exif(input_path: str, output_path: str) -> None:
        """
        Remove all EXIF metadata from an image.
    
        Parameters
        ----------
        input_path : str
            Path to the original image file.
        output_path : str
            Path where the cleaned image will be saved.
        """
        # Open the image using Pillow (ensures compatibility with many formats)
        with Image.open(input_path) as img:
            # Preserve the original format (JPEG, PNG, etc.)
            img_format = img.format
    
            # Remove EXIF by inserting an empty EXIF block
            empty_exif = piexif.dump({})
            img.save(output_path, format=img_format, exif=empty_exif)
    
        # Optional: Verify that EXIF is gone
        if os.path.getsize(output_path) == 0:
            raise RuntimeError("Failed to write stripped image.")
    

    Command‑Line Interface (CLI) for Batch Processing

    Most users need to clean dozens or hundreds of images at once. The following script adds a simple CLI using argparse and processes directories recursively.

    import argparse
    import os
    from pathlib import Path
    from tqdm import tqdm
    
    def process_folder(source: Path, destination: Path, overwrite: bool = False) -> None:
        """
        Recursively strip EXIF from all supported image files in `source`
        and write them to `destination`, preserving the folder structure.
        """
        supported_ext = {".jpg", ".jpeg", ".tif", ".tiff", ".png"}
    
        for root, _, files in os.walk(source):
            for file_name in files:
                src_file = Path(root) / file_name
                if src_file.suffix.lower() not in supported_ext:
                    continue
    
                # Build the output path, mirroring the source hierarchy
                relative_path = src_file.relative_to(source)
                dest_file = destination / relative_path
                dest_file.parent.mkdir(parents=True, exist_ok=True)
    
                # Skip if file exists and overwrite is False
                if dest_file.exists() and not overwrite:
                    continue
    
                try:
                    strip_exif(str(src_file), str(dest_file))
                except Exception as e:
                    tqdm.write(f"Error processing {src_file}: {e}")
    
    def main() -> None:
        parser = argparse.ArgumentParser(
            description="Python EXIF metadata stripper – remove all EXIF data from images."
        )
        parser.add_argument("source", type=Path, help="Folder containing original images")
        parser.add_argument("destination", type=Path, help="Folder for cleaned images")
        parser.add_argument("-f", "--force", action="store_true",
                            help="Overwrite existing files in destination")
        args = parser.parse_args()
    
        if not args.source.is_dir():
            raise NotADirectoryError(f"Source folder '{args.source}' does not exist.")
    
        args.destination.mkdir(parents=True, exist_ok=True)
    
        # Use tqdm for a nice progress bar
        total_images = sum(
            1 for p in args.source.rglob("*") if p.suffix.lower() in {".jpg", ".jpeg", ".tif", ".tiff", ".png"}
        )
        with tqdm(total=total_images, unit="image") as pbar:
            for root, _, files in os.walk(args.source):
                for file_name in files:
                    src_file = Path(root) / file_name
                    if src_file.suffix.lower() not in {".jpg", ".jpeg", ".tif", ".tiff", ".png"}:
                        continue
                    relative_path = src_file.relative_to(args.source)
                    dest_file = args.destination / relative_path
                    dest_file.parent.mkdir(parents=True, exist_ok=True)
    
                    if dest_file.exists() and not args.force:
                        pbar.update(1)
                        continue
    
                    try:
                        strip_exif(str(src_file), str(dest_file))
                    except Exception as exc:
                        tqdm.write(f"Failed {src_file}: {exc}")
                    finally:
                        pbar.update(1)
    
    if __name__ == "__main__":
        main()
    

    Key Features of the Tool

    • Cross‑platform: Works on Windows, macOS, and Linux.
    • Batch processing: Handles entire directory trees with a single command.
    • Preserves image quality: No recompression for lossless formats; JPEGs retain original quality settings.
    • Safety checks: Skips already‑cleaned files unless --force is used.
    • Progress feedback: Integrated tqdm progress bar for a smooth user experience.

    Best Practices for Using an EXIF Stripper

    1. Keep Original Files Backed Up

    Stripping EXIF is irreversible. Always retain a backup of the original images in a secure location. This practice prevents data loss if you later need the metadata for legal, forensic, or archival reasons.

    2. Verify the Output

    After processing, you can confirm that EXIF data has been removed using tools like exiftool:

    exiftool cleaned_image.jpg | grep -i "exif"
    # No output means EXIF is gone.
    

    3. Automate in CI/CD Pipelines

    If you run a website that hosts user‑uploaded images, integrate the stripper into your deployment pipeline. For example, a GitHub Action can invoke the script whenever new assets are pushed, guaranteeing that no EXIF data ever reaches production.

    4. Consider Legal Requirements

    Some jurisdictions (e.g., GDPR in the EU) treat location data as personal information. Stripping EXIF can help you stay compliant, but always consult a legal professional to ensure your overall data‑handling practices meet regional regulations.

    Frequently Asked Questions (FAQ)

    Can I preserve specific EXIF tags while removing the rest?

    Yes. piexif lets you edit the EXIF dictionary before re‑inserting it. Load the EXIF data, delete unwanted keys, then call piexif.dump() with the modified dictionary.

    Does the tool work with PNG files?

    PNG does not have native EXIF, but it can contain tEXt or iTXt chunks that store similar metadata

  • Python Static Site Generator Project

    Are you a Python enthusiast looking to turn markdown files, Jinja templates, or plain text into a blazing‑fast, SEO‑friendly website? A Python static site generator (SSG) project can give you full control over the build process, keep your site lightweight, and boost your development skills—all without the overhead of a heavyweight CMS. In this guide we’ll explore why static site generators matter, walk through the core components of a Python‑based SSG, and provide a step‑by‑step roadmap to launch your own project that ranks well, loads instantly, and scales effortlessly.

    Why Choose a Python Static Site Generator?

    Static sites are pre‑rendered HTML files that are served directly to browsers. Compared to dynamic sites, they offer:

    • Speed: No database queries or server‑side rendering at request time.
    • Security: Fewer attack vectors because there’s no runtime code execution on the server.
    • Scalability: Simple CDN deployment can handle millions of hits with virtually zero cost.
    • SEO advantages: Search engines crawl static HTML instantly, improving indexability.

    Python is an excellent language for building an SSG because of its readable syntax, powerful standard library, and vibrant ecosystem of templating (Jinja2), markdown (Markdown), and file‑system utilities (pathlib). Whether you’re creating a personal blog, documentation portal, or marketing landing page, a custom Python static site generator gives you the flexibility to tailor every step of the pipeline.

    Core Architecture of a Python SSG

    Below is a high‑level view of the typical components you’ll find in a well‑structured Python static site generator project:

    1. Content Source

    Content is usually stored as plain text files—most commonly markdown (.md) or reStructuredText (.rst). Organize them in a content/ directory with subfolders for posts, pages, and assets.

    2. Front‑Matter Parser

    Front‑matter provides metadata (title, date, tags, layout) at the top of each content file. YAML is the de‑facto standard:

    ---
    title: "Building a Python SSG"
    date: 2026-10-01
    tags: [python, static site, tutorial]
    layout: post.html
    ---
    

    Use a library like PyYAML to extract this data into a Python dictionary.

    3. Markdown Renderer

    Convert the markdown body into HTML with markdown or mistune. Enable extensions for tables, footnotes, and code highlighting to enrich the output.

    4. Templating Engine

    Jinja2 shines as the templating engine of choice. Define base layouts (base.html) and content‑specific templates (post.html, page.html). The renderer injects the parsed front‑matter and HTML body into the template context.

    5. Asset Pipeline

    Static assets (CSS, JavaScript, images) live in an assets/ folder. A simple copy step or a more advanced pipeline (e.g., csscompressor, jsmin) ensures they are bundled and versioned for cache busting.

    6. Build Script

    The heart of the project is a CLI script (often called build.py) that orchestrates the workflow:

    1. Walk the content/ directory.
    2. Parse front‑matter and markdown.
    3. Render each page with Jinja2.
    4. Write the resulting HTML to output/ preserving the URL structure.
    5. Copy assets and generate a sitemap.

    Step‑by‑Step: Building Your First Python SSG

    Let’s dive into a minimal yet production‑ready implementation. The following code snippets assume a fresh project layout:

    my_ssg/
    ├─ content/
    │  ├─ posts/
    │  │  └─ hello-world.md
    │  └─ pages/
    │     └─ about.md
    ├─ templates/
    │  ├─ base.html
    │  └─ post.html
    ├─ assets/
    │  ├─ css/
    │  │  └─ style.css
    │  └─ img/
    │     └─ logo.png
    ├─ build.py
    └─ requirements.txt
    

    2. Install Dependencies

    pip install markdown Jinja2 PyYAML python-slugify
    

    3. Create a Simple Base Template

    Save this as templates/base.html:

    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>{{ title }} | My Python SSG</title>
        <meta name="description" content="{{ description|default('A static site built with Python') }}">
        <link rel="stylesheet" href="/css/style.css">
    </head>
    <body>
        <header>
            <h1><a href="/">My Python SSG</a></h1>
        </header>
        <main>
            {{ content|safe }}
        </main>
        <footer>
            <p>&copy; {{ year }} My Python SSG</p>
        </footer>
    </body>
    </html>
    

    4. Post Template Extending the Base

    Save as templates/post.html:

    {% extends "base.html" %}
    {% block content %}
    <article>
        <h2>{{ title }}</h2>
        <p class="meta">Published on {{ date }}{% if tags %} | Tags: {{ tags|join(', ') }}{% endif %}</p>
        {{ body|safe }}
    </article>
    {% endblock %}
    

    5. The Build Script

    Below is a concise build.py that ties everything together. It includes SEO‑friendly features such as a sitemap, clean URLs, and automatic rel="canonical" handling.

    #!/usr/bin/env python3
    import os
    import shutil
    import datetime
    from pathlib import Path
    import yaml
    import markdown
    from jinja2 import Environment, FileSystemLoader, select_autoescape
    from slugify import slugify

    # Directories
    BASE_DIR = Path(__file__).parent
    CONTENT_DIR = BASE_DIR / "content"
    TEMPLATE_DIR = BASE_DIR / "templates"
    ASSET_DIR = BASE_DIR / "assets"
    OUTPUT_DIR = BASE_DIR / "output"

    # Initialize Jinja2
    env = Environment(
    loader=FileSystemLoader(TEMPLATE_DIR),
    autoescape=select_autoescape(['html', 'xml'])
    )

    def load_markdown(file_path):
    """Read a markdown file, split front‑matter and body."""
    with open(file_path, "r", encoding="utf-8") as f:
    raw = f.read()
    if raw.startswith("---"):
    _, fm, body = raw.split("---", 2)
    front_matter = yaml.safe_load(fm)
    else:
    front_matter = {}
    body = raw
    html_body = markdown.markdown(
    body,
    extensions=["extra", "codehilite", "toc", "tables"]
    )
    return front_matter, html_body

    def render_page(front_matter, html_body):
    """Select a template and render the final HTML."""
    layout = front_matter.get("layout", "base.html")
    template = env.get_template(layout)
    context = {
    "title": front_matter.get("title", "Untitled"),
    "description": front_matter.get("description", ""),
    "date": front_matter.get("date", ""),
    "tags": front_matter.get("tags", []),
    "body": html_body,
    "year": datetime.datetime.now().year,
    "canonical": front_matter.get("canonical", ""),
    }
    return template.render(**context)

    def write_output(relative_path, html):
    """Write rendered HTML to the output folder, preserving clean URLs."""
    # Example: posts/hello-world.md -> output/posts/hello-world/index.html
    output_path = OUTPUT_DIR / relative_path.with_suffix("")
    output_path = output_path / "index.html"
    output_path.parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w", encoding="utf-8") as f:
    f.write(html)

    def copy_assets():
    """Recursively copy static assets to the output directory."""
    if ASSET_DIR.exists():
    dest = OUTPUT_DIR / "assets"
    if dest.exists():
    shutil.rmtree(dest)
    shutil.copytree(ASSET_DIR, dest)

    def generate_sitemap(pages):
    """Create a simple XML sitemap for SEO."""
    sitemap_path = OUTPUT_DIR / "sitemap.xml"
    now = datetime.datetime.utcnow().date().isoformat()
    lines = ['',
    '']
    for page in pages:
    url = f"https://example.com/{page}/"
    lines.append(" ")
    lines.append(f" {url}")
    lines.append(f" {now}")
    lines.append(" weekly")
    lines.append(" 0.8")
    lines.append("
    ")
    lines.append("
    ")
    sitemap_path.write_text("\n".join(lines), encoding="utf-8")

    def main():
    # Clean previous build
    if OUTPUT

  • Python Whois Domain Lookup Tool

    When you type a domain name into your browser, a hidden world of registration data works behind the scenes. This data, known as WHOIS, reveals who owns a domain, when it expires, and where it’s hosted. For developers, security analysts, and digital marketers, accessing WHOIS information programmatically can unlock powerful insights—whether you’re tracking brand impersonation, auditing SSL certificates, or simply verifying the legitimacy of a new client. In this article we’ll explore how to build a robust Python WHOIS domain lookup tool, discuss the best libraries, walk through a complete code example, and share tips for handling rate limits, errors, and deployment. By the end, you’ll have a ready‑to‑run script and a clear roadmap for extending it into a full‑featured service.

    What Is WHOIS and Why It Matters

    Understanding the Core Data

    WHOIS is a protocol defined by the Internet Assigned Numbers Authority (IANA) that lets anyone query a domain’s registration record. The response typically includes:

    • Registrar information – the company that sold the domain.
    • Registrant contact – name, email, and sometimes phone number of the owner.
    • Creation and expiration dates – crucial for renewal monitoring.
    • Name servers – indicate where the domain’s DNS records are hosted.
    • Domain status – e.g., clientTransferProhibited or active.

    These details are valuable for:

    1. Detecting phishing or brand‑squatting attempts.
    2. Performing bulk domain audits for SEO campaigns.
    3. Automating renewal reminders for clients.
    4. Gathering intelligence for cybersecurity investigations.

    Python Libraries for WHOIS Lookups

    python‑whois (the most popular)

    The python-whois package parses raw WHOIS responses into a Python dictionary, handling many TLD quirks automatically. It works with both IPv4 and IPv6 addresses and supports a wide range of domain extensions.

    whois (alternative implementation)

    The whois library offers a slightly different API and focuses on speed. It’s a good choice when you need lightweight lookups without the extra parsing overhead.

    Raw socket approach

    For full control, you can open a TCP connection to the appropriate WHOIS server (usually port 43) and send the domain name manually. This method is educational but requires you to handle different server responses and rate‑limit policies yourself.

    Building a Simple WHOIS Tool in Python

    Step 1 – Install Dependencies

    pip install python-whois
    # Optional: install whois for comparison
    pip install whois
    

    Step 2 – Core Script Explanation

    Below is a concise, production‑ready script that accepts a domain name from the command line, performs a WHOIS lookup, and prints the most useful fields in a clean format.

    #!/usr/bin/env python3
    import sys
    import whois  # from python-whois package
    from datetime import datetime
    
    def format_date(value):
        """Convert WHOIS date strings to a readable format."""
        if isinstance(value, list):
            value = value[0]
        if isinstance(value, datetime):
            return value.strftime('%Y-%m-%d %H:%M:%S')
        return str(value) if value else 'N/A'
    
    def display_result(data):
        """Print selected WHOIS fields with clear labels."""
        fields = {
            'Domain': data.domain_name,
            'Registrar': data.registrar,
            'Creation Date': format_date(data.creation_date),
            'Expiration Date': format_date(data.expiration_date),
            'Updated Date': format_date(data.updated_date),
            'Name Servers': ', '.join(data.name_servers) if data.name_servers else 'N/A',
            'Status': ', '.join(data.status) if data.status else 'N/A',
            'Registrant Email': data.emails[0] if data.emails else 'N/A'
        }
        print('\n=== WHOIS Lookup Result ===')
        for label, value in fields.items():
            print(f'{label:<18}: {value}')
    
    def main():
        if len(sys.argv) != 2:
            print('Usage: python whois_lookup.py ')
            sys.exit(1)
    
        domain = sys.argv[1].strip().lower()
        try:
            result = whois.whois(domain)
            display_result(result)
        except Exception as e:
            print(f'Error retrieving WHOIS for {domain}: {e}')
            sys.exit(1)
    
    if __name__ == '__main__':
        main()
    

    This script demonstrates best practices:

    • Graceful error handling with try/except.
    • Normalization of date fields for human‑readable output.
    • Selective display of only the most relevant attributes.
    • Command‑line interface for quick testing.

    Advanced Features You Can Add

    Once the basic tool is working, consider extending it with the following capabilities to make it truly production‑grade:

    • Bulk processing – read a list of domains from a CSV file and output results to a new spreadsheet.
    • Cache layer – store recent lookups in Redis or SQLite to avoid repeated queries and respect rate limits.
    • API wrapper – expose the lookup as a Flask or FastAPI endpoint for integration with other services.
    • Rate‑limit handling – implement exponential back‑off when a WHOIS server returns 429 Too Many Requests.
    • Internationalized Domain Name (IDN) support – automatically convert Unicode domains to punycode before querying.
    • Result export – provide JSON, XML, or plain‑text output options for downstream processing.

    Handling Rate Limits and Errors

    Common Failure Scenarios

    WHOIS servers can be temperamental. Typical issues include:

    • Network timeouts – the server does not respond within the default socket timeout.
    • Server‑side throttling – many queries in a short period trigger a 403 or 429 response.
    • Parsing errors – some registrars return non‑standard formats that break the default parser.

    Robust Retry Logic

    import time
    import socket
    
    def safe_whois(domain, retries=3, backoff=2):
        for attempt in range(1, retries + 1):
            try:
                return whois.whois(domain)
            except (socket.timeout, whois.parser.PywhoisError) as e:
                if attempt == retries:
                    raise
                wait = backoff ** attempt
                print(f'Attempt {attempt} failed ({e}); retrying in {wait}s...')
                time.sleep(wait)
    

    This function exponentially backs off, giving the WHOIS server a chance to recover while keeping your script responsive.

    Deploying Your WHOIS Tool

    • Command‑Line Utility – package the script with setuptools and publish it to PyPI for easy installation via pip install my-whois-tool.
    • Web Service – wrap the lookup in a Flask route:
      from flask import Flask, request, jsonify
      app = Flask(__name__)
      
      @app.route('/whois')
      def whois_endpoint():
          domain = request.args.get('domain')
          if not domain:
              return jsonify(error='Domain parameter missing'), 400
          try:
              data = whois.whois(domain)
              return jsonify(data.__dict__)
          except Exception as e:
              return jsonify(error=str(e)), 500
      
    • Docker Container – create a lightweight image:
      FROM python:3.12-slim
      WORKDIR /app
      COPY requirements.txt .
      RUN pip install -r requirements.txt
      COPY whois_lookup.py .
      ENTRYPOINT ["python", "whois_lookup.py"]
      

    These deployment options let you serve WHOIS data to internal tools, integrate with CI pipelines, or expose a public API (respecting the terms of service of each WHOIS provider).

    SEO Keywords and Best Practices for Blog Visibility

    To ensure your article ranks well in search engines, incorporate these target keywords naturally throughout the content:

    • Python WHOIS lookup
    • domain registration information
    • whois python library
    • automated WHOIS script
    • bulk WHOIS queries
    • how to parse WHOIS data
    • Python network security tools

    Use the keywords in headings, meta descriptions (outside this HTML snippet), and alt text for any future images. Keep sentences concise, and aim for a readability score around 60–70 to appeal to both technical and non‑technical readers.

    Conclusion

    Building a Python WHOIS domain lookup tool

  • Python Dns Record Lookup Utility

    When it comes to troubleshooting network issues, verifying DNS records is one of the first steps every sysadmin or developer takes. While there are plenty of online tools that can fetch A, MX, TXT, and other DNS records, having a Python DNS record lookup utility at your fingertips gives you the flexibility to automate checks, integrate them into CI pipelines, or simply avoid the hassle of opening a browser. In this guide we’ll walk through why DNS lookups matter, explore the most popular Python libraries, and build a lightweight, extensible command‑line utility that can query any record type with just a few lines of code.

    Why a Dedicated DNS Lookup Tool in Python?

    Even though the nslookup and dig commands are powerful, they have a few limitations that a Python‑based solution can overcome:

    • Automation: Scripted lookups can be scheduled, logged, and combined with other monitoring tools.
    • Cross‑platform consistency: The same code works on Linux, macOS, and Windows without relying on external binaries.
    • Extensibility: You can parse results, trigger alerts, or feed data into dashboards directly from Python.
    • Integration: Use the results in web applications, Flask APIs, or Jupyter notebooks for real‑time diagnostics.

    Choosing the Right Library

    Python offers several ways to perform DNS queries. Below is a quick comparison of the most common options:

    1. socket (built‑in)

    • Pros: No external dependencies, simple for basic A/AAAA lookups.
    • Cons: Limited to IP address resolution; cannot query MX, TXT, SRV, etc.

    2. dns.resolver from dnspython

    • Pros: Full support for all DNS record types, asynchronous API, robust error handling.
    • Cons: Requires installing a third‑party package (pip install dnspython).

    3. requests + public DNS‑over‑HTTPS (DoH) services

    • Pros: Bypasses local resolver, useful for testing external DNS behavior.
    • Cons: Dependent on internet connectivity and third‑party APIs.

    For a versatile command‑line utility we’ll focus on dnspython because it provides the most comprehensive feature set while remaining easy to install.

    Setting Up the Environment

    First, ensure you have Python 3.8+ installed. Then install dnspython:

    pip install dnspython
    

    Optionally, you can create a virtual environment to keep dependencies isolated:

    python -m venv dns-env
    source dns-env/bin/activate  # On Windows: dns-env\Scripts\activate
    pip install dnspython
    

    Building the Core Lookup Function

    The following function abstracts the DNS query logic. It accepts a domain name and a record type (e.g., A, MX, TXT) and returns a list of human‑readable results.

    import dns.resolver
    from typing import List
    
    def dns_lookup(host: str, record_type: str = "A") -> List[str]:
        """
        Perform a DNS query for the given host and record type.
        
        Args:
            host: Domain name to query (e.g., 'example.com')
            record_type: DNS record type (A, AAAA, MX, TXT, CNAME, NS, SRV, etc.)
        
        Returns:
            A list of strings representing the answer section.
        """
        try:
            answers = dns.resolver.resolve(host, record_type, raise_on_no_answer=False)
            results = []
            for rdata in answers:
                if record_type.upper() == "MX":
                    # MX records have preference and exchange
                    results.append(f"{rdata.preference} {rdata.exchange}")
                elif record_type.upper() == "SOA":
                    results.append(str(rdata.mname))
                else:
                    results.append(str(rdata))
            return results
        except dns.resolver.NXDOMAIN:
            return [f"Error: {host} does not exist (NXDOMAIN)"]
        except dns.resolver.NoAnswer:
            return [f"Error: No {record_type} record found for {host}"]
        except dns.exception.DNSException as e:
            return [f"DNS query failed: {e}"]
    

    Command‑Line Interface (CLI) with argparse

    To make the utility user‑friendly, we’ll wrap the lookup function in a simple CLI using argparse. This allows users to specify the target domain and optionally the record type.

    import argparse
    
    def main():
        parser = argparse.ArgumentParser(
            description="Python DNS record lookup utility – query any DNS record type."
        )
        parser.add_argument("domain", help="Domain name to query (e.g., example.com)")
        parser.add_argument(
            "-t", "--type",
            default="A",
            help="DNS record type (A, AAAA, MX, TXT, CNAME, NS, SRV, SOA, etc.)"
        )
        args = parser.parse_args()
    
        results = dns_lookup(args.domain, args.type.upper())
        print(f"\n{args.type.upper()} records for {args.domain}:")
        for line in results:
            print(f"  - {line}")
    
    if __name__ == "__main__":
        main()
    

    Save the script as dnslookup.py, make it executable (chmod +x dnslookup.py on Unix), and run examples like:

    • python dnslookup.py example.com -t A
    • python dnslookup.py gmail.com -t MX
    • python dnslookup.py cloudflare.com -t TXT

    Advanced Features You Can Add

    Once the basic utility is in place, consider extending it with one or more of the following enhancements to turn it into a full‑featured DNS toolbox:

    1. Parallel queries: Use concurrent.futures.ThreadPoolExecutor to resolve multiple domains at once, speeding up bulk checks.
    2. Custom DNS resolvers: Allow users to specify a DNS server IP (e.g., 8.8.8.8) via a --resolver flag.
    3. JSON output: Add a --json switch that prints results in machine‑readable format for integration with monitoring systems.
    4. DNSSEC validation: Leverage dnspython’s DNSSEC support to verify signed records.
    5. DoH fallback: If the local resolver fails, automatically query a DNS‑over‑HTTPS endpoint like Cloudflare’s https://cloudflare-dns.com/dns-query.

    Testing and Debugging Tips

    Reliable DNS tools need thorough testing. Here are some practical suggestions:

    • Use pytest with parametrized tests for multiple domains and record types.
    • Mock dns.resolver.resolve using unittest.mock to simulate NXDOMAIN, timeout, and empty answer scenarios.
    • Run the script against known public domains (e.g., example.com for A, google.com for MX) to verify correctness.
    • Check for IPv6 compatibility by querying AAAA records on both IPv4‑only and dual‑stack networks.

    SEO‑Friendly Summary

    If you’re searching for a Python DNS record lookup utility, the combination of dnspython and a concise CLI script offers a powerful, cross‑platform solution. It covers all major record types, handles errors gracefully, and can be expanded with features like parallel processing, custom resolvers, and JSON output. By integrating this utility into your development workflow, you gain instant visibility into DNS configurations, reduce reliance on external web tools, and empower automated monitoring pipelines.

    Conclusion

    Building a Python DNS lookup tool is a rewarding exercise that blends networking fundamentals with practical scripting. Whether you need a quick one‑off query or a scalable component for continuous health checks, the code snippets above provide a solid foundation. Install dnspython, customize the CLI to your needs, and you’ll have a reliable, SEO‑optimized utility that keeps your DNS diagnostics fast, accurate, and fully under your control.

  • Python Ssl Certificate Expiration Checker

    Keeping your web services secure is non‑negotiable, and one of the most common blind spots is an expired SSL/TLS certificate. An expired certificate not only triggers scary browser warnings, it can also damage your brand’s reputation and affect SEO rankings. In this guide we’ll walk through everything you need to know about building a reliable Python SSL certificate expiration checker—from the underlying concepts to a ready‑to‑run script, automation tips, and best‑practice recommendations.

    Why Monitoring SSL Certificate Expiration Matters

    • Prevent downtime: An expired certificate blocks users from accessing your site, leading to lost traffic and revenue.
    • Maintain trust: Modern browsers display prominent security warnings that erode user confidence.
    • SEO impact: Google treats sites with SSL errors as less trustworthy, which can hurt rankings.
    • Compliance: Regulations such as PCI DSS require that certificates be valid and up‑to‑date.

    Core Concepts Behind SSL Certificate Validation

    Before diving into code, it helps to understand what makes a certificate valid:

    1. Expiration date (Not After): The date after which the certificate is no longer trusted.
    2. Validity period (Not Before): The start date of the certificate’s trust window.
    3. Certificate chain: A hierarchy of certificates that link your server certificate to a trusted root authority.
    4. Revocation status: Whether the certificate has been revoked before its natural expiration (via CRL or OCSP).

    For most monitoring needs, checking the Not After field is sufficient, and that’s exactly what our Python checker will focus on.

    Choosing the Right Python Libraries

    Python’s standard library already provides the tools you need:

    • ssl – Low‑level TLS handling and certificate parsing.
    • socket – Network communication to open a TLS handshake.
    • datetime – Date arithmetic for calculating remaining days.

    If you prefer a higher‑level approach, cryptography or pyOpenSSL can be added, but they are optional for a basic checker.

    Step‑by‑Step: Building the Checker

    1. Import Required Modules

    import ssl
    import socket
    from datetime import datetime
    

    2. Create a Function to Retrieve the Certificate

    def fetch_certificate(hostname: str, port: int = 443, timeout: int = 5):
        """Open a TLS connection and return the server certificate as a dict."""
        context = ssl.create_default_context()
        with socket.create_connection((hostname, port), timeout=timeout) as sock:
            with context.wrap_socket(sock, server_hostname=hostname) as ssock:
                cert = ssock.getpeercert()
        return cert
    

    3. Parse the Expiration Date

    The getpeercert() method returns a dictionary where the 'notAfter' key holds the expiration string.

    def get_expiration_date(cert: dict) -> datetime:
        """Convert the 'notAfter' field to a datetime object (UTC)."""
        exp_str = cert.get('notAfter')
        return datetime.strptime(exp_str, '%b %d %H:%M:%S %Y %Z')
    

    4. Calculate Days Remaining

    def days_until_expiration(expiration: datetime) -> int:
        """Return the number of whole days left before the cert expires."""
        delta = expiration - datetime.utcnow()
        return max(delta.days, 0)
    

    5. Combine Everything Into a User‑Friendly Report

    def check_ssl_expiration(hostname: str, port: int = 443):
        try:
            cert = fetch_certificate(hostname, port)
            exp_date = get_expiration_date(cert)
            days_left = days_until_expiration(exp_date)
    
            print(f"✅ {hostname}:{port}")
            print(f"   • Expiration Date : {exp_date.strftime('%Y-%m-%d %H:%M:%S UTC')}")
            print(f"   • Days Remaining  : {days_left}")
    
            if days_left < 30:
                print("   ⚠️  Warning: Certificate expires soon!")
            elif days_left == 0:
                print("   ❌  Error: Certificate has already expired!")
        except Exception as e:
            print(f"❗️  Could not retrieve certificate for {hostname}:{port} – {e}")
    

    6. Run the Checker

    Save the script as ssl_check.py and execute it with one or more hostnames:

    if __name__ == "__main__":
        hosts = ["example.com", "api.yourservice.com", "expired.badssl.com"]
        for h in hosts:
            check_ssl_expiration(h)
    

    Enhancing the Checker for Production Use

    The basic script works well for ad‑hoc checks, but real‑world monitoring usually demands a few extra capabilities.

    Automated Scheduling

    • Cron jobs (Linux/macOS): Add a line like 0 6 * * * /usr/bin/python3 /path/ssl_check.py >> /var/log/ssl_check.log 2>&1 to run daily at 6 AM.
    • Windows Task Scheduler: Create a task that triggers the script on a daily schedule.
    • Serverless options: Deploy the script as an AWS Lambda function triggered by EventBridge (cron) for zero‑maintenance execution.

    Alerting Mechanisms

    When a certificate is close to expiring, you’ll want to be notified automatically.

    • Send an email using smtplib or a service like SendGrid.
    • Post to Slack or Microsoft Teams via incoming webhooks.
    • Integrate with monitoring platforms (Datadog, Prometheus) by exposing a simple HTTP endpoint that returns JSON status.

    Supporting Multiple Ports and SNI

    Some services host multiple certificates on the same IP using different ports (e.g., 443 for HTTPS, 8443 for admin panels). The script already accepts a port argument, and server_hostname in wrap_socket enables SNI (Server Name Indication) for virtual hosts.

    Handling Revocation Checks (Advanced)

    While most organizations focus on expiration, checking revocation status adds an extra safety net. You can query the OCSP responder embedded in the certificate using the cryptography library, but this adds network latency and complexity. For most monitoring pipelines, expiration checks provide the best cost‑benefit ratio.

    Best Practices for SSL Certificate Management

    • Use short‑lived certificates: Let’s Encrypt issues 90‑day certificates, encouraging frequent renewal.
    • Automate renewal: Tools like certbot or acme.sh can automatically fetch and install new certificates.
    • Centralize monitoring: Keep all domain checks in a single configuration file (YAML/JSON) and loop through them in your script.
    • Version control your config: Store the list of monitored hosts in Git to track changes over time.
    • Document renewal procedures: Include step‑by‑step instructions for the team responsible for certificate updates.

    Sample Configuration File (YAML)

    # ssl_targets.yml
    targets:
      - host: example.com
        port: 443
      - host: api.yourservice.com
        port: 443
      - host: internal-app.local
        port: 8443
    

    Loading the YAML in Python

    import yaml
    
    def load_targets(file_path: str):
        with open(file_path, "r") as f:
            data = yaml.safe_load(f)
        return data.get("targets", [])
    
    if __name__ == "__main__":
        for entry in load_targets("ssl_targets.yml"):
            check_ssl_expiration(entry["host"], entry.get("port", 443))
    

    Testing Your Checker

    Before deploying, run the script against known good and bad certificates:

    • example.com – Valid, long‑lived certificate.
    • expired.badssl.com – Intentionally expired for testing error handling.
    • self-signed.badssl.com – Will raise a verification error unless you disable verification (not recommended for production).

    Observe the output and verify that warnings appear when days_left is below your threshold (commonly 30 days).

    Scaling Up: Monitoring Hundreds of Domains

    If you need to monitor a large inventory, consider these optimizations:

    • Threading or async I/O: Use concurrent.futures.ThreadPoolExecutor or asyncio with aiohttp to perform checks in parallel.
    • Cache results: Store the last‑checked timestamp in a lightweight SQLite DB to avoid duplicate checks within a short window.
    • Rate limiting: Respect remote servers by adding a small sleep (e.g., 0.2 s) between connections.

    Putting It All Together: A Production‑Ready Script