Python News Aggregator Web App Project

Looking for a hands‑on project that sharpens your Python skills while delivering real‑world value? A Python news aggregator web app is the perfect blend of data handling, API integration, and web development. In this guide you’ll learn how to design, build, and deploy a fully functional news aggregator that pulls headlines from multiple sources, filters content based on user preferences, and presents the results in a clean, responsive UI. Whether you’re a beginner eager to explore Flask or a seasoned developer interested in scaling with Django, the step‑by‑step walkthrough below will give you a solid foundation and plenty of room for customization.

What Is a News Aggregator and Why Build One in Python?

A news aggregator collects articles, headlines, and multimedia from a variety of feeds (RSS, JSON APIs, or custom scrapers) and consolidates them into a single, searchable interface. Building such an app in Python offers several advantages:

• Rich ecosystem: Libraries like requests, feedparser, and BeautifulSoup simplify data retrieval and parsing.
• Rapid prototyping: Frameworks such as Flask and Django let you spin up a web server with minimal boilerplate.
• Scalability: Python’s async capabilities (e.g., asyncio, aiohttp) make it easy to fetch dozens of feeds concurrently.
• SEO‑friendly output: Server‑side rendering ensures search engines can index your aggregated content, boosting organic traffic.

Key Features of a Modern News Aggregator

Core Functionality

• Multi‑source fetching (RSS, JSON, HTML scraping)
• Category and keyword filtering
• Automatic duplicate detection
• Article summarization (optional AI integration)
• User‑defined favorite sources
• Responsive UI with pagination and infinite scroll

Advanced Add‑Ons

1. OAuth login for personalized dashboards
2. Push notifications for breaking news
3. Full‑text search powered by Elasticsearch or Whoosh
4. Dockerized deployment for easy scaling
5. Analytics dashboard to track most‑read topics

Choosing the Right Tech Stack

Below is a recommended stack that balances simplicity with extensibility:

• Backend framework: Flask for lightweight projects or Django for built‑in admin, ORM, and authentication.
• HTTP client: requests (synchronous) or aiohttp (asynchronous).
• Feed parsing: feedparser for RSS/Atom, BeautifulSoup for HTML.
• Database: SQLite for prototypes, PostgreSQL for production.
• Front‑end: Bootstrap 5 + Jinja2 templates (Flask) or Django templates.
• Task queue (optional): Celery + Redis for periodic fetching.
• Containerization: Docker + Docker‑Compose.

Setting Up Your Development Environment

# Create a virtual environment
python -m venv venv
source venv/bin/activate   # Linux/macOS
venv\Scripts\activate      # Windows

# Install core dependencies
pip install flask requests feedparser beautifulsoup4
# Optional extras
pip install celery redis gunicorn

After installing the packages, initialize a Git repository and add a .gitignore file that excludes venv/, __pycache__/, and any secret keys.

Project Structure Overview

news_aggregator/
│
├─ app/
│   ├─ __init__.py          # Flask/Django app factory
│   ├─ routes.py            # View functions / URL patterns
│   ├─ models.py            # SQLAlchemy or Django ORM models
│   ├─ services/
│   │   ├─ fetcher.py       # Feed fetching logic
│   │   └─ parser.py        # Content parsing utilities
│   └─ templates/
│       └─ index.html
│
├─ static/
│   ├─ css/
│   └─ js/
│
├─ config.py                # Configuration (API keys, DB URI)
├─ requirements.txt
├─ docker-compose.yml
└─ run.py                   # Entry point

Step‑by‑Step Implementation

1. Initialize the Flask Application

# app/__init__.py
from flask import Flask
from .models import db

def create_app():
    app = Flask(__name__)
    app.config.from_object('config.Config')
    db.init_app(app)

    with app.app_context():
        from . import routes
        db.create_all()

    return app

2. Define the Database Model

# app/models.py
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime

db = SQLAlchemy()

class Article(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    source = db.Column(db.String(120), nullable=False)
    title = db.Column(db.String(250), nullable=False)
    url = db.Column(db.String(500), nullable=False, unique=True)
    summary = db.Column(db.Text)
    published = db.Column(db.DateTime, default=datetime.utcnow)
    category = db.Column(db.String(80))

3. Build the Feed Fetcher Service

# app/services/fetcher.py
import feedparser
import requests
from bs4 import BeautifulSoup
from datetime import datetime
from ..models import Article, db

FEED_URLS = [
    'https://rss.nytimes.com/services/xml/rss/nyt/Technology.xml',
    'https://feeds.bbci.co.uk/news/technology/rss.xml',
    # Add more sources here
]

def fetch_and_store():
    for url in FEED_URLS:
        feed = feedparser.parse(url)
        for entry in feed.entries:
            # Simple duplicate check
            if Article.query.filter_by(url=entry.link).first():
                continue

            # Optional: fetch full article for summary
            summary = _extract_summary(entry.link)

            article = Article(
                source=feed.feed.title,
                title=entry.title,
                url=entry.link,
                summary=summary,
                published=_parse_date(entry.get('published')),
                category=entry.get('tags')[0]['term'] if entry.get('tags') else 'General'
            )
            db.session.add(article)
    db.session.commit()

def _parse_date(date_str):
    try:
        return datetime(*feedparser._parse_date(date_str)[:6])
    except Exception:
        return datetime.utcnow()

def _extract_summary(url):
    try:
        resp = requests.get(url, timeout=5)
        soup = BeautifulSoup(resp.text, 'html.parser')
        # Grab first 
 as a naive summary
        paragraph = soup.find('p')
        return paragraph.get_text() if paragraph else ''
    except Exception:
        return ''

4. Schedule Periodic Fetching (Optional Celery Task)

# app/tasks.py
from celery import Celery
from .services.fetcher import fetch_and_store

celery = Celery('news_aggregator', broker='redis://redis:6379/0')

@celery.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    # Run every 15 minutes
    sender.add_periodic_task(900.0, fetch_and_store.s())

@celery.task
def fetch_and_store_task():
    fetch_and_store()

5. Create the Main Route and Template

# app/routes.py
from flask import render_template, request
from . import create_app
from .models import Article

app = create_app()

@app.route('/')
def index():
    page = request.args.get('page', 1, type=int)
    per_page = 20
    articles = Article.query.order_by(Article.published.desc()).paginate(page, per_page, False)
    return render_template('index.html', articles=articles)

Sample templates/index.html using Bootstrap:

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <title>Python News Aggregator</title>

    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css" rel="stylesheet">

</head>

<body class="bg-light">

    <nav class="navbar navbar-expand-lg navbar-dark bg-primary mb-4">

        <div class="container-fluid">

            <a class="navbar-brand" href="/">Python News Aggregator</a>

        </div>

    </nav>

    <div class="container">

        <div class="row">

            <div class="col-md-12">

                <h2 class="mb-4">Latest Headlines</h2>

                <ul class="list-group">

                    {% for article in articles.items %}

                    <