Posted inPython

Web Scraping with Python BeautifulSoup: Choose the Right Tool Before You Code

Python tutorial - IT technology blog
Python tutorial - IT technology blog

I got into automation through a small ~200-line script that collected product prices daily. That project has since ballooned to 2,000 lines, and one of the most painful lessons I learned was picking the wrong tool from the start — I wasted an entire week writing Selenium for a site that was completely server-side rendered, when requests + BeautifulSoup would have done the job in two hours.

Before writing a single line of code, you need to know that Python has three main approaches to web scraping:

  • requests + BeautifulSoup — Download static HTML and parse the DOM structure
  • Selenium / Playwright — Control a real browser and wait for JavaScript to finish rendering
  • Scrapy — A full-featured framework for large-scale crawling

Comparing the Three Approaches

These three tools aren’t competing with each other — each one solves a different type of problem. Picking the wrong one upfront, as I learned the hard way, means rewriting everything from scratch.

requests + BeautifulSoup

  • ✅ Up and running in 5 minutes, readable code, no GUI needed
  • ✅ Fast and lightweight — runs fine on a 512MB RAM VPS
  • ✅ Powerful enough for the majority of real-world use cases
  • ❌ Can’t handle JavaScript — sites built with React/Vue/Angular will return empty HTML or a skeleton
  • ❌ Can’t simulate clicks, scrolling, or form submissions

Selenium / Playwright

  • ✅ Handles fully JavaScript-rendered pages and SPAs
  • ✅ Can simulate any user interaction
  • ❌ Each browser instance consumes ~200MB RAM and runs 10–20x slower
  • ❌ Easier to detect — many sites now fingerprint headless browsers and immediately return a CAPTCHA or 403

Scrapy

  • ✅ Async, crawls thousands of pages in parallel
  • ✅ Built-in pipelines, middlewares, retry logic, CSV/JSON export
  • ❌ Steep learning curve — you need to understand Spiders, Items, and Pipelines before writing your first line
  • ❌ Complete overkill for a project scraping a few dozen pages

Decision Guide: When to Use What

Here’s the rule of thumb I apply before starting any scraping project:

  1. Open DevTools → View Page Source. If the data you need appears in the HTML source → use requests + BeautifulSoup.
  2. Open DevTools → Network tab → XHR/Fetch. If you spot an endpoint like /api/posts.json returning clean JSON → call that API directly, no need to parse HTML.
  3. Data only appears after scrolling, clicking, or logging in → use Playwright (preferred over Selenium for its modern API).
  4. Need to crawl hundreds or thousands of URLs concurrently with retry logic and pipelines → use Scrapy.

If this is your first scraping project, start with requests + BeautifulSoup. It’s ready in 5 minutes, runs on the cheapest VPS, and handles most problems that don’t require JS execution. The step-by-step guide below will get you started.

Step-by-Step Implementation with BeautifulSoup

Step 1: Installation

pip install requests beautifulsoup4 lxml

lxml is a faster parser than Python’s built-in html.parser. Install it from the start.

Step 2: Download HTML and Basic Parsing

import requests
from bs4 import BeautifulSoup

url = "https://example.com/articles"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()  # Raise an exception if status != 200

soup = BeautifulSoup(response.text, "lxml")
print(soup.title.text)  # Print the page title

Always set a User-Agent. Many servers return a 403 or serve completely different HTML when a request arrives without this header. timeout=10 prevents the script from hanging indefinitely when a server goes silent.

Step 3: Finding Elements with CSS Selectors and find()

BeautifulSoup offers two main ways to find elements:

# Method 1: find() and find_all() — search by tag, class, or id
title = soup.find("h1", class_="article-title")
all_links = soup.find_all("a", href=True)

# Method 2: select() — CSS selectors (familiar if you know CSS)
title = soup.select_one("h1.article-title")
all_links = soup.select("nav a[href]")

# Get text and attributes
print(title.get_text(strip=True))
print(all_links[0]["href"])

I personally prefer select() because CSS selectors are more flexible, especially when selecting by attribute or navigating complex parent-child relationships.

Step 4: Real-World Example — Scraping a List of Blog Posts

import requests
from bs4 import BeautifulSoup
import time

def scrape_blog_posts(url):
    headers = {"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)"}

    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "lxml")
    posts = []

    # Adjust selectors to match the target site's HTML structure
    for article in soup.select("article.post"):
        title_el = article.select_one("h2.entry-title a")
        date_el = article.select_one("time.entry-date")

        if title_el:
            posts.append({
                "title": title_el.get_text(strip=True),
                "url": title_el["href"],
                "date": date_el["datetime"] if date_el else None
            })

    return posts

# Crawl multiple pages — always add a delay between requests
all_posts = []
for page in range(1, 6):
    url = f"https://example.com/blog/page/{page}/"
    posts = scrape_blog_posts(url)
    all_posts.extend(posts)
    time.sleep(1.5)  # 1.5-second delay — respectful scraping

print(f"Scraped {len(all_posts)} posts")

time.sleep() is not optional — it’s the minimum courtesy when scraping. A server has no obligation to handle your requests at unlimited speed.

Step 5: Handling Edge Cases and Broken HTML

from urllib.parse import urljoin

# Safely handle missing elements — don't let the script crash
price_el = soup.select_one(".product-price")
price = price_el.get_text(strip=True) if price_el else "N/A"

# Get recursive text, stripping child HTML tags
description = soup.select_one(".description")
clean_text = description.get_text(separator=" ", strip=True)

# Convert relative URLs to absolute URLs
base_url = "https://example.com"
for link in soup.select("a[href]"):
    full_url = urljoin(base_url, link["href"])

# Fix encoding issues if characters appear garbled
response = requests.get(url, headers=headers)
response.encoding = "utf-8"  # Force UTF-8 instead of relying on auto-detection
soup = BeautifulSoup(response.text, "lxml")

Things I Wish I Knew Before Starting

After the project hit 2,000 lines and required a full rebuild, here are the things I wish I’d known from day one:

  1. Inspect the Network tab, not the Elements tab. Many sites load data through hidden JSON APIs. Open DevTools → Network → Filter XHR — sometimes you’ll immediately spot /api/v1/products.json, and calling that endpoint directly is far cleaner than parsing HTML.
  2. Save HTML locally when debugging. Download the HTML to a file and read from it during development — no need to hit the server on every test run, saving time and avoiding rate limits.
  3. Use soup.prettify() to understand the structure. Print out the formatted HTML before writing selectors — beats staring at a single-line, 10,000-character minified blob.
  4. Check robots.txt first. Visit example.com/robots.txt to see which paths the site allows crawling. It’s not a legal requirement, but it’s ethical practice — and some sites explicitly prohibit commercial scraping in their terms of service.
  5. Fragile selectors are real technical debt. A selector like div > div:nth-child(3) > span will break the moment the site changes its layout. Prefer selectors based on semantic class names like .product-title or data-testid attributes.

Next Steps

BeautifulSoup handles the vast majority of real-world scraping tasks — price monitoring, news aggregation, content backup, market research. The hard part isn’t the BeautifulSoup API itself. The hard part is correctly reading the target site’s HTML structure, then writing selectors that hold up when the layout changes three months later.

When should you switch tools? Need to crawl thousands of URLs in parallel with retry logic and export pipelines → try Scrapy. Hit a SPA with no hidden JSON API → switch to Playwright. Otherwise, don’t over-engineer it — BeautifulSoup solves about 80% of typical scraping problems.

Share:
Tags: