The 2 AM OOM Nightmare
A few months ago, I was tasked with crawling 500,000 products from a major e-commerce platform for a market report. At the time, I was quite overconfident: “It’s just data collection; using requests and BeautifulSoup should be fine.”
Exactly at 2 AM, the server started throwing red alerts. The script had been running for 3 hours before freezing. RAM spiked to 4GB and then crashed (Out of Memory). Worse, the server’s IP had been blacklisted by the partner after sending too many rapid-fire requests. The project started with just 200 lines of code, but after patching in retries, proxies, and multi-threading, it ballooned to nearly 2,000 lines and still ran inconsistently.
That was when I realized a hard truth. To build a Large-scale Web Crawler, you can’t rely on disconnected libraries. You need a real framework. Scrapy was the game-changer.
Why BeautifulSoup Often Falls Short in Large Projects
The core issue is that requests operates synchronously. It sends a request, waits for a response, and then moves on. For 500,000 pages, if each takes an average of 1 second, you’d be waiting for 138 hours straight. This is unthinkable in a production environment.
Writing custom code to handle errors, automatic retries, or saving data to a database without blocking the execution flow is a nightmare. Scrapy solves this completely with its asynchronous architecture based on the Twisted framework. Instead of waiting, Scrapy fires off dozens of requests simultaneously, maximizing bandwidth optimization.
Three Crawler Approaches: Where Do You Stand?
In the world of data scraping, developers usually choose one of these three paths:
- Using Threading/Multiprocessing: Combined with
requests. This method is extremely difficult to manage in terms of state. You can easily overflow memory if thread counts aren’t strictly controlled. - Using Selenium/Playwright: A “premium” option because it simulates a browser like a real human. However, it’s a massive resource hog. It should only be used when a website relies on complex JavaScript that you can’t extract via APIs.
- Using Scrapy: The ultimate performance king. It allows for hundreds of parallel requests while maintaining stable RAM consumption.
Implementing Scrapy: From Zero to Production System
First, let’s install Scrapy. I recommend always using a virtual environment (virtualenv) to keep your project clean.
pip install scrapy1. Standard Project Structure
Don’t cram all your code into a single file. Scrapy forces you to work modularly. This organization will save you as the project expands later.
scrapy startproject my_crawler
cd my_crawler
scrapy genspider ecommerce spider_site.com2. Defining Items (Data Model)
Instead of using messy dictionaries, clearly declare your data fields in items.py. It acts as a Schema for your data.
import scrapy
class ProductItem(scrapy.Item):
name = scrapy.Field()
price = scrapy.Field()
url = scrapy.Field()
sku = scrapy.Field()3. Writing the Spider – The Controlling Brain
This is where the extraction logic happens. Scrapy supports CSS Selectors and XPath, which are far more powerful than BeautifulSoup’s find(). Here’s how you extract data and handle automatic pagination.
import scrapy
from my_crawler.items import ProductItem
class EcommerceSpider(scrapy.Spider):
name = "ecommerce"
start_urls = ['https://example-shop.com/products']
def parse(self, response):
for product in response.css('div.product-item'):
item = ProductItem()
item['name'] = product.css('h2::text').get()
item['price'] = product.css('span.price::text').get()
item['url'] = response.urljoin(product.css('a::attr(href)').get())
yield item
next_page = response.css('a.next::attr(href)').get()
if next_page:
yield response.follow(next_page, self.parse)Tips for Operating at Scale
For production use, default configurations aren’t enough. Here are some adjustments I’ve refined after many struggles on the server.
Item Pipelines: The Clean Data Filter
Crawled data often contains noise like extra whitespace or strange characters. Don’t handle this inside the Spider. Push it to pipelines.py to keep your extraction code clean.
class CleanDataPipeline:
def process_item(self, item, spider):
if item['price']:
# Convert "$1,200" to 1200.0
item['price'] = float(item['price'].replace('$', '').replace(',', '').strip())
return itemMiddlewares: Techniques to Avoid Getting Blocked
Sending 1,000 requests per minute from a single IP is the fastest way to get banned. Open settings.py and configure User-Agent rotation or use a Proxy Service.
# settings.py
DOWNLOAD_DELAY = 0.5 # Delay between requests to avoid suspicion
CONCURRENT_REQUESTS = 32 # Number of concurrent requests
# Automatically rotate User-Agent to simulate different browsers
DOWNLOADER_MIDDLEWARES = {
'scrapy.middlewares.useragent.UserAgentMiddleware': None,
'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}Lessons in Resource Management: Don’t Overload RAM
A common mistake is gathering all data in memory before writing it to a file. With 1 million rows, your script will definitely crash. Scrapy has a very smart Feed Exports mechanism.
It allows you to stream data directly to a JSON or CSV file line by line:
scrapy crawl ecommerce -o data.jsonlPro tip: Use the .jsonl (JSON Lines) format. If the server suddenly loses power, the data already crawled remains intact, unlike traditional JSON which would result in a corrupted file.
Conclusion
Scrapy isn’t just a library; it’s a systematic mindset. It forces you to write clean, modular code and always prioritize performance. If you only need to scrape a few small websites, BeautifulSoup is still great. But if your goal is a stable system running 24/7 and processing millions of records, choose Scrapy. Investing the time to learn it now will help you sleep much better at 2 AM.
