How to Build a Reliable Python Web Scraper

Most web scrapers work on day one and die within a month. The site changes a class name, an anti-bot wall appears, pagination shifts — and the script that "worked" quietly returns half a dataset. After building 300+ production scrapers, the pattern behind the reliable ones is consistent. This guide walks through it.

Start with the data source, not the HTML

Before parsing any HTML, open your browser's network tab and watch what the page actually loads. A large share of modern sites fetch their content from a JSON API — and scraping that API directly is faster, more stable, and immune to CSS redesigns. Only fall back to HTML parsing when no structured endpoint exists.

# The order of preference for any target:
# 1. Hidden JSON API (network tab)      → fastest, most stable
# 2. Embedded JSON (__NEXT_DATA__ etc.) → stable, one request
# 3. Server-rendered HTML               → parse with BeautifulSoup
# 4. JavaScript-rendered page           → last resort: browser automation

Choose an HTTP client that survives anti-bot walls

Plain requests announces itself as a script: its TLS handshake looks nothing like a browser's, and modern anti-bot systems fingerprint exactly that. For protected targets I default to curl_cffi, which impersonates real browser TLS fingerprints, with httpx for friendlier APIs:

from curl_cffi import requests

resp = requests.get(
    "https://example.com/listings",
    impersonate="chrome",          # browser-grade TLS fingerprint
    timeout=30,
)
resp.raise_for_status()

Full browser automation (Playwright, Selenium) is the heaviest tool in the box — use it when content genuinely renders client-side, not as a first resort. A browser costs 100× the resources of an HTTP request.

Design pagination and retries as first-class logic

The two places scrapers silently lose data are pagination edges and transient failures. Treat both explicitly:

• Pagination: loop until the site says stop (empty page, missing next-cursor) — never hardcode a page count.
• Retries: retry 429/5xx responses with exponential backoff and a cap; log every giving-up so gaps are visible, not silent.
• Idempotence: key every record (listing ID, URL) so re-runs deduplicate instead of duplicating.

Validate output like it's someone else's data

A scraper's job is a clean dataset, not a pile of responses. Before writing output: enforce a schema (required fields present, types correct), deduplicate on the record key, and fail loudly when the extraction rate drops — 200 items from a page that usually yields 1,000 means the site changed, and you want an alert, not a quiet half-file. Ship CSV or JSON with consistent field names your spreadsheet or database can ingest directly.

Respect the target

Reliable scraping is polite scraping: throttle request rates, cache what you've already fetched, collect only publicly available data, and honor the legal basics — terms of service, copyright, and data-protection rules like GDPR where they apply. Slow and steady also gets blocked less.

FAQ