Handling Pagination in Web Scraping: Patterns That Don't Lose Data

Ask a scraper author where their dataset lost 30% of its records, and the answer is usually pagination: the loop stopped early, the site capped results, or infinite scroll hid everything past screen one. Pagination is where scrapers silently fail — here's how each pattern works and how to handle it.

Pattern 1: numbered pages

The classic ?page=2. The mistake is trusting the advertised page count. Loop until the site actually stops yielding:

page, items = 1, []
while True:
    batch = fetch_page(page)
    if not batch:          # empty page = the real end
        break
    items.extend(batch)
    page += 1

Watch for the result cap: many sites serve at most N pages regardless of matches (search engines and portals especially). If total results exceed the cap, shard the query — by category, price band, or geography — until each shard fits under it.

Pattern 2: cursor / token pagination

APIs return a next_cursor or continuation token with each response. Follow the token, never invent offsets — cursors tolerate inserts/deletes mid-crawl, which offset math does not:

cursor = None
while True:
    data = fetch(cursor=cursor)
    items.extend(data["results"])
    cursor = data.get("next_cursor")
    if not cursor:
        break

Pattern 3: infinite scroll & Load more

Infinite scroll is almost always a cursor API wearing a UI costume. Open the network tab, scroll, and copy the XHR request the page makes — then paginate that endpoint directly with HTTP instead of driving a browser. Only when the requests are cryptographically signed or heavily obfuscated is browser automation (Playwright scrolling) worth its cost.

Making any pattern robust

• Deduplicate by record key: pages shift while you crawl; unique IDs make overlaps harmless.
• Track totals: compare collected count against the site's advertised total; a large gap means a cap or an early stop.
• Checkpoint long crawls: persist the cursor/page so a crash resumes instead of restarting.
• Randomize politely: steady sub-second hammering across 500 pages is a block waiting to happen — pace requests.

FAQ