Web Scraping with curl_cffi: Beating TLS Fingerprinting

You copied the exact headers from your browser's network tab. Same User-Agent, same cookies, same everything. The browser gets a 200; your Python script gets a 403. What's different? The handshake that happens before a single header is sent.

How TLS fingerprinting identifies your script

When a client opens an HTTPS connection, its TLS ClientHello advertises supported cipher suites, extensions, and curves — in a specific order. That combination is fingerprintable (the JA3/JA4 families of hashes), and it is remarkably stable per client: every Chrome installation of a given version looks the same, and Python's requests looks like exactly what it is — OpenSSL driven by a script. Anti-bot vendors classify the connection before your carefully-copied headers ever arrive.

curl_cffi: a browser handshake for Python

curl_cffi binds Python to curl-impersonate, a patched libcurl that reproduces real browsers' TLS and HTTP/2 fingerprints. The API is deliberately requests-like:

from curl_cffi import requests

resp = requests.get(
    "https://protected-site.com/data",
    impersonate="chrome",
    timeout=30,
)
print(resp.status_code)   # 200 where plain requests got 403

One argument — impersonate="chrome" — swaps the entire handshake identity. Sessions, proxies, and POSTs work the way you expect from requests.

Where it fits in the toolbox

Situation	Right tool
Open API or unprotected site	httpx / requests
TLS-fingerprinting anti-bot (403s on clean requests)	curl_cffi
Content rendered client-side by JavaScript	Playwright / Selenium
JS rendering and fingerprinting	Browser automation + stealth measures

The efficiency difference matters at scale: a curl_cffi request costs milliseconds and megabytes; a headless browser costs seconds and hundreds of megabytes. If the data is server-rendered or API-backed, impersonated HTTP is the production choice — it's the backbone of most of the 300+ Actors I run on Apify.

Practical tips from production

• Match the whole identity: send headers consistent with the browser you impersonate — a Chrome TLS fingerprint with a curl User-Agent is its own red flag.
• Keep sessions: reuse a Session so cookies and connection state persist like a real visitor's.
• Rotate deliberately: when you add proxies, rotate fingerprint and proxy together — a fingerprint that hops IPs every request looks wrong.
• Stay polite: fingerprint evasion is about not being misclassified as malicious while collecting public data at reasonable rates — it is not a license to hammer a site or bypass authorization.

FAQ