Reverse Proxy
A server that sits in front of one or more web servers, intercepting client requests to handle load balancing, SSL/TLS termination, edge caching, and anti-bot security.
While a forward proxy (the kind scrapers use) sits in front of clients to protect and conceal the client's identity, a reverse proxy sits in front of backend web servers to protect, accelerate, and manage access to the website.
When you navigate to major websites like Amazon, GitHub, or Cloudflare-protected targets, your computer never communicates directly with the company's internal application servers. Instead, all traffic terminates at an edge reverse proxy.
Forward Proxy vs Reverse Proxy Architecture (link to this section)
The fundamental difference lies in who is being represented:
- Forward Proxy (Client-Side):
[ Client / Scraper ] --> [ Forward Proxy ] --> [ Public Internet ] --> [ Website ]Acts on behalf of the client. The destination website only sees the forward proxy's IP address.
- Reverse Proxy (Server-Side):
[ Client / Scraper ] --> [ Public Internet ] --> [ Reverse Proxy / WAF ] --> [ Origin Servers ]Acts on behalf of the server. The client believes it is communicating directly with the website, but it is interacting with an intermediary edge gateway (such as Cloudflare, Akamai, Nginx, or HAProxy).
The Core Functions of a Modern Reverse Proxy (link to this section)
Modern enterprise reverse proxies perform several critical infrastructure duties:
- Anti-Bot & DDoS Protection (Web Application Firewall):
- Evaluates incoming client requests before they ever touch the origin server.
- Inspects the client's TLS JA3/JA4 fingerprint, HTTP/2 frame parameters, IP reputation, and request frequency.
- Issues JavaScript challenges (Cloudflare Turnstile, DataDome slider) if the client appears automated.
- SSL/TLS Termination:
- Decrypts incoming HTTPS traffic at the edge network close to the user, offloading heavy cryptographic computations from backend application servers.
- Load Balancing:
- Distributes incoming traffic across dozens or hundreds of backend server instances using algorithms like Round Robin, Least Connections, or IP Hashing.
- Edge Caching:
- Stores static HTML, images, and API responses in high-speed RAM, returning cached assets in milliseconds without querying origin databases.
Why Scrapers Must Understand Reverse Proxies (link to this section)
For web scraping engineers, the reverse proxy is the primary adversary:
- When your scraper receives an HTTP
403 Forbiddenor503 Service Unavailablecontaining a Cloudflare challenge page, the request was blocked by the edge reverse proxy, not the website's backend code. - Origin Server IP Bypasses: In some misconfigured web architectures, developers leave the origin web server directly reachable on a public IP while configuring the domain behind Cloudflare. If a scraper discovers the real origin server IP (via historical DNS records on SecurityTrails, SSL certificate transparency logs, or outbound email headers), it can query the origin directly, completely bypassing all Cloudflare bot protection.
In practice
- Cloudflare acting as a global Anycast reverse proxy, terminating millions of SSL handshakes at edge data centers and challenging automated scrapers with Turnstile puzzles.
- An engineering team configuring Nginx as a reverse proxy in front of 8 Node.js Docker containers to balance traffic and cache static assets.
- A scraper identifying the unprotected origin IP address of an e-commerce platform via Censys to bypass Cloudflare bot management entirely.
- Deploying HAProxy as an edge gateway to route API requests to different microservices based on URL path prefixes.
Common use cases
Frequently asked
A forward proxy sits in front of clients to disguise client identities and route outbound requests. A reverse proxy sits in front of origin servers to protect server infrastructure, balance inbound traffic, cache assets, and block bots.
Modern CDNs and WAFs (like Cloudflare, DataDome, and AWS WAF) act as reverse proxies that inspect incoming TCP/TLS handshakes and client headers. If the request exhibits bot signals (like an untrusted ASN or headless browser signature), the reverse proxy drops the connection immediately.
If a website's underlying server IP is publicly accessible and not restricted to Cloudflare's IP ranges, scrapers can query the origin server directly. This completely bypasses Cloudflare's bot challenges, WAF rules, and rate limits.
SSL termination means the reverse proxy handles the cryptographic HTTPS handshake and decrypts the traffic at the edge. The traffic between the reverse proxy and internal backend servers can then travel over high-speed unencrypted local private networks.