Web scraping can be an extremely useful tool for extracting data from websites. However, many websites don't like scrapers accessing their data and will try to block them. Getting blocked while scraping can be frustrating, but there are ways to avoid it.
In this comprehensive guide, we'll explore the most common ways scrapers get blocked and how you can avoid these protections to scrape without getting blocked.
Common Ways Sites Detect and Block Scrapers
Websites and webmasters have good reason to be protective of their data. Scraping can impose heavy loads on servers and go against their terms of use. But that doesn't deter everyone, so sites employ a variety of technical methods to identify and block scrapers:
1. Analyzing HTTP Request Headers
The first and most basic way sites look for scrapers is by analyzing the HTTP request headers. Headers provide metadata about each request, including information like:
- User-Agent: The name and version of the browser or application making the request.
- Accept: The content types the client can accept (HTML, JSON, etc).
- Accept-Language: The preferred language of the user.
- Cache-Control: Whether the client caches responses and how.
When a real browser like Chrome or Firefox visits a website, it sends headers that identify it properly as Chrome or Firefox on a particular platform. But scrapers don't always spoof their headers well!
If the User-Agent contains the name of a scraping library instead of a real browser, that's a dead giveaway the visitor is a bot. Or if the Accept headers ask for content types a normal browser wouldn't, the site can flag the request as suspicious.
According to PerimeterX in 2019, nearly 25% of all web traffic came from scrapers. But the vast majority – around 90% – didn't even modify their User-Agents from the default values of popular scraping tools and libraries, making them easy to detect and block.
To blend in, scrapers need to send headers that are indistinguishable from a real web browser. More details on how later in this guide.
2. Blocking Scrapers by Their IP Addresses
Websites can identify and block scrapers simply based on their IP addresses. The main ways this happens:
- Traffic Volume: Scrapers make requests much more rapidly than humans. If a site detects too many requests coming from a single IP address in a short period, it will assume it's a bot and block the IP.
- Bad Proxies: To avoid getting blocked by volume, scrapers will often use proxy services to mask their IPs. But low-quality proxies, like free or public proxies, are well-known to sites and frequently get blocked.
- Previous Abuses: IP addresses that have been used for scraping or attacks in the past get logged and blocked by sites. Even rotated proxy IPs can get blocked this way if the proxy pool has a history of abuse.
According to Imperva, over 60% of web scraping traffic comes from proxy services to try to avoid IP blocks. But poorly managed proxies often do more harm than good when scraping. Later we'll cover how to properly leverage proxies to scrape without getting IP banned.
3. TLS & SSL Handshake Analysis
On secure HTTPS websites, the TLS handshake used to establish encrypted connections can be used to detect and block scrapers as well. TLS handshakes have some variability – different browsers and HTTP clients perform them slightly differently. These small differences in how TLS sessions are initialized can fingerprint clients.
A popular technique is JA3 fingerprinting. JA3 profiles clients based on specific fields in the TLS handshake like supported cipher suites, SSL versions, and extensions. Web scrapers using common HTTP libraries end up having very similar JA3 fingerprints, allowing them to be identified even if they try to spoof headers and IP addresses.
According to a 2020 study by Akamai, over 30% of sites now use JA3 or similar techniques to identify web scrapers and other unwanted bots. Like request headers, the TLS handshake needs to be carefully spoofed.
4. Javascript Based Fingerprinting & Detection
Modern sites rely heavily on Javascript, which can be used to fingerprint scrapers. Browser automation tools like Selenium and Puppeteer are vulnerable because they expose interfaces like navigator.webdriver that betray the presence of automation.
Javascript can also extract detailed browser and environment info like screen size, fonts, time zone, etc. These can be used to assemble unique fingerprints.
5. CAPTCHAs and Other Human Challenges
The most direct way sites block scrapers is by requiring users to pass a Turing test before accessing data, commonly implemented as a CAPTCHA. CAPTCHAs require the user to perform visual tasks like identifying distorted text or objects that are easy for humans but extremely difficult for bots without advanced computer vision capabilities.
Many sites will initially allow access and only show CAPTCHAs if they suspect the visitor is a bot based on fingerprinting and usage patterns. This avoids impacting most real users. Newer options like reCAPTCHA v3 don't even require visual challenges. They run analytics in the background and only block visitors who fail security checks.
6. Monitoring Traffic Patterns
Sites can look for usage patterns that differ from normal human behavior, like an extremely high frequency of requests, too many requests for one area of the site, or always following the same navigation paths.
Scrapers are often repetitive whereas humans show more variety. Analytics software makes monitoring traffic easy.
7. Dedicated Anti-Scraping Services
In recent years, there has been huge growth in services dedicated specifically to detecting and blocking web scrapers and other unwanted bots. Some major players include:
- Imperva Incapsula – Used by Glassdoor, Lenovo, Trivago
- PerimeterX – Used by Thomson Reuters, Major League Baseball
- DataDome – Used by LebonCoin, Deezer, ClubMed
- Cloudflare – Provides bot management as part of their WAF
- Akamai – Bot Manager integrates with their CDN and security products
These companies maintain databases of known scrapers and proxies to block. More importantly, they apply machine learning algorithms trained to detect patterns like unusual traffic volumes and speeds that suggest bots.
With APIs and fingerprints shared across customer sites, it becomes very hard for scrapers to go undetected when major sites use these anti-bot services.
How to Avoid Getting Blocked While Scraping
Now that we've thoroughly explored how websites identify and block scrapers let's look at countermeasures you can apply to scrape effectively without getting blocked:
Use Proxy Rotation Services
The first essential technique for avoiding blocks is using proxy rotation services. Proxies work by routing your traffic through intermediary servers, masking your real IP address. With enough proxies from diverse sources being rotated rapidly, blocks by IP address can be largely avoided. Some top proxy providers include:
- Bright Data – Over 72 million IPs. Used by Amazon, Google, and Microsoft.
- Soax – 8.5+ million IPs with detailed targeting options.
- Smartproxy – 55+ million IPs, integrating with major scraping tools.
- Proxy-Seller – Cheaper residential proxies starting at $10/GB.
The exact approach to proxy rotation does matter:
- Use multiple proxy providers, not just one. This adds diversity to IP sources.
- Prioritize residential, not datacenter IPs. Residential IPs are less likely to be already blacklisted.
- Rotate proxies frequently. Never scrape with the same IP twice in a row. Use each for 1 request max.
- Avoid poor-quality public/free proxies. They are well-known and frequently blocked.
- Acquire proxies matching your targets. Getting US IPs for scraping US sites is ideal.
With enough quality proxies being swapped for each request, blocks become much harder.
Randomize and Mimic Browser Headers
We need to send headers that perfectly mimic a real browser. This means randomizing fields like:
- User-Agent: Rotate between common values from browsers like Chrome, Firefox, and Safari. Always match the platform too.
- Accept-Language: Vary languages between en-US, en-GB, fr, etc.
- Cache-Control: Use common browser caching directives.
And, importantly, ensure the order and formatting of headers match real browsers. Unusual ordering or separators will stand out.
Tools like Smartproxy and ScrapeBox simplify header rotation by providing browser-like proxy requests out of the box. But it is good to understand how to randomize headers properly in any scraper. Take care also to keep headers consistent with other settings like proxies and user agents. A Russian IP but English-only headers could trigger suspicion.
Handle TLS Handshakes Carefully
To avoid TLS handshake fingerprinting, we need to ensure the handshake aligns with major browsers:
- Use a robust mainstream HTTP library like Python Requests rather than an obscure one.
- Configure your library to handle TLS 1.2 or 1.3 properly and offer strong cipher suites compatible with modern browsers. Disable old TLS versions.
- Set TLS options like extensions, cipher ordering, and signature algorithms to browser-like values when possible.
- Consider using a proxy service that proxies TLS for you to generate browser-like handshakes naturally.
Basically, avoid standing out – blend in with ordinary browser handshakes as much as possible.
Browser Automation Configuration
If scraping interactive sites built with modern JavaScript, you will likely need to use browser automation rather than simple requests. This opens up additional anti-bot challenges, but the popular tools Puppeteer, Playwright, and Selenium provide options to evade detection:
- Spoof all browser fingerprints like User-Agent, languages, and touch support to match real browsers.
- Disable headless mode and run a full browser for more realistic interactions.
- Disable webdriver flags that expose automation like
navigator.webdriver. - Randomize fingerprintable values like viewport size, time zone, and navigator platform.
- Emulate human patterns like variable typing speed, scrolling, and mouse movements.
Configuring browsers to act human takes work but prevents red flags that expose automation. Fooling reCAPTCHA in particular, requires fine-tuned browsers. Tools like Puppeteer Extra, Selenium Stealth, and Playwright Extra can help automate spoofing.
Use Realistic Crawl Patterns
Even with all the right headers, IPs, and configurations – bots stand out due to their systematic, predictable crawling patterns. Some ways to introduce randomness and imperfection:
- Throttle requests to variable human speeds – don't crawl as fast as possible.
- Vary the order pages are accessed rather than sequential paths.
- Scroll to randomized positions and hover/click page elements like humans.
- Refresh pages and revisit the ones you've already crawled.
- Occasionally duplicate or skip pages at random, like a distracted user.
- Crawl across days/times, not all at once. Break patterns.
Analytics systems look for unusual traffic patterns in the aggregate. The key is ensuring your scraper acts imperfectly lifelike, not robotic.
Avoiding Honeypots and Traps
Keep an eye out for intentional traps aimed at scrapers and avoid them:
- Links to nowhere or nonsense URLs – scrapers follow anything.
- Forms with no clear purpose – bots fill anything.
- Hidden text/code only bots will extract – human eyes won't see it.
- Fake ads and content – humans know to ignore.
Sites continually experiment with novel honeypots. When in doubt, avoid anything that seems like a bot trap with no legitimate purpose.
Outsource CAPTCHAs and reCAPTCHAs
When faced with CAPTCHAs and reCAPTCHAs that block automated scraping, specialized services can solve them:
- 2Captcha – 1 million+ CAPTCHAs solved daily starting at $2/1000 CAPTCHAs.
- AntiCaptcha – 50 million+ CAPTCHAs solved monthly starting at $2.99/1000.
- DeathByCaptcha – Specializes in solving complex reCAPTCHAs for automation.
These systems employ humans around the world to manually solve CAPTCHAs and input the answers, providing an API for your bot to submit and retrieve solutions in real time. While an imperfect solution, it may be necessary to proceed with certain difficult sites. Just try not to trigger too many CAPTCHAs to avoid extra costs and scrutiny.
Leverage Scraping-as-a-Service Platforms
If you want to entirely outsource blocking avoidance, consider scraper-as-a-service platforms:
- ScrapingBee – Browser API and proxies to simplify scraping. Handling 1B+ requests monthly.
- ProxyCrawl – Rotating proxies and browsers to avoid blocks.
- ScraperAPI – AI optimized and trained per site to avoid blocks. 70M+ successful scrapes.
- Apify – Headless browser scraping with built-in proxy rotation.
These tools handle proxies, browsers, captchas, and human patterns behind the scenes so you can focus on data extraction. Their frameworks are optimized specifically for each site and trained on human patterns to avoid blocks in ways hard to replicate yourself.
Of course, these services aren't free, with pricing starting around $49/month. But for many, the cost may be worth avoiding the anti-scraping headache.
Ethical Considerations for Scrapers
While this guide aims to help you scrape without getting blocked from a technical standpoint, it's also important to consider the ethics of scraping:
- Respect robots.txt rules – Avoid scraping sites that explicitly prohibit it in robots.txt.
- Don't overtax servers – Use throttling/delays and proxy rotation to minimize server load.
- Check terms of use – Many sites prohibit scraping in their TOS – consider if it's worth the legal risk.
- Consider data sensitivity – Be careful not to collect or mishandle private/restricted info.
- Use scrapped data responsibly – Don't use it for harassment, exploitation, or other harmful purposes.
- Make Opt-outs accessible – Provide easy ways for sites to opt out if they contact you.
- Be transparent – When possible, identify yourself rather than hide scraping activities.
Make sure you scrape ethically and legally. While sites may technically be able to block you, that alone doesn't make scraping right without considering the broader context and impact.
Conclusion
Scraping without getting blocked is tricky but possible. The keys are using proxies, mimicking browsers perfectly, and introducing randomness and human-like behavior. Take time to understand how a site tries to detect bots, then engineer around their specific defenses. No universal formula works on all sites.
With the right tools and techniques, you can scrape effectively without getting shut out by anti-scraping systems. Just remember to respect sites' wishes if they prohibit scraping.
