Web scraping typically involves downloading many web pages and parsing the data from the HTML. However, there is often a lot of waiting time for sending requests and receiving the responses from the server. This waiting can be a bottleneck in terms of scraping performance. Asynchronous scraping aims to speed up scraping by running multiple scrape tasks concurrently and eliminating wait time.
In this comprehensive guide, we cover everything you need to know about asynchronous web scraping, including key concepts, tools, tips and techniques for developing highly performant scrapers.
Synchronous vs. Asynchronous Web Scraping
The main difference lies in how the code wait for the responses from the web server:
Synchronous Web Scraping:
- Executes scrape tasks sequentially one by one
- Has to wait for each request to finish before next one starts
- Leads to a lot of idle waiting time
Asynchronous Web Scraping:
- Allows running multiple scraping tasks concurrently
- Eliminates waiting by asynchronously handling requests and responses
- Results in much faster scraping due to parallel execution
Let's look at sample Python code to demonstrate:
# Synchronous scraping
import requests
from time import time
start = time()
urls = [
"https://www.example.com",
"https://www.sample.com",
"https://www.demo.net"
]
for url in urls:
response = requests.get(url)
data = response.text
# parse data
print(f"Scraped {len(urls)} pages in {time() - start:.2f} seconds")
# Asynchronous scraping
import asyncio
import aiohttp
from time import time
async def fetch(session, url):
async with session.get(url) as response:
data = await response.text()
# parse data
return data
async def main():
start = time()
urls = [
"https://www.example.com",
"https://www.sample.com",
"https://www.demo.net"
]
async with aiohttp.ClientSession() as session:
tasks = []
for url in urls:
task = asyncio.create_task(fetch(session, url))
tasks.append(task)
data = await asyncio.gather(*tasks)
print(f"Scraped {len(urls)} pages in {time() - start:.2f} seconds")
asyncio.run(main())The synchronous version has to wait for each request to finish before making the next request. The asynchronous implementation allows all requests to be made concurrently eliminating wait times.
For just 3 links, asynchronous scraping might only be slightly faster. But for large datasets, it can result in exponentially faster run times.
How Asynchronous Web Scraping Works
Asynchronous scraping works by leveraging asynchronous programming techniques:
Event Loop
An event loop runs asynchronously allowing managing multiple tasks and switching between them when one is waiting for I/O. This is more efficient than sequential execution.
Non-Blocking I/O
I/O requests are made non-blocking by using async network libraries. This allows the event loop to process other things while waiting.
Parallel Execution
Tasks that are independent can execute in parallel. Web scraping with its many network calls is highly parallelizable leading to performance gains.
In Python, libraries like asyncio, trio, twisted provide frameworks and tools to write asynchronous scraping code leveraging these techniques.
Tools for Asynchronous Web Scraping in Python
Some popular Python libraries for asynchronous scraping:
aiohttp
A powerful HTTP client/server library for asyncio. Great for making requests concurrently.
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
return await response.json()
async def main():
async with aiohttp.ClientSession() as session:
urls = [
"https://api.github.com/events",
"https://api.github.com/repos",
"https://api.github.com/users"
]
tasks = []
for url in urls:
task = asyncio.ensure_future(fetch(session, url))
tasks.append(task)
data = await asyncio.gather(*tasks)
print(data)
asyncio.run(main())httpx
A next generation HTTP client with both sync and async support. Easy to migrate existing code to async.
import httpx
urls = [
"https://www.example.com",
"https://www.sample.com",
"https://www.demo.net"
]
async with httpx.AsyncClient() as client:
tasks = [client.get(url) for url in urls]
results = await asyncio.gather(*tasks)
print(results)Trio
A friendly async library for concurrency and I/O. Focuses on usability and ergonomics.
import trio
async def fetch(url):
async with trio.open_url(url) as resp:
return await resp.text()
async def main():
urls = [
"https://www.example.com",
"https://www.sample.com",
"https://www.demo.net"
]
async with trio.open_nursery() as nursery:
for url in urls:
nursery.start_soon(fetch, url)
trio.run(main)More Tips for Effective Asynchronous Web Scraping
- Handle errors correctly using exception handling
- Tune concurrency based on bandwidth, proxies etc.
- Use proxies and rotation to avoid getting blocked
- Ensure network calls are properly awaited
- Async process item pipelines to parse data concurrently
- Visualize performance to compare sync vs async
- Stress test scrapers to identify bottlenecks
How Scraping Frameworks Handle Asynchrony
Many Python scraping frameworks have inbuilt support for asynchronous scraping:
- Scrapy: Has an
asyncioreactor allowing spiders to leverage async/await. Released in Scrapy 2.0. - Scrapyd: Has an experimental
-a io_loop=asynciooption to enable asyncio event loop. - Crawlera: Scraper API that handles asynchronous requests efficiently in the cloud backend.
- Portia: Visual scraping tool by Scrapinghub with default asyncio reactor for async crawling.
Conclusion
Asynchronous programming techniques can yield large speed and efficiency improvements in web scraping by running multiple scraping tasks concurrently. Python has fantastic async frameworks like asyncio, trio and powerful network libraries like aiohttp and httpx perfect for asynchronous scraping.
By embracing asynchronous scraping methodologies, you can develop highly performant scrapers to retrieve data faster. Concurrency helps eliminate bottlenecks related to I/O wait times.
Many popular scraping tools are also integrating asyncio making it easier to write async scrapers. As async coding gains more adoption, the future of high-performance web scraping is asynchronous!
