Web Scraping with Playwright and Python

Web scraping is the process of extracting data from websites automatically. It allows you to collect large amounts of data that would be difficult or impossible to gather manually. Python is one of the most popular languages for web scraping due to its simplicity and vast ecosystem of scraping tools. Playwright is a new browser automation library that makes it easy to scrape dynamic websites built with JavaScript.

In this comprehensive guide, you'll learn how to use Playwright with Python to scrape complex web pages that traditional scraping libraries struggle with.

Why Playwright for Web Scraping?

Modern websites are highly dynamic – content loads asynchronously via AJAX requests, infinite scrolling fetches new data as you reach the bottom, DOM elements update without page reloads. Traditional Python scraping libraries like Beautiful Soup and Scrapy are designed for static content. They struggle with scraping dynamic pages that require execution of JavaScript code.

Playwright controls an actual browser like Chrome and Firefox. It executes full avaScript and perfectly renders web pages like a real user. This makes it ideal for scraping dynamic websites. Other advantages of using Playwright for web scraping:

  • Works across browsers (Chrome, Firefox, WebKit)
  • Supports multiple languages (Python, JavaScript, C#, Java)
  • Interact with pages by clicking buttons, filling forms
  • Mock geo-location, device types, throttling
  • Stealth mode to avoid bot detection
  • Network request interception

Overall, Playwright provides a very robust browser automation solution for Python web scraping.

Installation

Install Playwright for Python with:

pip install playwright

This will download browser binaries for Chromium, Firefox, and WebKit. We only need Chromium for this tutorial. Install a sync version that allows easier scraping scripts:

pip install playwright-sync

Optionally install IPython for experimenting in an interactive shell:

pip install ipython

Scraping Basics

The basic steps for scraping with Playwright:

  1. Launch a browser
  2. Navigate to URL
  3. Wait for page load
  4. Extract data
  5. Rinse and repeat

Let's go through a simple example to scrape the Hacker News homepage.

Launch Browser

Launch a headless Chromium browser:

from playwright.sync_api import sync_playwright

browser = sync_playwright().start().chromium.launch(headless=True)

Headless mode runs the browser in the background without opening a GUI window.

Create Page

Open a new browser page/tab to navigate:

page = browser.new_page()

We'll execute all our scraping code in the context of this page.

Navigate to URL

Use page.goto() to navigate to a URL:

page.goto("https://news.google.com/")

This loads www.news.google.com on our browser page.

Wait for Page Load

After navigation, we need to wait for the page to fully load before scraping:

page.wait_for_load_state('networkidle') # wait for AJAX/XHR requests

Other wait options:

  • load¬†– initial HTML document
  • domcontentloaded¬†– HTML & DOM parsed
  • networkidle¬†– no network connections for 500ms

Extract Data

We can now extract page data using Playwright's query selectors:

for link in page.query_selector_all('.storylink'):
  title = link.text_content()
  url = link.get_attribute('href')
  print(title, url)

This prints all the story titles and URLs into the terminal. The full code so far:

from playwright.sync_api import sync_playwright

browser = sync_playwright().start().chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://news.google.com/")
page.wait_for_load_state('networkidle') 

for link in page.query_selector_all('.storylink'):
  title = link.text_content()
  url = link.get_attribute('href')
  print(title, url)

This covers the scraping basics with Playwright – launch the browser, navigate to URLs, wait for load, and extract data. Pretty straightforward! Next, let's look at more advanced scraping capabilities.

Scraping Dynamic Content

A key strength of Playwright is interacting with dynamic JavaScript-heavy sites, like those using infinite scroll, dropdowns, popups, etc. Let's scrape an infinite scroll page – Unsplash photo feed. As you scroll down it lazily loads more images via AJAX requests. Playwright can automate scrolling to scrape all items.

Infinite Scroll Scraping

The main steps:

  1. Scroll to bottom of page
  2. Wait for new photos to load
  3. Extract updated photo data
  4. Repeat until no more photos

Here's how it looks in Python:

from playwright.sync_api import sync_playwright
import time 

browser = sync_playwright().start().chromium.launch()
page = browser.new_page()
page.goto("https://unsplash.com")

scroll_delay = 2

while True:
  print("Scrolling...")

  page.evaluate("""
    window.scrollBy(0, window.innerHeight);
  """)

  time.sleep(scroll_delay)

  new_photos = page.query_selector_all('.photo')  
  print(f"Found {len(new_photos)} photos")

  if len(new_photos) == 0:
    break

print("Done!")

The main logic:

  • Use JavaScript¬†window.scrollBy()¬†to scroll down the length of current viewport
  • Wait few seconds for lazy loading of images
  • Check if any new photos were loaded using¬†query_selector_all()
  • If not, we reached the end and can break the loop

Some points:

  • page.evaluate()¬†runs JS in the browser context, so we can access the webpage¬†window
  • Scrolling to bottom triggers the lazy loading mechanism
  • After scroll, wait briefly for loading using¬†time.sleep()
  • Check if new photos appeared with¬†query_selector_all()
  • If no new photos, we scrolled to the end

This allows scraping infinitely long pages by automating the scrolling!

Handling Dropdowns

Another common dynamic element is dropdown selections. Let's see how to deal with those using the Airbnb site. Here's how to open the guests dropdown and print the options:

page.goto("https://www.airbnb.com")

page.click("._1k463rt") # click guests field

options = page.query_selector_all("._gig1e7")

for option in options:
  print(option.text_content())

The steps:

  • Use¬†page.click()¬†to open the dropdown
  • Fetch all the¬†._gig1e7¬†options
  • Print text content of each option

This allows interaction with dropdowns and selecting options programmatically.

Scraping Iframes

Some sites load content in iframes that are embedded as separate documents. To scrape iframes, we first need to switch page context into the iframe:

page.goto("https://somesite.com") 

# wait for iframe to load
iframe = page.wait_for_selector("iframe")

# switch context into iframe
frame = iframe.content_frame()

# now can extract data from inside frame
texts = frame.query_selector_all("p")

The key points:

  • Wait for iframe to load using¬†wait_for_selector()
  • Get the iframe content frame with¬†content_frame()
  • Switch context to interact inside the iframe
  • Query selectors now run within iframe document

This allows scraping data from complex pages with nested iframe documents.

Dealing with Bot Detection

A downside of automation is that websites can detect scraping bots and block them. Playwright provides ways to mimic human behavior and be less detectable when scraping.

Stealth Mode

Enable stealth mode to spoof or disable automation signals that can get detected:

browser = sync_playwright().start().chromium.launch(headless=False)
context = browser.new_context(
  viewport={"width": 1920, "height": 1080},
  color_scheme="light", # or "dark"
  ignore_https_errors=True, # skip cert checks
  stealth=True,
)

With stealth mode Playwright will:

  • Mask its agent / Chrome version
  • Override device descriptions
  • Disable extensions, plugins, web notifications
  • Prevent WebDriver flags and APIs
  • Spoof/block sensors like motion, touch, etc

This makes Playwright automation blend in more like a real user browser.

Slowing Down Interactions

Bots can be detected by fast inhuman interactions speeds. Add delays to mimic human level speeds:

from time import sleep

sleep(1) # delay between actions  

page.click("button", delay=500) # delay after click

page.type("input", "text", delay=100) # delay between key presses
  • Add sleeps between scraping steps
  • Use delay args for click, type to slow them down
  • Vary random delays to add natural human chaos

Slowing down makes scraping behavior appear more life-like.

Scrolling and Mouse Movement

Smooth natural scrolling and mouse movements also help avoid bot patterns:

# smooth scroll 
page.evaluate("""
  window.scrollTo({
    top: document.body.scrollHeight,
    left: 0, 
    behavior: 'smooth'
  });
""")

# humanized mouse movement
page.mouse.move(200, 300, steps=100)
  • Use¬†behavior: smooth¬†for natural scrolling
  • Slow down mouse movements over multiple micro-steps

Browser Profiles

Having many scrapers share one browser profile is suspicious. Create a new user profile for each page instance:

browser = sync_playwright().start().chromium.launch_persistent_context(
   user_data_dir="/tmp/new_profile", # unique temp dir
)

page = browser.new_page()
  • Set a custom¬†user_data_dir¬†for new profile
  • Launch persistent context to reuse profile
  • Now each page instance has own profile

Dedicated profiles mimic real users better.

Proxy Servers & Residential IPs

Using proxy servers and residential IP addresses helps avoid IP based blocking. Here's how to route Playwright through a proxy:

browser = sync_playwright().start().chromium.launch(
  proxy={
    "server": "http://proxy:8080", 
    "username": "user",
    "password": "pass"  
  }
)
  • Set¬†proxy dict with your provider's credentials
  • Requests will route through the proxy server

Scraping from many different IPs and locations makes you harder to detect and block.

Advanced Techniques

Let's look at some advanced tricks to extend Playwright scraping capabilities.

Executing Custom JavaScript

For maximum flexibility, we can directly run any JavaScript code:

result = page.evaluate("""
  // can access full JS DOM API
  const links = Array.from(document.querySelectorAll('a'));
  return links.map(link => link.href)
""")

print(result)

Advantages:

  • Access any JavaScript API from browser context
  • Return data back to Python context
  • Avoid limitations of Playwright's built-in selectors

Allows doing virtually anything a browser can do!

Blocking Resources

By default Playwright loads all assets – images, CSS, ads, trackers, etc. We can block requests to speed up scraping and reduce bandwidth:

# block images
page.route("**/*.{png,jpg,jpeg}", route.abort())

# block analytics
page.route(
   "https://www.google-analytics.com/analytics.js",
   route.abort() 
)

Some ways to filter requests:

  • File types –¬†.png,¬†.jpg, etc
  • Domain names –¬†google-analytics.com
  • URL patterns –¬†/tracking,¬†/advert
  • Resource types –¬†image,¬†media,¬†font

Streamlines scraping by avoiding unnecessary downloads.

Intercepting Requests

Intercept network requests to monitor or modify them:

def intercept(route):
  # log request
  print(route.request)  

  # update headers
  route.request.headers['User-Agent'] = 'MyBot 1.0'

  # mock response 
  if route.request.url == 'https://api.site.com/token':
    route.fulfill(
      body='{"access": "1234"}'
    )
  else:
    route.continue_()

page.route('**/*', intercept)

Use cases:

  • Log requests for debugging
  • Modify headers
  • Mock API responses
  • Replay requests to cache data
  • Decode encrypted requests

Powerful for analyzing network activity.

Browser Contexts

Playwright pages share browser states like cookies, local storage, etc. For isolation, create pages in separate contexts:

# default shared context
browser.new_context() 

# isolated context
browser.new_context(storage_state="/tmp/new_storage")

Benefits:

  • Dedicated cookies, caches, settings per context
  • Simulate different users/sessions
  • Switch contexts to reuse browser

Helpful when you need stronger separation between page instances.

Python Integration

While Playwright provides the browser automation, we'll want to integrate it with Python libraries to build robust scraping pipelines.

Downloading Files

Use Python libraries like requests for file downloads:

import requests

urls = page.evaluate("""
  const images = document.querySelectorAll('img');
  return Array.from(images).map(img => img.src); 
""")

for url in urls:
  # stream image downloads
  response = requests.get(url, stream=True) 
  response.raw.decode_content = True
  
  with open(f"images/{url.split('/')[-1]}", 'wb') as file:  
    shutil.copyfileobj(response.raw, file)
  • Use JS to extract resource URLs from page
  • Stream download using¬†requests
  • Save resulting files

This leverages Python's better file-handling capabilities.

Parsing Data

For parsing HTML we can use Python libraries like Beautiful Soup:

from bs4 import BeautifulSoup

html = page.content()
soup = BeautifulSoup(html, 'html.parser')

print(soup.find('h1').text)

Benefits of Playwright selectors:

  • A more mature and robust library
  • Very flexible querying API
  • Ability to manipulate DOM tree

Great combination for extraction + parsing.

Asynchronous Support

Playwright has async APIs for concurrent scraping:

import asyncio
from playwright.async_api import async_playwright

async def scrape(playwright):
  browser = await playwright.chromium.launch()
  page = await browser.new_page()
  # ... scraping logic ...
  await browser.close()

async def main():
  async with async_playwright() as playwright: 
    await scrape(playwright)

asyncio.run(main())

With asyncio:

  • Launch browsers concurrently
  • Scale up to many pages
  • Faster overall throughput

Integrates well with other Python async libraries.

Retries & Failures

Robust scrapers need to handle intermittent failures and retries.

from playwright._impl._api_types import Error

try:
  # scraping logic 
except Error as exc:
  print(f"Error: {exc}")

  if retry_count < 3: 
    retry_count += 1
    print(f"Retrying {retry_count}...")
    time.sleep(5)
    
    # re-run failed logic 

  else:
    print("Max retries reached")
    raise Exception("Failed to scrape")

Key points for resilience:

  • Wrap scraping in¬†try/except
  • Catch¬†playwright.Error¬†exceptions
  • Track retry counts
  • Exponential backoff sleeps
  • Reraise after max retries

This provides a robust scraping loop with failure handling.

Conclusion

Some key takeaways:

  • Playwright¬†provides a powerful Python browser automation solution for dynamic scraping.
  • It launches real Chrome/Firefox browsers and executes JavaScript.
  • Interact with pages by scrolling, clicking, typing, etc.
  • Extract data using query selectors or JavaScript.
  • Options for stealth, throttling, mobile simulation.
  • Mature API for flexibility.
  • Build robust pipelines by integrating Playwright with Python data science and web scraping stacks.

Hopefully, this gives you a comprehensive overview of web scraping in Python with Playwright!

John Rooney

John Rooney

John Watson Rooney, a self-taught Python developer and content creator with a focus on web scraping, APIs, and automation. I love sharing my knowledge and expertise through my YouTube channel, My channel caters to all levels of developers, from beginners looking to get started in web scraping to experienced programmers seeking to advance their skills with modern techniques. I have worked in the e-commerce sector for many years, gaining extensive real-world experience in data handling, API integrations, and project management. I am passionate about teaching others and simplifying complex concepts to make them more accessible to a wider audience. In addition to my YouTube channel, I also maintain a personal website where I share my coding projects and other related content.

We will be happy to hear your thoughts

      Leave a reply

      Proxy-Zone
      Compare items
      • Total (0)
      Compare
      0