How to Scrape Redfin Real Estate Property Data in Python?

18 Views

Redfin is one of the largest real estate websites in the United States, with detailed listings for millions of properties across the country. As an experienced web scraper, I often get questions about the best way to extract data from Redfin. In this comprehensive guide, I'll share everything you need to know to build an effective web scraper for Redfin listings using Python.

Whether you're new to web scraping or an expert, you'll learn how to scrape key fields, avoid roadblocks, and construct a robust Redfin data collection system. Follow along for code snippets, explanations, and pro tips for every step of the journey. Let's dive in!

Why Scrape Redfin?

There are several great reasons to scrape data from Redfin:

Comprehensive dataset: Redfin has data on millions of properties across the US, with new listings added daily. Scraping Redfin provides access to one of the largest real estate datasets available.
Granular details: Listing pages on Redfin provide dozens of details like price, bedrooms, bathrooms, size, taxes, MLS ID, and much more. Redfin's data is far more detailed than what's available on many other sites.
Photos and virtual tours: Listing pages include multiple photos of each property along with some virtual tour videos. Great for gathering media on properties.
Agent info: Contact information for the listing agent is provided, helpful for building a database of agents.
Historical data: Older listings often still have data available, enabling analysis of price history and days on market.

This wealth of structured data makes Redfin ideal for data science and analytics use cases in real estate. The data can be used for valuation modeling, investment analysis, and drawing insights into housing inventory and market conditions.

Overview of the Redfin Scraping Process

Scraping seems complicated at first, but can be broken into simple steps:

Find Listing Pages – Discover URLs to scrape with tools like sitemaps.
Send Requests – Programmatically visit each listings page URL.
Parse Pages – Use libraries like BeautifulSoup to extract data.
Store Data – Save scraped info to CSV, JSON, etc.
Expand – Add logic to scrape more fields, locations, etc.
Optimize – Improve speed, avoid blocks, scrape intelligently.

While basic scrapers can be built quickly, mastering these steps takes knowledge and experience. I'll share tricks that took me years to learn along the way. Now let's get coding! I recommend following along by building a Jupyter notebook – we'll be doing a lot of hands-on Python work in this guide.

Importing Python Libraries

Web scraping involves sending HTTP requests, parsing content, and saving data. We'll use some standard Python libraries for this:

import requests
from bs4 import BeautifulSoup
import csv
from time import sleep
import re

requests – Sends HTTP requests to fetch pages.
BeautifulSoup – Parses HTML and XML documents.
csv – Supports saving data to CSV files.
time – Allows delaying requests.
re – Helpful for extracting text with regular expressions.

I recommend getting familiar with the Requests and BeautifulSoup documentation, as we'll rely heavily on them.

Parsing a Sample Listing Page

Let's start by fetching and parsing a single listing page to see what data we can extract:

# Sample listing URL
url = 'https://www.redfin.com/CA/Los-Angeles/1819-S-4th-Ave-90019/home/6988054'

# Fetch page content using Requests  
response = requests.get(url)
page_html = response.text

# Create BeautifulSoup object from HTML
soup = BeautifulSoup(page_html, 'html.parser')

With the page downloaded and parsed, we can now analyze its structure and content. I like to manually review the HTML in my browser to find patterns and locate data. Here's what I notice on this sample page:

Key details like price, beds, baths in <div class="home-summary-row">
An ID home-details-v2 contains extended info
Data attributes like data-rf-test-id="abp-beds" identify some fields
Listing history and tax data sit in scripts
Agent contact info is in <div class="agent-list">

These observations will inform how we extract data – HTML patterns are different on every site. Now let's pull out some sample fields using CSS selectors:

# Use CSS selectors to extract data
price = soup.select_one('.home-summary-row .statsValue').text.replace('\n','').replace(' ','')

beds = soup.select_one('[data-rf-test-id="abp-beds"]').text

baths = soup.select_one('[data-rf-test-id="abp-baths"]').text

size = soup.select_one('[data-rf-test-id="abp-sqft"]').text.replace(',','')

The .select_one() method grabs the first element matching our CSS selector. We also do some cleanup like removing newlines and commas. BeautifulSoup makes extracting data pretty straightforward – we just need to find the right selectors.

Saving scraped data to CSV

To keep our scraped data, we'll write the values to a CSV file:

import csv

# Open a CSV file for writing
with open('redfin_listings.csv', 'w') as file:

  # Create CSV writer 
  writer = csv.writer(file)

  # Write header row  
  writer.writerow(['Price', 'Beds', 'Baths', 'SqFt'])

  # Write data row
  writer.writerow([price, beds, baths, size])

Now we have a CSV with columns for each scraped field, which we can append more rows to. CSV is great for storage and analysis. But JSON, SQL databases, or data formats like Apache Parquet also work.

Scraping Multiple Listing Pages

To collect comprehensive data, we need to scrape details from many listings across cities, states, and price ranges. This involves:

Generating a list of URLs to scrape
Looping through and fetching each page
Extracting and saving the data for each listing

Let's modify our code to scrape 50 sample listings:

from time import sleep 

# List of listing URLs
urls = [
  'https://www.redfin.com/CA/Los-Angeles/1819-S-4th-Ave-90019/home/6988054',
  'https://www.redfin.com/CA/Los-Angeles/4755-W-18th-St-90019/home/6971884', 
  # 50 total URLs
]

# Open CSV file 
with open('redfin_listings.csv', 'w') as file:

  writer = csv.writer(file)
  
  # Write header row
  writer.writerow(['Price', 'Beds', 'Baths', 'SqFt'])  

  # Loop through URLs
  for url in urls:

    # Fetch page HTML
    response = requests.get(url)  
    page = response.text

    # Create BeautifulSoup object
    soup = BeautifulSoup(page, 'html.parser')

    # Extract data
    price = # Get price

    beds = # Get beds

    baths = # Get baths

    size = # Get size

    # Write data row   
    writer.writerow([price, beds, baths, size])
    
    # Pause between pages
    sleep(1)

This gives us a template for scraping and saving data from many listings to CSV. Now let's discuss where to find those listing URLs…

How to Find Redfin Listing Pages to Scrape

The web is a big place – how do we locate the actual listing pages to feed into our scraper?

Search URL Patterns

One option is to notice the URL structure. Listings follow a pattern like:

https://www.redfin.com/city/address/home/123456789

We could build URLs by varying the city, address, and ID. Useful for small scale scrapers.

Redfin Sitemaps

A better approach is to use Redfin's sitemap indexes, which list all available listing pages. Sitemaps help search engines index sites and are ideal for scrapers. Under https://www.redfin.com/robots.txt we see sitemap paths like:

https://www.redfin.com/stingray/do/real-estate-sitemap-index [...]

Redfin API

Unfortunately Redfin shut down their public API years ago. But many sites still offer API access – always check for that first before scraping!

Generate Listing Leads

Use a service like Redfin Estimate to generate listing leads for specific cities and properties you're interested in. We'll focus on sitemaps for now as they provide structured access to all of Redfin's listings.

Parsing Redfin Sitemaps in Python

Let's write a function to extract clean listing URLs from a Redfin sitemap:

import requests
from bs4 import BeautifulSoup

# Fetch sitemap index
sitemap = requests.get('https://www.redfin.com/stingray/do/real-estate-sitemap-index').text  

# Create soup object 
soup = BeautifulSoup(sitemap, 'xml')

# Find all URLs in sitemap
listings = []
for url in soup.find_all('url'):
  
  # Get listing path
  path = url.find('loc').text

  # Remove unneeded parameters
  clean_url = path.split('?')[0]

  listings.append(clean_url)

print(listings[:5]) # Print first 5 records

This grabs the sitemap XML, extracts the <loc> tag from each <url> element to find listing paths, and cleans them up to get a list of URLs to feed into our scraper. With a sitemap powering our URL list, we can gather listings across all of Redfin's markets from a single source.

Scraping Additional Data Fields

So far we've scraped basics like price, beds, baths. But each Redfin listing contains dozens more fields we may want:

Square footage
Lot size
Year built
Price/sqft
HOA dues
Days on market
Property type
Price history
Tax history
School district
Neighborhood
Agent details
Brokerage
Commission
Photos
Virtual tour videos
Parcel number
County
Status
Description

Let's expand our scraper to extract additional fields. I noticed key details sitting in a <div> with id “home-details-v2”:

<div id="home-details-v2">
  <div>Beds: 4</div>
  <div>Baths: 2</div>
  <div>SqFt: 2,358</div>
  <!-- And more details -->
</div>

We can load this entire div and loop through the elements inside to extract fields:

# Get home details div
details = soup.find('div', {'id': 'home-details-v2'})

# Loop over details  
for detail in details:
  
  # Split into key/value
  key, value = detail.text.split(': ')
  
  # Add to dictionary
  data[key] = value

This gives us a dictionary with additional fields! Some data like tax history requires digging into <script> tags in the page source.ctrl + U in your browser makes reviewing source easy. The more you manually analyze pages, the better you'll get at locating data. Expect to spend time experimenting!

Writing Scraped Data to CSV

Now with more fields extracted, let's revisit writing to CSV:

# List of column names 
columns = ['Price', 'Beds', 'Baths', 'Sqft', 'Lot Size', 'Year Built']

# Open CSV for writing
with open('redfin_data.csv', 'w') as file:

  # Create CSV writer
  writer = csv.writer(file)  

  # Write header row
  writer.writerow(columns)

  # Loop through listings
  for url in listing_urls:

    # Scrape listing
    # ...

    # Write scraped data as row
    writer.writerow([price, beds, baths, sqft, lot_size, year_built])

The CSV will now contain columns for each field we scrape. This makes analyzing the data easy. I also recommend exploring formats like JSON and databases for structured storage:

import json 

# Scraped data 
data = {
  'price': '400000',
  'beds': '3',
  # etc.
}

# Save as JSON
with open('listing.json', 'w') as file:
  json.dump(data, file)

-- Create table
CREATE TABLE listings (
  id INT,
  price INT,
  beds INT,
  -- Columns for each field  
  );

-- Insert scraped record
INSERT INTO listings 
  VALUES (
    1001,
    400000, 
    3,
    -- Scraped data
  );

Many data projects require cleaning and combining data from multiple sources. Proper storage will facilitate that down the road.

Scraping Details at Scale

We're now scraping key fields from individual listing pages. To build a production-level scraper:

Add concurrency – Process multiple pages simultaneously with threads or asyncio.
Expand geographies – Gather data from all cities, states, and countries.
Continuously scrape – Check for new listings daily/hourly.
Add proxies – Use proxies/residentials to distribute requests.
Deploy in the cloud – Run your scraper on services like AWS.
Containerize with Docker – Package your scraper as a Docker container.

Let's discuss some best practices for robust, scalable scraping…

Scraping Concurrently with Threads

Serial scraping (one page at a time) is slow. We can speed things up by fetching multiple pages concurrently. Python's threading module makes this easy:

from threading import Thread

# Function to scrape one listing
def scrape_listing(url):
  
  # Scrape logic...
  
  print(f'Scraped {url}')

# List of URLs 
listings = ['listing1', 'listing2', ...] 

threads = []

# Create thread for each listing
for url in listings:
  t = Thread(target=scrape_listing, args=[url])
  threads.append(t)
  t.start()

# Wait for threads to complete  
for t in threads:
  t.join()

print('Scraping complete!')

This basic example runs a thread for each listing URL to fetch/parse pages simultaneously. Just be careful not to create too many threads and overload the server! Start with ~10 threads and experiment.

Scraping Nationwide (and Worldwide)

To build a robust real estate dataset, we need broad geographical coverage:

All major metro areas – Scrape popular cities like LA, NYC, Miami, etc.
Nationwide coverage – Gather listings even in less popular markets.
Rural regions – Don't neglect rural and exurban areas.
Worldwide scope – Expand to markets globally – housing data is useful everywhere!

This means feeding our scraper URLs across many regions. We have a few options:

Custom URL Lists

Manually research and generate URLs for regions of interest. Quick but doesn't scale well.

Crawl Neighborhood Pages

Start from a city page, scrape links to each neighborhood, then listings. Can be fragile.

Leverage Sitemaps

Use Redfin's real estate sitemaps which provide listings across all markets they cover. Just parse the sitemap index to find state/city/region sitemaps, then scrape the URLs within each. This takes advantage of sitemaps existing structure to gather organized nationwide data.

Say we want to scrape all California listings. We'd:

1. Fetch the California state sitemap:

ca_sitemap = 'https://www.redfin.com/stingray/api/mapi/geo-sitemap?region=ca&v=2'

2. Extract all city/county/region sitemaps contained in it:

# Parse CA sitemap
soup = BeautifulSoup(requests.get(ca_sitemap).text, 'xml')

# Find sitemap elements    
sitemaps = soup.find_all('sitemap')

# Extract location sitemaps
local_sitemaps = [s.find('loc').text for s in sitemaps]

3. Loop through those local sitemaps and grab the listing URLs inside each:

# Iterate over local sitemaps
for sitemap in local_sitemaps:

  # Parse sitemap
  soup = BeautifulSoup(requests.get(sitemap).text, 'xml')  

  # Find all listings
  listings = [url.find('loc').text for url in soup.find_all('url')]
  
  # Scrape listings...

This gives us structured access to California listings. Do this for all 50 states to build a nationwide dataset. The same process works for other countries – scrape country sitemaps, then local region sitemaps they link to. Redfin has 90+ regional sitemaps globally to leverage. Sitemaps are a scraper's best friend!

Checking for New Listings Every Hour

Real estate markets move fast – new listings appear daily. To keep our data current, we need to check for new listings frequently. Ideally we'd scrape Redfin for updates every hour. Here's one way to implement that:

1. Database to store listings

We need to remember what listings we've already scraped to check for new ones. A database table works well:

CREATE TABLE listings (
  id INT, 
  url VARCHAR(255),
  scraped_at DATETIME
  );

Whenever we scrape a page, we'll insert a record with the listing ID and scrape time.

2. Cron job to run hourly

Cron is a popular UNIX tool for running scripts on schedules. We'll create a cron job to trigger our scraper every hour:

# Scrape script to run
0 * * * * /scraper/redfin-scraper.py

This will execute our scraper on the 0th minute of every hour.

3. Check for new listings

Our scraper will then:

Fetch latest listing URLs from the sitemap
Compare to existing URLs in our database
Insert any new listings found into the table
Scrape newly added pages

This gives us hourly fresh data! For even faster updates, we could poll every 5 minutes, push new listings via API to our system, and scrape in real time.

Using Proxies to Avoid Blocking

If you start heavily scraping a site, there's a good chance you'll eventually face blocking. Proxies help avoid this by spreading requests across many different IPs.

Residential Proxies: Services like Smartproxy provide millions of residential IP proxies located in cities around the world. Mimics real human users.
Datacenter Proxies: Alternatively, datacenter proxies offer reliable performance but are easier to detect as scrapers, like Proxy-Seller.
Proxy Rotation: Rotate between hundreds of proxies, such as Soax, to distribute scraping volume. Changes the source IP with each request.
Proxy Management: Use a module like BrightData Proxy Manager to handle proxy allocation behind the scenes.

With the right proxy solution, you can scrape aggressively without tripping brute force protection and captchas.

Containerizing Your Scraper with Docker

Once we have a robust scraper, we want an easy way to deploy and run it at scale. That's where Docker comes in handy.

Dockerfile

We can create a Dockerfile to package our scraper into an image:

FROM python:3.8

COPY . /app/

RUN pip install requests bs4 lxml 

CMD [ "python", "/app/redfin-scraper.py"]

This installs Python, copies our code, installs dependencies, and sets the run command.

Build Image

Then building creates an optimized Docker image containing everything needed to run the scraper:

docker build -t redfin-scraper .

Run Container

Finally, spinning up containers from the image launches scrapers quickly:

docker run redfin-scraper

With Docker, we can deploy massively parallelized scraping infrastructure on services like AWS ECS easily.

Scraping Redfin with Selenium

Up to this point we've relied on Requests and BeautifulSoup to fetch and parse listing pages. This works great in most cases, but sometimes JavaScript-heavy sites require browsers. That's where Selenium comes in. Selenium controlling an actual browser like Chrome allows scraping dynamic pages that Requests alone struggles with.

Set Up

First, we'll need to install Selenium and a browswe driver like ChromeDriver:

pip install selenium

# Install ChromeDriver

Launch Browser

Then we can open a browser instance and navigate to pages:

from selenium import webdriver

browser = webdriver.Chrome()

browser.get('https://redfin.com')

Extract Data

Now we can write our normal parsers, but using Selenium's browser to access page content:

soup = BeautifulSoup(browser.page_source, 'html.parser')

price = soup.select_one('#price').text

Headless Mode

To hide the browser, use headless mode:

from selenium.webdriver.chrome.options import Options

options = Options() 
options.headless = True

browser = webdriver.Chrome(options=options)

This provides a great way to scrape complicated sites when needed!

Debugging Web Scrapers

Even with perfect code, scrapers sometimes fail in the real world. Server errors, changes to page structure, blocked requests, and more. Here are my top debugging tips for web scrapers:

Print liberally – Log URLs, HTTP codes, and data to pinpoint issues.
Inspect traffic – Use browser dev tools or a proxy to analyze requests and responses.
Check errors – Handle exceptions gracefully and log details.
Monitor performance – Track metrics like pages scraped per minute.
Review datasets – Spot check sampled data for anomalies.
Version control – Track changes with Git to revert bugs.
Use debugger – Step through code line by line to isolate problems.
Write tests – Unit test parsers to quickly flag regressions.
Try manually – Replicate the logic step-by-step to uncover discrepancies.
Take breaks – Walk away temporarily if stuck debugging for too long.

Don't get discouraged by bugs – they happen to all scrapers. Stay calm, lean on tooling, and methodically solve each issue.

Summary

Web scraping can provide valuable real estate data for analytics and modeling. Using the approach outlined here, you can build a Redfin scraper to supply your projects with property insights. I hope you found this guide helpful!