Real estate data is incredibly valuable for understanding housing markets and spotting opportunities, which is why real estate investors and analysts spend so much time analyzing it. In the digital age, much of this data is available online on real estate listing sites like Zillow, Realtor.com, and Redfin. While these sites provide some analysis tools, the data they make available is limited compared to what can be extracted through web scraping. By scraping real estate listing data and analyzing it yourself, you can gain deeper insights to inform your investing strategy.
In this comprehensive guide, I'll walk you through how to scrape key real estate data points from popular listing sites using Python. With just a little bit of coding, you can build a real estate data pipeline to fuel your own custom analytics.
Why Scrape Real Estate Data with Python?
Before we dive into the how, let's look at why scraping real estate data can be so useful for investors:
- Deeper analysis – Listing sites only provide limited filtering and analytics. Scraping gives you the raw data to analyze however you want.
- More data points – Listing sites don't expose all details. Scraping lets you extract things like full price history, days on market, school districts, and more.
- Market tracking – Regular scrapes let you monitor market trends beyond what listing sites show. You can analyze price changes, new construction, days on market, etc.
- Competitor tracking – Follow listings from specific brokers/agents to analyze their performance.
- Location analytics – Geocode listings and visualize opportunity areas on maps.
- Automation – Automatically pull fresh data instead of manual exports. Build real estate apps and dashboards on top of scraped data.
Python is the ideal programming language for web scraping thanks to libraries like Scrapy, BeautifulSoup, Selenium, and Requests. It makes it easy to write scrapers that extract data from multiple sites. The data can then be loaded into Pandas for analysis.
While you could pay for access to real estate data APIs, scraping gives you more flexibility to gather and analyze the exact data points you need. Scraping listing sites directly gives you fresher data than many APIs provide.
Overall, if you want to unlock the full potential of real estate market data, scraping with Python is the way to go. The rest of this guide will teach you the techniques you need to know.
Key Data Points to Scrape
Before writing a real estate web scraper, it helps to make a list of the key data points you want to extract from listings. Here are some of the most useful fields to target:
- Address/Location
- Price
- Price history
- Square footage
- Lot size
- Bedrooms
- Bathrooms
- Year built
- Property type (single family, condo, multifamily)
- Sale type (for sale by owner, broker listing)
- School district
- County
- Days on market
- Views/saves
- Agent/broker name
- Agent/broker details
- Full description
- All photos
- Virtual tour links
- Tax assessed value
- Property taxes
- HOA fees
- Interior features
- Exterior features
- Parking/garage details
- URL
- Source website
Additional data like walking scores, crime rates, amenities, and demographics can be added later by merging scraped listing data with other sources. But scraping the fields above will give you a rich dataset to work with.
Not every site will contain every data point, but many of the top listing sites have overlapping data. By scraping multiple sites, you can build a more complete view of each property. Now let's look at how to extract these fields from the most popular real estate listing websites.
Scraping Zillow
Zillow is the largest real estate listing portal in the US. All of the key listing details we want are available on Zillow's listing pages, although sometimes take some CSS digging to extract. Here are some tips for scraping Zillow listings with Python:
Finding listing pages
- The main way to locate listing pages is through Zillow's search API. You can search by location and filter by criteria like property type, price range, etc.
- Extract the listing ID from the API response, then construct listing URLs like
https://www.zillow.com/homedetails/{listingId}_zpid/ - Can also scrape listing pages from search results, but API gives more options for finding relevant listings.
Extract key data points
- Address, price, beds/baths, square footage, lot size, broker name etc are in the listing summary section.
- Additional details like year built and parking require CSS selectors to extract from page HTML.
- Price history and days on Zillow are loaded dynamically. Need to scrape these by extracting data from
window.__REDUX_STATE__. - Use Selenium to click through all photos to download.
Example Zillow scraper in Python
Here is some sample Python code that searches Zillow, extracts listing IDs, builds listing URLs, scrapes key data points, and stores results to a Pandas DataFrame:
import requests
from bs4 import BeautifulSoup
import pandas as pd
listings = [] # Store listing data
# Search API request
api_url = "https://www.zillow.com/search/GetSearchPageState.htm"
params = {
"searchQueryState": {
"pagination": {},
"usersSearchTerm": "New York, NY",
"mapBounds": {},
"regionSelection": [],
"isMapVisible": False,
"filterState": {
"isMakeMeMove": False,
"isAllHomes": {
"value": True
},
"isForSaleByAgent": {
"value": False
},
"isNewConstruction": {
"value": False
},
"isForSaleByOwner": {
"value": False
},
"isComingSoon": {
"value": False
},
"isAuction": {
"value": False
}
},
"isListVisible": True
},
"mapZoom": 11,
"regionSelection": [],
"isMapVisible": False,
"filterState": {
"sortSelection": {
"value": "globalrelevanceex"
},
"isAllHomes": {
"value": True
}
}
}
response = requests.post(api_url, json=params)
data = response.json()
# Extract listing IDs
for listing in data['searchResults']['listResults']:
zpid = listing['zpid']
# Construct listing URL
url = f"https://www.zillow.com/homedetails/{zpid}_zpid/"
# Download listing page
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
# Extract data points
title = soup.select_one(".ds-home-details-banner-ad .ds-chip").getText()
address = soup.select_one(".ds-home-details-banner-ad .ds-heading-2").getText()
price = soup.select_one(".ds-home-details-chip").getText()
beds = soup.select_one(".ds-bed-bath-living-area .ds-bed-bath-living-area-bed").getText()
baths = soup.select_one(".ds-bed-bath-living-area .ds-bed-bath-living-area-bath").getText()
sqft = soup.select_one(".ds-bed-bath-living-area .ds-bed-bath-living-area-sqft").getText()
broker_name = soup.select_one(".ds-home-details-chip.ds-text-title").getText()
# Store data
listings.append({
"title": title,
"address": address,
"price": price,
"beds": beds,
"baths": baths,
"sqft": sqft,
"broker_name": broker_name
})
# Convert to Pandas DataFrame
df = pd.DataFrame(listings)This covers the basics of extracting key fields from Zillow. More advanced techniques like parsing the Redux state and using Selenium can help extract additional data points not shown in this example.
Scraping Realtor.com
Realtor.com is another top real estate listing portal in the US. The underlying data is fairly similar to Zillow, so the scraping techniques are comparable:
Finding listing pages
- Use Realtor's search API to lookup listings by location/criteria and extract listing IDs
- Construct listing URLs like
https://www.realtor.com/realestateandhomes-detail/{listingId}
Extracting data points
- Main fields like price, beds, baths, sqft are in the listing summary
- CSS selectors needed for some additional fields like parking, year built
- Price history and days on market require parsing the page Redux state
- Use Selenium to gather all photos
Example Python code
import requests
from bs4 import BeautifulSoup
import pandas as pd
listings = []
# Realtor API request
search_url = "https://realtor.p.rapidapi.com/properties/v2/list-for-sale"
params = {
"sort": "relevance",
"city": "New York",
"limit": "50",
"offset": "0",
"state_code": "NY"
}
headers = {
"X-RapidAPI-Key": "YOUR_API_KEY",
"X-RapidAPI-Host": "realtor.p.rapidapi.com"
}
response = requests.get(search_url, params=params, headers=headers)
results = response.json()["properties"]
for listing in results:
mlsId = listing["mls_id"]
url = f"https://www.realtor.com/realestateandhomes-detail/{mlsId}"
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
title = soup.select_one(".property-title").getText().strip()
address = soup.select_one(".street-address").getText()
price = soup.select_one(".ds-beds-baths-sqft > .ds-product-price").getText()
beds = soup.select_one(".ds-bed > .ds-product-beds").getText()
baths = soup.select_one(".ds-bath > .ds-product-baths").getText()
sqft = soup.select_one(".ds-sqft > .ds-product-sqft").getText()
listings.append({
"title": title,
"address": address,
"price": price,
"beds": beds,
"baths": baths,
"sqft": sqft
})
df = pd.DataFrame(listings)Again, this covers the basics but more advanced techniques can pull additional fields like agent info, taxes, HOA fees etc. The overall parsing process is very similar to Zillow.
Scraping Redfin
Redfin has listings across the US and Canada, making it another good source for scraping real estate data. The steps are similar:
Finding listings
- Redfin has a places API that can be searched by location to get listing IDs
- Construct listing URLs like
https://www.redfin.com/stingray/do/property-details?listing_id={listingId}
Extracting details
- Main fields in listing summary section
- Additional fields require CSS selection of page elements
- Parse Redux state for price history, days on market
- Use Selenium to gather all photos
Python scraping script
import requests
from bs4 import BeautifulSoup
import pandas as pd
listings = []
# Redfin places API request
api_url = "https://redfin.com/stingray/do/location-autocomplete"
params = {
"location": "New York, NY",
"limit": 50
}
response = requests.get(api_url, params=params)
data = response.json()
for result in data['locations']:
listing_id = result['value']
url = f"https://www.redfin.com/stingray/do/property-details?listing_id={listing_id}"
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
title = soup.select_one(".headline").getText().strip()
address = soup.select_one(".street-address").getText()
beds = soup.select_one(".beds").getText()
baths = soup.select_one(".baths").getText()
sqft = soup.select_one(".sqft").getText()
listings.append({
"title": title,
"address": address,
"beds": beds,
"baths": baths,
"sqft": sqft
})
df = pd.DataFrame(listings)Scraping International Sites
In addition to the major US portals, don't forget about scraping international real estate sites to get data for markets outside the US.
For example:
- Rightmove – UK
- REA Group – Australia
- Century21 – Global franchise with country sites
- Juwai – Chinese international listings
The parsing logic is largely the same across these sites. The main differences are in finding the search APIs and listing page structures between the different platforms.
Analyzing Scraped Real Estate Data
Once you've built scrapers for one or more listing sites, you can combine and analyze the aggregated data however you want. For example, you can load all the scraped listing details into a Pandas DataFrame for analysis:
import pandas as pd
# Load scraped data
zillow_data = pd.read_csv('zillow.csv')
redfin_data = pd.read_csv('redfin.csv')
# Concatenate multiple data sources
listings = pd.concat([zillow_data, redfin_data])
# Analyze combined dataset
listings_by_zipcode = listings.groupby("zipcode").mean()
listings_by_type = listings.groupby("property_type").count()Beyond Pandas, scraped real estate data can be loaded into SQL or NoSQL databases for further analysis using tools like Python's SQLAlchemy library. You can also visualize trends in the housing data using Python visualization libraries like Matplotlib and Plotly Express. Interactive dashboards can be built with Panel and Streamlit.
The possibilities are endless once you have structured real estate market data extracted through web scraping!
Scraping Best Practices
When scraping real estate listing sites, keep these best practices in mind:
- Use proxies – Rotating IP proxies is essential for avoiding blocks when scraping aggressively. Proxy services like BrightData, Smartproxy, Proxy-Seller, and Soax provide millions of residential IPs ideal for real estate scraping.
- Add random delays – Insert random delays between requests to mimic human browsing patterns.
- Check robots.txt – Avoid scraping pages blocked in a site's robots.txt file.
- Limit request rate – Make requests slowly to stay under a site's throttling limits.
- Use caches – Cache downloaded pages to avoid repeat requests for unchanged data.
- Retry failures – Retry failed requests up to 3-5 times before giving up.
- User agents – Spoof a variety of desktop/mobile user agents.
- Handle captchas – Pause scraping when encountering captchas. Some services can automatically solve captchas.
- Stay updated – Check sites regularly for changes in APIs, HTML, and anti-scraping measures.
Following web scraping best practices helps avoid problems and ensures reliable data collection over time.
Conclusion
Scraping real estate listing data opens up many possibilities for better understanding of housing markets. With the techniques covered in this guide, you can now leverage sites like Zillow, Realtor, and Redfin to extract key property details at scale using Python.
The scraped data can fuel advanced analytics, visualization dashboards, market tracking over time, and more. While listing portals provide their own limited analysis tools, scraping gives you the flexibility to analyze the raw data however you want.
From identifying undervalued properties to predicting home price trends and mapping opportunity zones, scraping real estate data unlocks superior insights for investing and research. Give it a try and see where it takes your analysis capabilities!
