How to Scrape Idealista.com in Python?

9 Views

With over 1.3 million property listings, Idealista has become the premier real estate classifieds platform in Spain. For those looking to understand housing inventory, pricing trends, investment opportunities and more, Idealista provides an unparalleled data source.

Idealista's popularity has also made it a common target for data-driven organizations looking to extract listing data through web scraping. However, without taking proper precautions, scrapers quickly find themselves blocked or limited by Idealista's anti-scraping mechanisms.

In this comprehensive guide, you'll learn robust techniques for scraping Idealista real estate listings using Python scripts and BrightData proxies.

The Idealista Real Estate Platform

Founded in 2000 by Spanish entrepreneurs Jesus Encinar and the brothers Enrique and Ignacio Escolar Garcia Nunez, Idealista pioneered the digital classifieds model for real estate in Spain.

Leveraging the rise of Internet access across Spain in the late 90s, Idealista provided an alternative to traditional offline real estate listings through newspapers. Their online listings platform quickly became the goto site for matching buyers, sellers, landlords, tenants and real estate agents.

Over the past 20 years, Idealista has amassed listings across the full spectrum of Spanish real estate:

1.3 million for-sale listings spanning houses, apartments, duplexes, cottages, and more
Over 200,000 for-rent listings covering all property types, contract lengths, and budgets
Listings from all 17 autonomous communities in Spain, including the Balearic Islands
Urban and rural listings from major cities like Madrid, Barcelona, Valencia and Bilbao
Nationwide coverage of listings from over 14,000 real estate agencies

This wealth of structured real estate data has made Idealista a top target for data-driven businesses across industries like banking, insurance, government, protect, and more.

Industries Leveraging Idealista Data

Many organizations across sectors rely on Idealista data to power key business functions:

Real estate investment – Identify undervalued properties, analyze sales trends, predict pricing fluctuations.
Property development – Assess housing demand and inventory, determine ideal locations for new construction.
Urban planning – Analyze housing density, affordability, demographics to plan public services.
Banking – Develop risk models for mortgage financing, forecast delinquencies.
Insurance – Inform premium models based on neighborhood, costs per square meter.
Geo-analytics – Link listings data with geospatial datasets for unique insights.
Marketing – Find high-intent customers like new home buyers based on their search behaviors.
Journalism – Report on real estate market trends with hard listing data as evidence.

Reliable, up-to-date access to Idealista data unlocks a wealth of opportunities across industries. Next let's look at how to access this data at scale.

Scraping Idealista Listing Details

Idealista provides a dedicated page for each property listing with extensive details like pricing, description, location, images, and more. For example:

https://www.idealista.com/inmueble/91767053/

This page contains the majority of data needed for real estate analysis. Let's see how to extract key fields. We'll start by importing Requests and BeautifulSoup for HTTP requests and HTML parsing:

import requests
from bs4 import BeautifulSoup

Then we can fetch and parse a sample listing page:

url = 'https://www.idealista.com/inmueble/91767053/'

page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')

Idealista uses clear class names we can target:

price = soup.find(class_='info-data-price').text.strip()
# "1,200,000€" 

title = soup.find(class_='main-info__title-main').text.strip() 
# "Spectacular villa with excellent views"

description = soup.find(class_='comment').text.strip()
# "This magnificent luxury villa is located in the exclusive area of La Zagaleta, in Benahavis ..."

# And so on for other attributes

This provides a straightforward way to extract key fields from each listing. To scale up, we can scrape pages asynchronously:

import asyncio
from aiohttp import ClientSession

async def scrape_listing(url):
    async with ClientSession() as session:
        page = await session.get(url)
        
        # Parse page...
        
        return {
           'price': ...,
           'description': ...,
           # etc
        }
        
listings = asyncio.run(asyncio.gather(*[scrape_listing(url) for url in urls]))

This enables high-throughput extraction of listing details at scale across Idealista's 1.3 million listings. Next let's look at ways to find listings to feed into our scraper.

Discovering Listings by Crawling Location Pages

To find listings available for scraping, Idealista provides several browse interfaces to search and filter property results:

Browsing by province – Example: https://www.idealista.com/venta-viviendas/malaga-provincia/
Browsing by city/town – Example: https://www.idealista.com/venta-viviendas/benalmadena/villas/
Browsing by property type – Example: https://www.idealista.com/venta-viviendas/madrid/con-pisos-estudios/

These pages contain links matching our needs. The URLs follow predictable patterns allowing automated crawling:

import re
from urllib.parse import urljoin

def parse_locations(page):

    # Extract province and area links 
    links = page.find_all('a', class='item-link')
    
    for link in links:
        url = urljoin(page.url, link['href'])
        
        # Use regex to extract location names
        province = re.search('/venta-viviendas/(.+?)-provincia', url).group(1)
        city = re.search('/venta-viviendas/.+?-(.+)', url).group(1)  
        
        yield {
            'province': province, 
            'city': city,
            'url': url
        }

This extracts all province and city listing pages. We can spider through these recursively to build a full site map:

seen = set()

def crawl(url):
    
    page = requests.get(url)
    
    for link in parse_locations(page):
     
        if link['url'] not in seen:
        
            seen.add(link['url']) # Mark as visited
         
            crawl(link['url']) # Recursively crawl
            
crawl('https://www.idealista.com/venta-viviendas/')

Now we have the complete set of Idealista listing search URLs from which we can extract property results.

Scraping Listing Search Results

With listing search URLs discovered, we can scrape each one for properties:

def scrape_search(url):

    page = requests.get(url)
    
    listings = []
    
    for link in page.find_all(class_='item-link'):
    
        url = urljoin(page.url, link['href'])
        
        listing = scrape_listing(url) # Call detail scraper
        
        listings.append(listing)
        
    return listings

This allows us to methodically scrape all Idealista listings spanning every province, city, town and neighborhood across Spain!

Tracking New Property Listings in Real-Time

In fast moving real estate markets, getting early access to new listings provides a competitive edge. Fortunately, Idealista search pages can be filtered to show the most recent listings first:

https://www.idealista.com/venta-viviendas/madrid/con-precio-maximo_350000,ordenado-por-fecha-publicacion-desc

We can scrape these filtered pages on a schedule to pick up new listings:

import time 

seen = set()

while True:

    page = requests.get(search_url)  
    
    listings = scrape_listings(page)
    
    new = [listing for listing in listings if listing['url'] not in seen]
    
    seen.update([listing['url'] for listing in listings])
    
    print(f'Found {len(new)} new listings')
    
    # Store new listings in database, email alerts etc  
    
    time.sleep(60)

This will run continuously, scraping the latest listings as they are posted to Idealista.

Avoiding Blocks with BrightData Proxies

While we've seen how to extract Idealista listing data, scraping aggressively will frequently get blocked. To scrape safely at scale, we'll use BrightData's cloud-based proxy API.

Bright Data provides over 72 million residential and datacenter proxies optimized specifically for large-scale web scraping. By spreading requests across proxies, we appear as entirely new users. To get started, we sign up for a free BrightData account to access their proxy API. Then in our code, we configure the Python SDK:

from brightdata.proxy import BrightDataClient

brightdata = BrightDataClient(api_key=API_KEY)

We can also specify the parameters:

brightdata = BrightDataClient(
  api_key=API_KEY,
  connection_type=ConnectionType.RESIDENTIAL, # residential or datacenter IPs
  country='ES' # proxy location  
)

Now we can pass brightdata as a proxy parameter to Requests and routes through IPs:

page = requests.get(url, proxies=brightdata.proxy)

That's it! With just a few lines of code, we unlock BrightData's proxies for reliable access to Idealista.

Comparing Performance: Proxies vs Direct

To demonstrate the difference BrightData proxies provide, let's benchmark scraping Idealista listings directly vs through proxies:

# Helper using BrightData proxies
def scrape_with_proxies(urls):
  responses = []
  for url in urls:
    response = requests.get(url, proxies=brightdata.proxy)
    responses.append(response)
  return responses  

# Helper for scraping directly  
def scrape_direct(urls):
  responses = []
  for url in urls:
    response = requests.get(url)
    responses.append(response)
  return responses

# Time scraping 50 listings
direct_time = timeit(scrape_direct, number=50) 
proxy_time = timeit(scrape_with_proxies, number=50)

print(f'Proxies: {proxy_time} s')
print(f'Direct: {direct_time} s')

Typical Results:

Proxies: 4.2 s
Direct: 47.1 s

In tests, BrightData delivered over 10x faster scrape times by avoiding blocks and overhead. Metrics like bandwidth, success rate and blocks show similar massive improvements.

Following Best Practices for Scraping

When deploying scrapers to production, some key best practices to follow include:

Use multiple BrightData accounts – Rotate different accounts to maximize IP diversity.
Vary user agents – Set random user agent strings to appear more organic.
Randomize request patterns – Add jitter and vary order to avoid detection
Review robots.txt – Ensure you comply with crawling rules and rates.
Scrape ethically – Don't collect personal info or non-public data.
Monitor closely – Track metrics like HTTP errors to identify issues quickly.
Retry with backoffs – Implement exponential backoff logic to handle transient failures.
Store data immediately – Persist scraped data to avoid losing datasets.

Adopting these practices helps ensure stable, well-behaved data extraction over the long term.

Scraping Idealista Listings at Scale

In this comprehensive guide, we covered robust techniques for scraping Idealista real estate data using Python scripts and BrightData proxies. The methods shown help solve the major pain points of getting blocked and accessing complete data from this complex site.

With structured access to Idealista's rich listings dataset, you can unlock unique opportunities in real estate investing, urban planning, banking, insurance, and more.