How to Scrape Nordstrom?

6 Views

E-commerce scraping lets you unlock data riches – if you know where to dig. As one of the largest US fashion retailers, Nordstrom offers an abundance of product information just waiting to be unearthed. In this guide, I'll map out everything you need to extract Nordstrom's data gold using Python.

Here's what we'll uncover:

Why Nordstrom deserves your attention
Navigating legal digging rights
Surveying the Nordstrom data landscape
Choosing your scraping tools
Step-by-step scraping instructions
Handling Nordstrom's data defenses
Pulling it all together into a data pipeline
Expanding your data gathering beyond products
Avoiding data mining pitfalls

Let's get started – rich seams of data await.

Why Focus Your Scraping on Nordstrom?

With hundreds of e-commerce sites out there, why target Nordstrom?

Sheer data volume – Nordstrom offers one of the largest fashion catalogs including over 1,000 brands and 500,000+ products across categories like clothing, shoes, accessories, beauty, kids, home, and more.
Premium positioning – As a high-end yet accessible retailer, Nordstrom attracts brands focusing on quality, fashion, and aspirational lifestyles – making for great data.
On-trend assortment – Nordstrom prides itself on expertly curated selections reflecting the latest styles and niche designers – ideal for trend analysis.
High fidelity data – Nordstrom product pages offer extensive structured data – descriptions, bullet points, images, pricing, variants, and more.
Data diversity – Beyond products, Nordstrom provides order data, brand relationships, store details, marketing info, and other assets.
Significance – As a barometer for US fashion commerce, Nordstrom data offers insights into a multi-billion dollar industry.

For web scraping, bigger is often better. With Nordstrom, you gain access to a true data goldmine to fuel all kinds of applications.

Establishing Your Legal Digging Rights

Before setting out to scrape any website, you need to make sure you have the legal right to do so. Here are some key guidelines for Nordstrom data:

Scraping Nordstrom's public site data is generally permissible – Products, pricing, catalog data is all considered public information. Private user data is off limits.
Watch your scraping intensity – Nordstrom may object to extremely high volumes over extended periods. Take care not to overload their systems.
Attribute properly – If using Nordstrom's data in public projects, provide appropriate credit and attribution.
Observe copyrights – User-generated content like customer reviews may hold certain copyrights depending on jurisdiction.
When in doubt, seek counsel – Laws vary globally. Consult an attorney if you need clarification for your specific situation.
Respect robots.txt – This file provides guidance on allowable vs restricted scraping activities.

By staying mindful of Nordstrom's rights, you can confidently scrape to your heart's content! Just be sure to do so in a responsible manner.

Surveying The Nordstrom Site Data Landscape

To extract the motherlode, we first need to understand how Nordstrom's website is structured. Here are some key points:

Catalog architecture – Nordstrom categories data across departments, brands, product types, features, and price filters for easy browsing.
Consistent templates – Product, category, and brand pages share similar underlying HTML structures we can target.
Dynamic serving – Nordstrom leverages React and JS frameworks to load content dynamically – requires some special tools.
API availability – Much of Nordstrom's data is sourced from internal JSON APIs accessible through calls.
Heavy pagination – Large category results are spread across many numbered result pages.

While complex under the hood, Nordstrom's unified architecture gives us clear scraping entry points across its data assets. Now we just need the right tools to capitalize.

Choosing Your Scraping Tools

Nordstrom presents some unique scraping challenges between pagination, dynamic JS sites, and blocking. Here are my top Python tool recommendations:

ScraperAPI– For proxy management, CAPTCHAs solving, and block circumvention. Critical for large-scale efforts.
Scrapy – Full-featured scraping framework with middleware, proxies, and other automation.
Selenium + BeautifulSoup – For rendering JS sites and then parsing the HTML. A common pairing.
scraperbox – Cloud proxy API delivering 1M+ IPs with fast IP rotation for anonymity.
Google Sheets – For storing scraped data sets. Integrates well with many Python scraping libraries.
ScrapeOps – Containerizes and manages scrapers at scale with Kubernetes – great for Nordstrom's vast catalog.

With the right tools, we can power through Nordstrom's defenses to extract the goods. Now let's get scraping!

Step-by-Step Nordstrom Scraping Instructions

Alright, let's walk through the nitty-gritty Nordstrom scraping process from start to finish:

Phase 1 – Product Page Discovery

First, we need to find the actual product pages across the Nordstrom catalog. A few approaches:

Categories – Navigate Nordstrom's taxonomy to target products by type, brand, segment etc. Broad coverage.
Search – Use Nordstrom site search for keyword filters like “denim jackets” or “mens sneakers”. More precise but potentially incomplete.
Sitemap crawling – Parse Nordstrom's sitemap.xml for all indexed URLs (see Scraperbox example below). Comprehensive but messy.

I suggest starting with categories for a managed scraping flow. Here's an example with Selenium/BeautifulSoup:

from selenium import webdriver 
from bs4 import BeautifulSoup
import time

# Initialize headless Chrome browser via Selenium
browser = webdriver.Chrome()

# Load target category page 
browser.get('https://www.nordstrom.com/browse/women/clothing?breadcrumb=Home%2FWomen%2FClothing&offset=0&page=1')
time.sleep(3)

# Parse rendered page HTML
soup = BeautifulSoup(browser.page_source, 'html.parser')

# Calculate number of result pages  
results = int(soup.find('p', 'result-count').text.split()[-1].replace(',',''))
per_page = len(soup.select('li.product-grid'))
last_page = math.ceil(results / per_page)

# Generate category page URLs
pages = []
for page in range(1, last_page+1):
  params = f'&offset=0&page={page}'
  pages.append(f'https://www.nordstrom.com/browse/women/clothing?{params}') 
  
browser.quit()

print('Category pages to scrape:', len(pages))
print(pages)

This handles JS rendering then parses out the paginated category URL list – our Nordstrom data breadcrumb trail. Now we can iterate these category pages and extract product detail links:

product_urls = []

for category_url in pages:

  browser.get(category_url)
  soup = BeautifulSoup(browser.page_source, 'html.parser')
  
  products = soup.select('li.product-grid a.product-grid')
  
  for product in products:
    url = product['href']
    product_urls.append(url)

print('Found product urls:', len(product_urls)) 
print(product_urls[:5])

Rinse and repeat this for all your target Nordstrom categories. That's page one down!

Phase 2 – Data Extraction

With a list of product URLs, we're ready to scrape. Here's an example extraction flow:

import json
from selenium import webdriver
from bs4 import BeautifulSoup 

product_url = 'https://www.nordstrom.com/s/bp-lace-inset-midi-skirt-regular-petite/5964050'

browser = webdriver.Chrome()
browser.get(product_url)
soup = BeautifulSoup(browser.page_source, 'html.parser')
browser.quit()

# Title
title = soup.find('h1', 'bc-heading').text.strip()

# Price
price = soup.find('span', 'price').text.strip()  

# Description
desc = soup.find('div', {'id': 'description'}).text.strip()

# Images 
imgs = [img['src'] for img in soup.select('img.hero-image')]

# Product data JSON
data = json.loads(soup.find('script', type='application/json').string)
category = data['page']['product']['productCategory']['displayName']

print('Title:', title)
print('Price:', price)  
print('Description:', desc[:150], '...')
print('Images:', imgs)
print('Category:', category)

The key steps:

Use Selenium to render JS
Parse HTML with BeautifulSoup
Extract details from tags
Load JSON object for additional data
Zip together into final product record

Running this for all products gives us the full Nordstrom dataset. We can then export to CSV, SQL, Google Sheets, etc.

Phase 3 – Handling Pagination

As mentioned, Nordstrom spreads catalog data across many pagination pages. We'll need to handle these to gather all the products. The method is the same as shown earlier for categories:

Fetch total result count
Calculate pages needed
Generate numbered page URLs
Loop through page range scraping each

Building this logic into your scraper lets you traverse paginated searches, brand catalogs, top products, sales, etc. Key for large-scale Nordstrom scraping.

Bypassing Nordstrom's Defenses

Like many sites, Nordstrom deploys protections against scraping including:

IP blocking – Nordstrom blacklists suspicious IPs sending too many requests.
Traffic shaping – Bandwidth limiting temporarily blocks scrapers exceeding thresholds.
CAPTCHAs – Nordstrom may trigger human verification challenges under a heavy load.
Legal threats – Excess scraping may prompt Nordstrom to send warnings or legal notices.

To avoid blocks, you'll need to carefully manage your scraper. Here are some tips:

Use proxies – Rotate different outbound IPs to distribute requests anonymously.
Enable delays – Crawl politely with randomized intervals between page visits.
Limit daily volumes – Keep sessions under 10K to 50K pages spaced over time.
Vary user agents – Cycle through different browser UA strings using a pool.
Solve CAPTCHAs automatically – Leverage tools like 2Captcha to bypass human checks when triggered.
Consult Nordstrom allow list – Review Nordstrom's robots.txt for guidance on permissible scraping.
Scrape via proxy API services – Leverage vendor proxy networks like BrightData and Smartproxy to automate proxy management at scale.

With the right precautions, you can confidently scrape Nordstrom without tripping alarms. Proxies and commercial tools help tame Nordstrom's defenses.

Refining Your Nordstrom Data Pipeline

While we've covered core extraction techniques, let's talk workflow. For smooth Nordstrom data mining, consider:

Scraping framework – Use a structured framework like Scrapy or ParseHub for easier management.
Incremental scraping – Only re-scrape updated content to avoid redundancy. Pipelines help here.
Data validation – Clean malformed data, filter junk, de-dupe, etc. to protect pipeline.
Persistent storage – Warehouse data in a database, data lake, or Google BigQuery rather than local files.
Containerization – Dockerize your scraper for simplified deployment and scaling.
Monitoring – Use scraper platform tools like ScrapeOps to monitor runtimes, catch errors, etc.
Automation – Schedule extraction runs for hands-free overnight and weekly large scrapes.
Maintenance – Check for site changes and update parsers accordingly.

Investing in these areas pays dividends for long-term production scraping.

Expanding Your Data Gathering

While products make up the bulk of Nordstrom data, you can extend scraping to related assets:

Brand pages – Provide supplemental details like brand bios, designer backgrounds, related products, etc.
Product grouping pages – Aggregate products in editorial collections for trend analysis.
Category drill-downs – Mine low-level niche categories like “Petite Linen Pants” for hidden products.
Top results – ‘Best sellers' and other ranked pages surface popular products.
Search filters – Filter by attributes like size, color, price, rating for more segmentation.
Reviews – Tons of text data but copyright considerations. Use selectively.
Inventory – Check real-time store and warehouse inventory for out of stocks.
Pricing – Sale / markdown schedules and regions for price point analysis.
Order data – Transactions, product performance, sales velocity, customer details. Difficult to directly access.

Get creative digging into these sources for 360 insights beyond individual product specs.

Avoiding Nordstrom Data Mining Pitfalls

Let's wrap up with some common Nordstrom scraping mistakes and how to avoid them:

Bad proxies – Low-quality datacenter proxies are easily flagged. Use residential IPs you fully control like Bright Data, Smartproxy, Proxy-Seller, and Soax.
No JS rendering – Nordstrom's React-based site needs true headless browser rendering. PhantomJS can also work.
Aggressive crawling – Spread scraping volume over multiple days or weeks. Bursts are suspicious.
Unstructured data – Plan parsing fields and data modeling ahead of time. Don't just scrape blindly.
Lack of monitoring – Track runtimes, errors, blocks etc. to catch problems early.
No change detection – Periodic checks help surface HTML and API shifts breaking your parser.
Outdated tools – Don't let Python packages lag – Nordstrom evolves so your scraper must too.
No legal review – Even public data may have specific terms. Do due diligence.
Copyright overreach – Be careful when storing customer reviews and UGC. Restrict as needed.

A bit of care goes a long way for successful large-scale extraction.

Conclusion

Nordstrom provides a true web scraping goldmine – if you mine it right. By approaching extraction carefully and methodically, you can build Nordstrom data assets to power all kinds of apps – from market research to inventory monitoring and beyond. Now get out there, stake your Nordstrom data claims, and dig into some digital riches!