XPath is a powerful query language for selecting elements in XML documents. One of its most useful features is the ability to select elements by their class attribute. Selecting by class allows you to easily grab elements that share a common CSS class name.
In this comprehensive guide, you'll learn how to leverage XPath to target elements by class name. We'll cover the basics of XPath, dive into using the @class attribute, look at advanced selection techniques, and more. By the end, you'll be an expert at extracting elements from HTML and XML using class-based XPath selectors.
XPath Basics
Before diving into class selection specifics, let's recap some core XPath basics that will be useful. XPath models an XML/HTML document as a tree structure. Elements are nodes that can be selected by traversing this tree.
Some key syntax includes:
/to select root node children//for any descendant node..to select the parent node@for attributes like@class[]predicates to filter nodes
For example:
//div[@class="header"]
This selects <div> elements anywhere in the document with a class attribute equal to "header". Expressions can also use wildcards (*) and conditions (and, or). This gives XPath great power to target elements based on attribute values, position, text content, and more.
Now let's see how these basic building blocks can be used for sophisticated class-based selection.
Exact Match Class Selection
The simplest technique is to select elements where the class attribute exactly matches a specific class name. For example, to select <div> elements with class "header":
//div[@class="header"]
The @class="header" predicate limits matching <div> elements to only those with a class attribute value equal to the string "header". This is great when we know an exact class name to target.
Some examples:
//input[@class="submit-btn"] //ul[@class="product-listing"] //p[@class="error-text"]
However, an exact match requires the class name match precisely. This brings some limitations. First, it doesn't allow partial matching – @class="header-text" wouldn't be selected. Second, elements with multiple classes like <div class="header extra-large"> would be skipped.
So while exact matching gives precision, it can also be limiting. Next let's look at more flexible techniques.
Flexible Class Matching with Contains
A more flexible way to select elements with a certain class is using XPath's contains() function. This allows matching class values that contain a specific substring, rather than requiring an exact match.
For example:
//div[contains(@class, "header")]
This will select <div class="main-header">, <div class="header">, or any <div> with a class that includes "header". Much more flexible! And useful when dealing with elements that have multiple classes:
<div class="product-listing clearance sale">
contains(@class, "product-listing") would still match this element, despite other classes being present. One thing to watch is that contains() will match any element with the substring anywhere in its class. So it could unintentionally match partial class values like disabled-header-text.
To mitigate this, we can be a bit more strict with space normalization:
//div[contains(concat(' ', normalize-space(@class), ' '), ' header ')]This adds spaces before/after the class value before checking for the contain string. In general though, contains() is immensely useful for flexible class selection in XPath.
Excluding Specific Classes
Another useful technique is selecting elements that do not contain a certain class name. The != operator can be used to exclude elements from selection based on their class attribute.
For example, to select <div> elements that don't have class "header":
//div[@class!="header"]
This will match any <div>s lacking the exact "header" class. Building on that, we can exclude multiple specific classes using and:
//div[@class!="header" and @class!="footer"]
This will select <div> elements without either "header" or "footer" in their class. Exclusion gives us another tool for targeting subsets of elements by class.
Handling Multiple Classes
A very common scenario is elements with multiple class names assigned.
For example:
<div class="product-listing clearance sale">
To robustly target elements like this, we have a few options:
Normalize Whitespace
As mentioned earlier, we can normalize whitespace in the class with concat():
//div[contains(concat(' ', normalize-space(@class), ' '), ' product-listing ')]This collapses all whitespace to a single space, then checks for a contain match.
Look for Starts/Ends With
Alternatively, we can look for the class name at the start or end of the attribute value:
//div[starts-with(@class, 'product-listing') or ends-with(@class, 'product-listing')]
This is less strict than an exact match, but more precise than a contains match.
Match Each Class
Finally, if we need to explicitly match multiple classes, we can chain together multiple contains():
//div[contains(@class, 'product-listing') and contains(@class, 'clearance')]
This ensures an element has both necessary class names we're looking for. No matter the approach, properly handling multi-class elements is crucial for robust XPath class selection.
Advanced Class Selection Techniques
Those are the most common and useful techniques. But XPath also provides some more advanced functions and syntax for class selection scenarios requiring extra flexibility.
Ends With
The ends-with() function allows selecting elements where the class ends with a given substring:
//div[ends-with(@class, '-text')]
This would match <div class="header-text"> but not <div class="disabled-text">.
Starts With
Similarly, starts-with() selects elements where the class starts with a substring:
//div[starts-with(@class, 'header-')]
Matching <div class="header-banner"> but not <div class="footer-text">.
Regex Matching
For ultimate flexibility, we can use matches() to match the class attribute against a regular expression:
//div[matches(@class, 'header-.+\')]
This allows powerful regex patterns to target elements.
Class Axis
There's also a built-in “class” axis that selection elements by class name:
//div/class::header-text
Which can be a bit simpler syntax in some cases.
Combining Techniques
And we can combine together all these techniques for sophisticated logic, like:
//div[contains(@class, 'header') and matches(@class, '.*-text') and not(starts-with(@class, 'footer'))]
Chaining together different matching functions gives immense flexibility to handle complex real-world class attributes.
XPath Class Selection in Python
Now that we've seen a variety of selection techniques, let's look at actually integrating XPath class matching into Python scripts. For web scraping and automation tasks, Python is likely calling the shots to handle fetching pages, parsing HTML, and extracting data. Helpfully, popular Python libraries like lxml and parsel make executing XPath queries simple.
Class Selection with lxml
Using lxml, we first need to parse the HTML content into an lxml document:
from lxml import html
tree = html.parse('page.html')We can then run XPath expressions against this document to select elements:
headers = tree.xpath("//div[contains(@class,'header')]")And access properties like text content on the resulting elements:
print(headers[0].text_content())
Put together, here is a full lxml example to extract text from <h1> elements with class "title":
from lxml import html
import requests
page = requests.get('http://example.com')
tree = html.fromstring(page.content)
titles = tree.xpath("//h1[contains(@class,'title')]")
for title in titles:
print(title.text_content())Class Selection with parsel
The parsel library provides a similar API, first loading an HTML document:
from parsel import Selector sel = Selector(text=html_text)
Then using XPath to select elements:
headings = sel.xpath("//h2[contains(@class, 'headline')]")And accessing properties like inner HTML:
for heading in headings: print(heading.get())
Parsel makes it easy to integrate XPath class selection into Scrapy spiders and other Python scraping scripts. Both libraries provide great options for harnessing the power of XPath for HTML document parsing and data extraction.
Comparison to CSS Selectors
For web scraping tasks, we also have the choice to use CSS selectors instead of XPath. CSS offers similar capability to target elements by class attribute. For example, in CSS we could match the “header” class with:
div.header { }Some notable differences:
- CSS can't do partial matching like XPath's
contains() - No explicit exclusion like XPath's
!=operator - More limited logic and functions compared to XPath
- CSS selectors are faster for simple queries
In general, XPath class selection provides more flexibility and power compared to plain CSS.
How Class Selection Differs By Language
XPath isn't the only language that supports targeting elements by class. This table compares some alternatives:
| Language | Class Selection Example |
|---|---|
| XPath | //h1[contains(@class, "title")] |
| CSS | h1.title { } |
| jSoup (Java) | doc.select(".title") |
| BeautifulSoup (Python) | soup.find_all(class_="title") |
| Puppeteer (JS) | page.$('.title') |
XPath stands out with verbose but powerful syntax for matching logic. CSS is simple yet limited. Each programming language offers its own class selection API.
When to Choose Class vs ID/Other Attributes
We've focused exclusively on the class attribute so far. But worth noting XPath can select elements based on any attribute. Some common choices beyond class:
id– Unique identifier for an elementname– Like input names or anchor nameshref– Link URLssrc– Image sourcedata-*– Custom data attributes
In general, class provides the best balance for maintainable and reusable selectors. Some rules of thumb:
- Prefer class for site-wide patterns – Class names consistently applied across pages
- Use ID for unique elements – When targeting a single specific element
- Choose name for form fields – Input names that serve as data keys
- Prefer class over tag names – Rely less on element types
Think about selector re-use and how consistent naming is when deciding between targeting class vs other attributes in XPath.
Real-World Examples
Let's now look at some real-world examples applying class selection to actual web scraping and automation tasks.
Scraping Product Listings
A common web scraping task is extracting products from ecommerce category pages. For example, Target uses consistent class="ListItem__Content" markup to denote product list items:
<article class="ListItem__Content"> <a href="product1-url"...>Product 1</a> </article> <article class="ListItem__Content"> <a href="product2-url"...>Product 2</a> </article>
We can reliably select these with:
products = sel.xpath("//article[contains(@class, 'ListItem__Content')]")This will work across any Target category to extract all products.
Analyzing Search Result Relevance
Evaluating search result relevance often requires identifying specific classes. For example, Google uses classes like LC20lb for top stories and ilX0i for ads in results:
<div class="LC20lb">Top Stories</div> <div class="ilX0i">Ad</div>
We can select these special result types with:
top_stories = sel.xpath("//div[contains(@class, 'LC20lb')]")
ads = sel.xpath("//div[contains(@class, 'ilX0i')]")Analyzing these classes provides insights into result composition and organic vs paid results.
Consistent Element Targeting
Classes also allow targeting common page elements like navigation menus, footers, etc. For example, Twitter's menu component:
<nav class="css-4rbku5"> <!-- menu links --> </nav>
Can be selected on any page with:
menu = sel.xpath("//nav[contains(@class, 'css-4rbku5')]")Reliably accessing repeating components by class makes test automation and site-wide scraping much easier.
Sanitizing Irrelevant Classes
One final example – using class exclusion to remove irrelevant elements. Suppose we want to extract only review text, ignoring things like author info. We can select reviews while excluding some common extra element classes:
reviews = sel.xpath("//div[contains(@class, 'review-text') and not(contains(@class, 'author-name')) and not(contains(@class, 'date'))]"]Excluding classes provides precision when extracting specific content from complex pages. As you can see, class selection powers a wide variety of real-world scraping and automation tasks.
Conclusion
With CSS classes being so ubiquitous, mastering class-based selection gives immense power for targeting relevant content in documents. Hopefully, this guide provided a comprehensive overview of the variety of techniques available for selecting elements by class in XPath. You're now equipped to handle even complex class-matching scenarios with robust XPath selectors.
