How to Find Elements by XPath in Puppeteer?

17 Views

XPath is a powerful query language for selecting elements in HTML and XML documents. When combined with a browser automation tool like Puppeteer, XPath provides a robust way to extract data from web pages.

In this comprehensive guide, you'll learn how to use XPath selectors in Puppeteer to find and extract elements from a web page.

What is XPath?

XPath stands for XML Path Language. It's a syntax for defining parts of an XML document. XPath uses path expressions to navigate through elements and attributes in an XML document. Although XPath was originally designed for XML, it can also be used to query HTML documents. This is because HTML can be represented as a specialized form of XML (XHTML).

XPath expressions look like this:

/html/body/div/p

This XPath will select the <p> element inside <div> inside <body> of an HTML document. Some key things to know about XPath syntax:

Uses / to separate levels in the document hierarchy
No spaces between steps
// does a deep scan to find all matches
[index] specifies element index if multiple matches
[@attribute='value'] filters on attributes

This makes XPath a very versatile querying language. You can craft highly specific XPath selectors to extract just the content you need from a web page.

Finding Elements by XPath with Puppeteer

Puppeteer is a popular Node.js library by Google for controlling headless Chrome. It provides an API for automating actions in Chrome like page navigation, input, and extracting content. Puppeteer uses the DevTools Protocol under the hood to communicate with the browser. The page.$x() method in Puppeteer gives access to using XPath queries.

Here is an example of using page.$x() to find all <p> elements on a page:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com'); 

  const paragraphs = await page.$x('//p');

  console.log(paragraphs.length); // logs number of <p> elements

  await browser.close();
})();

The page.$x() method returns an array of ElementHandle objects that match the XPath selector. You can then interact with these elements using the Puppeteer API. Some key things to note:

$x() will always return matches as an array
Use [@attribute] filters to narrow down matches
Elements must exist in DOM before using $x()
Can use page.evaluate() to extract text/attributes

Now let's go through some more examples of using XPath selectors in Puppeteer.

Matching a Single Element

To extract just one element, use an XPath with specific filters. For example:

const [heading] = await page.$x('/html/body/h1');

This will return the first <h1> element on the page. The [0] extracts the first match from the results array. You can also use [@id="value"] or [@class="value"] in the selector to filter on attributes:

const [submit] = await page.$x('//button[@id="submit"]');

This will find the <button> with id="submit".

Extracting Text from Elements

The evaluate() function runs inside the browser context, allowing you to call DOM properties/methods on the matched element handle. You can simplify this by combining $x() and evaluate():

const [heading] = await page.$x('//h1');

const text = await heading.evaluate(node => node.textContent);

This evaluates the XPath selector directly and returns the result.

Extracting Attributes from Elements

You can also use evaluate() to get attribute values from matched elements:

const [link] = await page.$x('//a');

const href = await link.evaluate(node => node.href);

This will return the href value for the first <a> link on the page. Again this can be shortened with $eval():

const href = await page.$eval('//a', node => node.href);

Finding Multiple Elements

To get multiple elements, remove the [0] from the XPath selector:

const links = await page.$x('//a');

This will return an array of all <a> elements on the page. You can then loop through the matches:

const links = await page.$x('//a');

for (let link of links) {
  const text = await link.evaluate(node => node.textContent);
  console.log(text); 
}

This will log the text content of every link on the page.

Using XPath Axes

XPath includes special axes selectors to find elements relative to other elements:

/parent:: – parent element
/child:: – direct child element
/descendant:: – any descendant under parent
/preceding-sibling:: – previous sibling element
/following-sibling:: – next sibling element

For example, to get all <p> tags inside a <div>:

const paras = await page.$x('//div/child::p');

Or to get the previous sibling of a paragraph:

const [prevElement] = await page.$x('//p[2]/preceding-sibling::*[1]');

These axes selectors provide more options to target elements precisely.

Using XPath Functions

Along with axes selectors, XPath includes a number of built-in functions. For example, contains() lets you filter elements by text content:

const searchResults = await page.$x("//div[contains(., 'search term')]");

This finds <div> elements containing the phrase “search term”. Some other useful XPath functions:

starts-with(., 'text') – match elements starting with text
ends-with(., 'text') – match elements ending with text
position() = 3 – match element by position
last() – select last element in matched nodes

These give you more flexibility in crafting your XPath locators.

Using XPath with page.evaluate()

In addition to using $x() and $eval(), you can also evaluate raw XPath expressions directly:

const firstParaText = await page.evaluate(() => {
  const xpath = document.evaluate("//p[1]", document, null, XPathResult.STRING_TYPE, null);
  return xpath.stringValue;
});

This evaluates the XPath in the browser context and returns the result. The major benefit here is being able to use more complex XPath features like recursion. However, the downside is this will return raw DOM nodes instead of Puppeteer element handles. So you lose access to the Puppeteer API functionality.

Waiting for Elements to Exist

A common gotcha when using XPath in Puppeteer is trying to extract elements before they have loaded. Dynamic pages rendered with JavaScript will not have elements available immediately on page.goto().

Instead, you need to wait for the XPath to match before using $x():

await page.goto('https://example.com');

await page.waitForXPath('//p'); // wait for <p> elements to exist    

const paragraphs = await page.$x('//p');

This will pause execution until at least one <p> is available in the DOM. You can improve reliability by using explicit waits before trying to extract any data:

const { waitFor } = require('puppeteer');

// ...

await page.goto(url);

await waitFor(2000); // wait 2 seconds

const elements = await page.$x(myXPath);

Adding small buffers for network requests and page rendering will prevent flaky mismatches in dynamic pages.

Tips for Robust XPath Selectors

Here are some best practices to create reliable, resilient XPath locators:

Prefer ID or class attributes – [@id="name"] is the most robust locator
Avoid positional indices – //p[1] will break if order changes
Leverage unique text – //h1[text()="Page Title"] won't match wrongly
Utilize ancestors – /html/body/main//p reduces risk of matching the wrong area
Keep it short – More steps mean more fragile. //input[@name="email"] is better than a long path
Use explicit waits – Allow time for XPath to become available before extracting
Plan for staleness – Cache queries and re-find elements if needed

Following these tips will optimize your XPath selectors to be as unique and robust as possible.

Example: Extracting Tabular Data

Let's walk through an full example of using XPath and Puppeteer to extract tabular data from a web page.

<table id="data-table">
  <tr>
    <th>Company</th>
    <th>Contact</th>
    <th>Country</th>
  </tr>
  <tr>
    <td>Alfreds Futterkiste</td> 
    <td>Maria Anders</td>
    <td>Germany</td>
  </tr>
  <tr>
    <td>Berglunds snabbköp</td>
    <td>Christina Berglund</td>
    <td>Sweden</td> 
  </tr>
  ...
</table>

Here are the steps:

Launch Puppeteer and navigate to the page
Wait for table to load by waiting for #data-table to exist
Use $x() to get all <tr> rows inside #data-table
Iterate over <tr> handles, extracting <td> text using $eval()
Print out extracted cell data

const puppeteer = require('puppeteer');

(async () => {

  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://thewebminer.com/demo/table');

  // wait for table to load
  await page.waitForXPath('//*[@id="data-table"]'); 

  // get all rows
  const rows = await page.$x('//*[@id="data-table"]/tbody/tr');

  // extract each cell 
  for(let row of rows) {
    
    const company = await row.$eval('td[1]', node => node.textContent);
    const contact = await row.$eval('td[2]', node => node.textContent);
    const country = await row.$eval('td[3]', node => node.textContent);

    console.log({company, contact, country});

  }

  browser.close();

})();

This provides a complete workflow for leveraging XPath in Puppeteer to scrape structured data tables. The same approach could be adapted to extract pricing data, directory contacts, product listings, etc.

Summary

XPath is an invaluable tool for extracting specific content from HTML and XML documents. When combined with a headless browser tool like Puppeteer, XPath provides a robust way to automate scraping of websites and web apps. With the above techniques, you can leverage XPath to develop reliable scrapers that can extract data from even complex dynamic web pages.