Extracting schema.org Microdata Using Scrapy Selectors and XPath

Extracting schema.org Microdata Using Scrapy Selectors and XPath

EDIT: 2015-10-30

We have released an lxml-based version of this code as an open source library called extruct. Source code is on Github, and the package is available on PyPI. Enjoy!


Web pages are full of data, that is what web scraping is mostly about. But often you want more than data, you want meaning. Microdata markup embedded in HTML source helps machines understand what the pages are about: contact information, product reviews, events etc.

Web authors have several ways to add metadata to their web pages: HTML “meta” tags, microformats, social media meta tags (Facebook’s Open Graph protocol, Twitter Cards).

And then there is Schema.org, an initiative from big players in the search space (Yahoo!, Google, Bing and Yandex) to “create and support a common set of schemas for structured data markup on web pages.”

Here are some stats from Web Data Commons project on the use of microdata in the Common Crawl web corpus:

In summary, we found structured data within 585 million HTML pages out of the 2.24 billion pages contained in the crawl (26%).

Let’s focus on Schema.org syntax for the rest of this article. The markup looks like this (example from schema.org “Getting started” page):

<div itemscope itemtype ="http://schema.org/Movie">
 
 
 
<h1 itemprop="name">Avatar</h1>
 
 
 
  <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span>
  <span itemprop="genre">Science fiction</span>
  <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
</div>

Let’s assume you want to extract this microdata and get the movie item from the snippet above.

Using Scrapy Selector let’s first loop on elements with an itemscope attribute (this represents a container for an item’s properties) using .//*[@itemscope], and for each item, get all properties, i.e. elements that have an itemprop attribute:

>>> from scrapy.selector import Selector
>>> selector = Selector(text="""
...


<div itemscope itemtype ="
http://schema.org/Movie
">
...


<h1 itemprop="name">Avatar</h1>
 
 

...   <span>Director: <span itemprop="director">James Cameron</span> (born August 16, 1954)</span>
...   <span itemprop="genre">Science fiction</span>
...   <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
... </div>
 
 
 
""", type="html")
>>> selector.xpath('.//*[@itemscope]')
[<Selector xpath='.//*[@itemscope]' data=u'
 
 
<div itemscope itemtype="http://schema.o'>]
>>> 

Let’s print out more interesting stuff, like the item’s itemtype and the properties’ values (the text representation of their HTML element):

>>> for item in selector.xpath('.//*[@itemscope]'):
...     print item.xpath('@itemtype').extract()
...     for property in item.xpath('.//*[@itemprop]'):
...         print property.xpath('@itemprop').extract(),
...         print property.xpath('string(.)').extract()
... 
[u'http://schema.org/Movie']
[u'name'] [u'Avatar']
[u'director'] [u'James Cameron']
[u'genre'] [u'Science fiction']
[u'trailer'] [u'Trailer']
>>> 

Hm. The value for trailer isnt that interesting. We should have selected the href of <a href="../movies/avatar-theatrical-trailer.html">Trailer</a>.

For the following extended example markup, taken from Wikipedia,

<div itemscope itemtype="http://schema.org/Movie">
 
 
 
<h1 itemprop="name">Avatar</h1>
 
 
 
 
 
 
<div itemprop="director" itemscope itemtype="http://schema.org/Person">
  Director: <span itemprop="name">James Cameron</span> 
(born <time itemprop="birthDate" datetime="1954-08-16">August 16, 1954</time>)
  </div>
 
 
 
  <span itemprop="genre">Science fiction</span>
  <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
</div>

we see it’s not always href attributes that are interesting, but datetime perhaps, or src for images, or content for meta element like “… any attribute really.

Therefore we need to get itemprop text content and elements attributes (we use the @* XPath expression for that).

Let’s do that on the 2nd HTML snippet:

>>> selector = Selector(text="""
...
 
 
<div itemscope itemtype="
http://schema.org/Movie
">
...
 
 
<h1 itemprop="name">Avatar</h1>
 
 
 
...
 
 
<div itemprop="director" itemscope itemtype="
http://schema.org/Person
">
...   Director: <span itemprop="name">James Cameron</span> 
... (born <time itemprop="birthDate" datetime="1954-08-16">August 16, 1954</time>)
...   </div>
 
 
 
...   <span itemprop="genre">Science fiction</span>
...   <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">Trailer</a>
... </div>
 
 
 
""", type="html")
>>> for item in selector.xpath('.//*[@itemscope]'):
...     print item.xpath('@itemtype').extract()
...     for property in item.xpath('.//*[@itemprop]'):
...         print property.xpath('@itemprop').extract(),
...         print property.xpath('string(.)').extract(),
...         print property.xpath('@*').extract()
... 
[u'http://schema.org/Movie']
[u'name'] [u'Avatar'] [u'name']
[u'director'] [u'n  Director: James Cameron n(born August 16, 1954)n  '] [u'director', u'', u'http://schema.org/Person']
[u'name'] [u'James Cameron'] [u'name']
[u'birthDate'] [u'August 16, 1954'] [u'birthDate', u'1954-08-16']
[u'genre'] [u'Science fiction'] [u'genre']
[u'trailer'] [u'Trailer'] [u'../movies/avatar-theatrical-trailer.html', u'trailer']
[u'http://schema.org/Person']
[u'name'] [u'James Cameron'] [u'name']
[u'birthDate'] [u'August 16, 1954'] [u'birthDate', u'1954-08-16']
>>> 

Wait a minute! We’re getting only attribute values, not attribute names (and itemprop attribute twice for that matter, once for @itemprop and once for @*).

To get names of attributes, one cannot apparently use name() on an attribute… BUT you can do name(@*[i]), i being the attribute position:

>>> for item in selector.xpath('.//*[@itemscope]'):
...     print "Item:", item.xpath('@itemtype').extract()
...     for property in item.xpath('.//*[@itemprop]'):
...         print "Property:",
...         print property.xpath('@itemprop').extract(),
...         print property.xpath('string(.)').extract()
...         for position, attribute in enumerate(property.xpath('@*'), start=1):
...             print "attribute: name=%s; value=%s" % (
...                 property.xpath('name(@*[%d])' % position).extract(),
...                 attribute.extract())
...         print
...     print
... 
Item: [u'http://schema.org/Movie']
Property: [u'name'] [u'Avatar']
attribute: name=[u'itemprop']; value=name
 
Property: [u'director'] [u'n  Director: James Cameron n(born August 16, 1954)n  ']
attribute: name=[u'itemprop']; value=director
attribute: name=[u'itemscope']; value=
attribute: name=[u'itemtype']; value=http://schema.org/Person
 
Property: [u'name'] [u'James Cameron']
attribute: name=[u'itemprop']; value=name
 
Property: [u'birthDate'] [u'August 16, 1954']
attribute: name=[u'itemprop']; value=birthDate
attribute: name=[u'datetime']; value=1954-08-16
 
Property: [u'genre'] [u'Science fiction']
attribute: name=[u'itemprop']; value=genre
 
Property: [u'trailer'] [u'Trailer']
attribute: name=[u'href']; value=../movies/avatar-theatrical-trailer.html
attribute: name=[u'itemprop']; value=trailer
 
 
Item: [u'http://schema.org/Person']
Property: [u'name'] [u'James Cameron']
attribute: name=[u'itemprop']; value=name
 
Property: [u'birthDate'] [u'August 16, 1954']
attribute: name=[u'itemprop']; value=birthDate
attribute: name=[u'datetime']; value=1954-08-16
 
 
>>> 

There’s still something wrong, right? Duplicate properties.

It’s because the 2nd HTML snippet is using embedded items. Indeed, James Cameron is the director of “Avatar”, but he’s also a Person (yes, he is!). That’s why the markup says <div> with an itemscope attribute.

How can we fix that, only selecting properties at the current scope, and leave nested properties when we reach the nested item?

Well, it happens that Scrapy Selectors support some EXSLT extensions, notably the sets operations. Here, we’ll use set:difference: we’ll select itemprops under the current itemscope, and then exclude those that are themselves children of another itemscope, under the current one.

>>> for item in selector.xpath('.//*[@itemscope]'):
...     print "Item:", item.xpath('@itemtype').extract()
...     for property in item.xpath(
...             """set:difference(.//*[@itemprop],
...                               .//*[@itemscope]//*[@itemprop])"""):
...         print "Property:", property.xpath('@itemprop').extract(),
...         print property.xpath('string(.)').extract()
...         for position, attribute in enumerate(property.xpath('@*'), start=1):
...             print "attribute: name=%s; value=%s" % (
...                 property.xpath('name(@*[%d])' % position).extract(),
...                 attribute.extract())
...         print
...     print
... 
Item: [u'http://schema.org/Movie']
Property: [u'name'] [u'Avatar']
attribute: name=[u'itemprop']; value=name
 
Property: [u'director'] [u'n  Director: James Cameron n(born August 16, 1954)n  ']
attribute: name=[u'itemprop']; value=director
attribute: name=[u'itemscope']; value=
attribute: name=[u'itemtype']; value=http://schema.org/Person
 
Property: [u'genre'] [u'Science fiction']
attribute: name=[u'itemprop']; value=genre
 
Property: [u'trailer'] [u'Trailer']
attribute: name=[u'href']; value=../movies/avatar-theatrical-trailer.html
attribute: name=[u'itemprop']; value=trailer
 
 
Item: [u'http://schema.org/Person']
Property: [u'name'] [u'James Cameron']
attribute: name=[u'itemprop']; value=name
 
Property: [u'birthDate'] [u'August 16, 1954']
attribute: name=[u'itemprop']; value=birthDate
attribute: name=[u'datetime']; value=1954-08-16
 
 
>>> 

But wouldn’t it be nice to keep a reference to the Person item as property of the Movie director itemprop? We’d need to uniquely identify items in the markup, maybe the position of each itemscope element, or its number.

You can use XPath count() function for that, counting other itemscopes that are siblings before (preceding::*[@itemscope]) or ancestors of the current one (ancestor::*[@itemscope]):

>>> for item in selector.xpath('.//*[@itemscope]'):
...     print "Item:", item.xpath('@itemtype').extract()
...     print "ID:", item.xpath("""count(preceding::*[@itemscope])
...                              + count(ancestor::*[@itemscope])
...                              + 1""").extract()
... 
Item: [u'http://schema.org/Movie']
ID: [u'1.0']
Item: [u'http://schema.org/Person']
ID: [u'2.0']
>>> 

Here’s a cleaned up example routine, referencing items by their ID when itemprops are also itemscopes:

>>> import pprint
>>> items = []
>>> for itemscope in selector.xpath('//*[@itemscope][@itemtype]'):
...     item = {"itemtype": itemscope.xpath('@itemtype').extract()[0]}
...     item["item_id"] = int(float(itemscope.xpath("""count(preceding::*[@itemscope])
...                                                  + count(ancestor::*[@itemscope])
...                                                  + 1""").extract()[0]))
...     properties = []
...     for itemprop in itemscope.xpath("""set:difference(.//*[@itemprop],
...                                                       .//*[@itemscope]//*[@itemprop])"""):
...         property = {"itemprop": itemprop.xpath('@itemprop').extract()[0]}
...         if itemprop.xpath('@itemscope'):
...             property["value_ref"] = {
...                 "item_id": int(float(itemprop.xpath("""count(preceding::*[@itemscope])
...                                                      + count(ancestor::*[@itemscope])
...                                                      + 1""").extract()[0]))
...             }
...         else:
...             value = itemprop.xpath('normalize-space(.)').extract()[0]
...             if value:
...                 property["value"] = value
...         attributes = []
...         for index, attribute in enumerate(itemprop.xpath('@*'), start=1):
...             propname = itemprop.xpath('name(@*[%d])' % index).extract()[0]
...             if propname not in ("itemprop", "itemscope"):
...                 attributes.append((propname, attribute.extract()))
...         if attributes:
...             property["attributes"] = dict(attributes)
...         properties.append(property)
...     item["properties"] = properties
...     items.append(item)
... 
>>> pprint.pprint(items)
[{'item_id': u'1.0',
  'itemtype': u'http://schema.org/Movie',
  'properties': [{'itemprop': u'name', 'value': u'Avatar'},
                 {'attributes': {u'itemtype': u'http://schema.org/Person'},
                  'itemprop': u'director',
                  'value_ref': {'item_id': u'2.0'}},
                 {'itemprop': u'genre', 'value': u'Science fiction'},
                 {'attributes': {u'href': u'../movies/avatar-theatrical-trailer.html'},
                  'itemprop': u'trailer',
                  'value': u'Trailer'}]},
 {'item_id': u'2.0',
  'itemtype': u'http://schema.org/Person',
  'properties': [{'itemprop': u'name', 'value': u'James Cameron'},
                 {'attributes': {u'datetime': u'1954-08-16'},
                  'itemprop': u'birthDate',
                  'value': u'August 16, 1954'}]}]
>>> 

This dict output is not quite what W3C details in the microdata specs, but it’s close enough to leave the rest to you as an exercise 😉

You can find a Github Gist of the above code here.

Be the first to know. Gain insights. Make better decisions.

Use web data to do all this and more. We’ve been crawling the web since 2010 and can provide you with web data as a service.

Tell me more

2 thoughts on “Extracting schema.org Microdata Using Scrapy Selectors and XPath

Leave a Reply

Your email address will not be published. Required fields are marked *