ScrapyRT: Turn Websites Into Real-Time APIs

If you’ve been using Scrapy for any period of time, you know the capabilities a well-designed Scrapy spider can give you.

With a couple lines of code you can design a scalable web crawler and extractor that will automatically navigate to your target website and extract the data you need. Be it e-commerce, article or sentiment data.

The one issue that traditional Scrapy spiders poses however, is the fact that in a lot of cases spiders can take a long time to finish their crawls and deliver their data if it is a large job. Making Scrapy spiders unsuitable for near real-time web scraping. A growing and very useful application for many data aggregation and analysis efforts.

With the growth of data based services and data-driven decision making, end users are increasingly looking for ways to extract data on demand from web pages instead of having to wait for data from large periodic crawls.

And that’s where ScrapyRT comes in…

Enter ScrapyRT

Originally evolving out of a Scrapinghub for Google Summer of Code project in 2014, ScrapyRT (Scrapy Realtime) is an open source Scrapy extension that enables you to control Scrapy spiders with HTTP requests.

Simply send your Scrapy HTTP API a request containing the Scrapy Request Object (with URL and callback as parameters) and the API will return the extracted data by the spider in real-time. No need to wait for the entire crawl to complete.

Usually, spiders are run for long periods of time, and proceed step by step, traversing the web from a starting point and extracting any data that matches their extraction criteria.

This mode of operation is great if you don’t know the location of your desired data. However, if you know the location of the data then there is a huge amount of redundancy if the spider has to complete all the intermediary steps.

ScrapyRT allows you to schedule just one single request with spider, parse it in callback, and get response returned immediately as JSON instead of having the data saved in a database.

By default spider’s start_requests spider method is not executed and the only request that is scheduled with spider is Request generated from API params.

How ScrapyRT Works?

ScrapyRT’s architecture is very simple. It is a web server written in Python Twisted tied with custom Crawler object from Scrapy.

Twisted is one of the most powerful Python asynchronous frameworks, so was a natural choice for ScrapyRT as Twisted works great for asynchronous crawling and Scrapy uses Twisted for all HTTP traffic. Ensuring easy integration with Scrapy.

Once added to your project, ScrapyRT runs as a web service, retrieving data when you make a request containing the URL you want to extract data from and the name of the spider you would like to use.

ScrapyRT will then schedule a request in Scrapy for the URL specified and use the ‘foo’ spider’s parse method as a callback. The data extracted from the page will be serialized into JSON and returned in the response body. If the spider specified doesn’t exist, a 404 will be returned.

The ScrapyRT web server is customizable and modular. You can easily override GET and POST handlers. This means that you can add your own functionality to the project - you can write your own Twisted Resources inheriting from main ScrapyRT handlers, for example you can write code to return response in XML or HTML instead of JSON, and add it to configuration.

One thing to keep in mind is that ScrapyRT was not developed with long crawls in mind.

Remember: after sending a request to ScrapyRT you have to wait for the spider to finish before you get response.

So if the request requires the spider to crawl an enormous site and generates 1 million requests in callback, then ScrapyRT isn’t the best option for you as you will likely have to sit in front of a blank screen waiting for your crawl to finish and return the item.

One possible way of solving this problem would involve modifying ScrapyRT so that it could use websockets or HTTP push notifications - this way API could send items as they arrive in API.

Currently data from spider is returned in response to initial request - so after sending each request you have to wait for the response until data is returned by spider. If you expect that your spider will generate lots of requests in callback but you actually don’t need all of them you can limit the amount of requests by passing max_requests parameter to Scrapy.

If you would like to learn more about ScrapyRT or contribute to the open source project, then check out the ScrapyRT documentation and GitHub repository.

Your Data Extraction Needs

At Scrapinghub we specialize in turning unstructured web data into structured data. If you have a need to start or scale your web scraping projects then our Solution Architecture team is available for a free consultation, where we will evaluate and develop the architecture for a data extraction solution to meet your data and compliance requirements.

At Scrapinghub we always love to hear what our readers think of our content and would be more than interested in any questions you may have. So please, leave a comment below with your thoughts and perhaps consider sharing what you are working on right now!