Scraping the Steam Game Store with Scrapy

Scraping the Steam Game Store with Scrapy

This is a guest post from the folks over at Intoli, one of the awesome companies providing Scrapy commercial support and longtime Scrapy fans. Introduction The Steam game store is home to more than ten thousand games and just shy of four million user-submitted reviews. While all kinds of Steam data are available either through official APIs or other bulk-downloadable data dumps, I could not find a way to download the full review dataset. 1 If you want to perform your own…

Read More Read More

Do Androids Dream of Electric Sheep?

Do Androids Dream of Electric Sheep?

Comments 0 Comment

It got very easy to do Machine Learning: you install a ML library like scikit-learn or xgboost, choose an estimator, feed it some training data, and get a model which can be used for predictions. Ok, but what’s next? How would you know if it works well? Cross-validation! Good! How would you know that you haven’t messed up the cross validation? Are there data leaks? If the quality is not good enough, how to improve it? Are there data preprocessing…

Read More Read More

Deploy your Scrapy Spiders from GitHub

Deploy your Scrapy Spiders from GitHub

Up until now, your deployment process using Scrapy Cloud has probably been something like this: code and test your spiders locally, commit and push your changes to a GitHub repository, and finally deploy them to Scrapy Cloud using shub deploy. However, having the development and the deployment processes in isolated steps might bring you some issues, such as unversioned and outdated code running in production. The good news is that, from now on, you can have your code automatically deployed…

Read More Read More

Looking Back at 2016

Looking Back at 2016

Comments 1 comment

We started 2016 with an eye on blowing 2015 out of the water. Mission accomplished. Together with our users, we crawled more in 2016 than the rest of Scrapinghub’s history combined: a whopping 43.7 billion web pages, resulting in 70.3 billion scraped records! Great work everyone! In the what follows, we’ll give you a whirlwind tour of what we’ve been up to in 2016, along with a quick peek at what you can expect in 2017. Platform Scrapy Cloud It’s…

Read More Read More

How to Increase Sales with Online Reputation Management

How to Increase Sales with Online Reputation Management

Comments 1 comment

One negative review can cost your business up to 22% of its prospects. This was one of the sobering findings in a study highlighted on Moz last year. With over half of shoppers rating reviews as important in their buying decision, no company large or small can afford to ignore stats like these – let alone the reviews themselves. In what follows I’ll let you in on how web scraping can help you stay on top. What is online reputation…

Read More Read More

How to Build your own Price Monitoring Tool

How to Build your own Price Monitoring Tool

Comments 1 comment

Computers are great at repetitive tasks. They don’t get distracted, bored, or tired. Automation is how you should be approaching tedious tasks that are absolutely essential to becoming a successful business or when carrying out mundane responsibilities. Price monitoring, for example, is a practice that every company should be doing, and is a task that readily lends itself to automation. In this tutorial, I’ll walk you through how to create your very own price monitoring tool from scratch. While I’m…

Read More Read More

How You Can Use Web Data to Accelerate Your Startup

How You Can Use Web Data to Accelerate Your Startup

Comments 1 comment

In just the US alone, there were 27 million individuals running or starting a new business in 2015. With this fiercely competitive startup scene, business owners need to take advantage of every resource available, especially given a high probability of failure. Enter web data. Web data is abundant and those who harness it can do everything from keeping an eye on competitors to ensuring customer satisfaction. Web Data and Web Scraping You can get web data through a process called…

Read More Read More

An Introduction to XPath: How to Get Started

An Introduction to XPath: How to Get Started

Comments 1 comment

XPath is a powerful language that is often used for scraping the web. It allows you to select nodes or compute values from an XML or HTML document and is actually one of the languages that you can use to extract web data using Scrapy. The other is CSS and while CSS selectors are a popular choice, XPath can actually allow you to do more. With XPath, you can extract data based on text elements’ contents, and not only on…

Read More Read More

Why Promoting Open Data Increases Economic Opportunities

Why Promoting Open Data Increases Economic Opportunities

Comments 0 Comment

During the 2016 Collision Conference held in New Orleans, our Content Strategist Cecilia Haynes interviewed conference speaker Dr. Tyrone Grandison. At the time of the interview, he was the Deputy Chief Data Officer at the U.S. Department of Commerce. Tyrone is currently the Chief Information Officer for the Institute for Health Metrics and Evaluation. Dr. Tyrone Grandison Coming fresh off his talk on “Data science, apps and civic responsibility“, Cecilia was thrilled to chat with Tyrone all about the democratization of…

Read More Read More

Interview: How Up Hail uses Scrapy to Increase Transparency

Interview: How Up Hail uses Scrapy to Increase Transparency

Comments 0 Comment

During the 2016 Collision Conference held in New Orleans, Scrapinghub Content Strategist Cecilia Haynes had the opportunity to interview the brains and the brawn behind Up Hail, the rideshare comparison app. Avi Wilensky is the Founder of Up Hail Avi sat down with Cecilia and shared how he and his team use Scrapy and web scraping to help users find the best rideshare and taxi deals in real time. Fun fact, Up Hail was named one of Mashable’s 11 Most…

Read More Read More