Scrapinghub: A Remote Working Success Story

When Scrapinghub came into the world in 2010, one thing we wanted was for it to be a company which could be powered by a global workforce, each individual working remotely from anywhere in the world. Reduced commuting time and as a consequence increased family time were the primary reasons for this. In Uruguay, Pablo was commuting long distances to do work which realistically could have be done...

Why we moved to Slack

We are veterans in the chat group arena. We have been using one form of another since we started Scrapinghub in 2010 and I've been personally using corporate group chats since 2004. We started Scrapinghub using our own hosted version of ejabberd, then moved to HipChat in 2013 and we just finished moving to Slack. Thanks to Slack's migration tools, the process went pretty smoothly. In this post...

The History of Scrapinghub

Joanne O’Flynn meets with Pablo Hoffman and Shane Evans to find out what inspired them to set up web crawling company Scrapinghub.

Skinfer: A Tool for Inferring JSON Schemas

Imagine that you have a lot of samples for a certain kind of data in JSON format. Maybe you want to have a better feel of it, know which fields appear in all records, which appear only in some and what are their types. In other words, you want to know the schema for the data that you have.

Handling JavaScript in Scrapy with Splash

A common roadblock when developing spiders is dealing with sites that use a heavy amount of JavaScript. Many modern websites run entirely on JavaScript, and require scripts to be run in order for the page to render properly. In many cases, pages also present modals and other dialogues that need to be interacted with to show the full page. In this post we’re going to show you how you can use...