Looking Back at 2013

This time last year Pablo and I were chatting about the previous year and what to expect in 2013. I noticed that our team had almost doubled in size in the previous year and we wondered could that possibly continue in 2013?

It turns out it did! We went from 20 team members to almost 40 and the number of projects, customers, etc. all doubled. We've become even more distributed as we've grown, covering 19 countries:

 

Join us and put your country on the map! Click for interactive map of current team

 

What the numbers don't show is that these new hires have become indispensable members of the team and it's hard to imagine working without them.

Over 2 billion web pages were scraped using our platform in 2013! We're pleased this was done politely, without receiving a single complaint. More users participated in our beta and we saw a 4x growth in our platform usage[1]:

platform_blog_chart

This would not have been possible without our new architecture, for reasons explained in our most popular blog post of 2013.

Our open source contributions increased - we had 2 large stable Scrapy releases (0.18 and 0.20) and started several new projects (e.g. splash, webstruct, webpagerscrapyjs and loginform). Scrapy usage (github stars, forks, etc.) doubled again this year and is currently number 12 in github's trending python repositories this month. We have more exciting releases planned, such as a new open source annotation UI for Autoscraping and Scrapy 1.0!

We are very proud of our team and what we have achieved this year. We would like to thank all our customers and supporters. Happy New Year everyone!


  1. These numbers exclude crawls that do not store data on our servers. Some crawls use S3 or other external data storage
November 22, 2018 In "Scrapinghub" , "Lisbon 2018"
June 19, 2018 In "Scrapinghub"
June 07, 2018 In "Alternative Financial Data" , "Scrapinghub"
Scrapinghub

Be the first to know. Gain insights. Make better decisions.

Use web data to do all this and more. We’ve been crawling the web since 2010 and can provide you with web data as a service.

Tell me more

Welcome

Here we blog about all things related to web scraping and web data.

If you want to learn more about how you can use web data in your company, check out our Data as a Services page for inspiration.

Learn More

Recent Posts