Crawlera is a smart downloader that allows you to crawl and scrape websites responsibly. It rotates IP addresses and keeps track of which ones have been blocked by websites, ensuring that your crawls continue uninterrupted. Since Crawlera has always been a mainstay of Scrapinghub, we wanted to revamp its presentation to help you crawl the web and extract data more effectively.
With that in mind, say hello to the dashboard that allows you to visualize how you are using Crawlera, to examine what sites and specific URLs you are targeting, and to manage multiple accounts.
Read on as we walk you through the various features of the dashboard and show you how best to use it.
The main benefits of the dashboard include seeing which websites are being crawled the most and understanding the responses you’re getting via Crawlera overall and for each individual account. You can also create and manage different Crawlera accounts from your dashboard, a feature which was not previously available.
Having all of your accounts in one place allows you to more accurately manage and monitor your Crawlera use.
In the past you could only see two global usage graphs: the number of requests you’ve made per day for the last 30 days and per month for the last 12 months. Now you can also view the most recent requests performed by Crawlera in any of your accounts as well as filter the state of each request (succeeded, failed, banned) per website. The graphs are also more comprehensive and representative of your Crawlera activity.
Plans for the future
In the future, we plan to develop support for managing the regions that you use for your Crawlera account(s) as well tools to understand how Crawlera impacts your crawls. We’ll also provide additional resources for our Enterprise clients, such as fine-tuned management of a dedicated pool of IP addresses.
Resources to Get Started
Here is the official Crawlera documentation to help you get started. We will also be rolling out a tutorial video, so keep your eyes peeled in the coming weeks.
To our longtime Crawlera users, we hope that you enjoy the enhanced features of the Crawlera Dashboard. To users new to Crawlera, we’re thrilled that you’re joining us during the release of this new interface.
Please let us know what you think of the dashboard and feel free to reach out with further suggestions on Twitter. Happy scraping!