Proxy Management: Should I Build My Proxy Infrastructure In-House Or Use AN Off-The-Shelf Proxy Solution?

Proxy management is the thorn in the side of most web scrapers. Without a robust and fully featured proxy infrastructure, you will often experience constant reliability issues and hours spent putting out proxy fires.

A situation no web scraping professional wants to deal with. Us web scrapers are interested in extracting and using web data, not managing proxies.

In this article, we’re going to...

A Sneak Peek Inside Crawlera: The World’s Smartest Web Scraping Proxy Network

“How does Scrapinghub Crawlera work?” is the most common question we get asked from customers who after struggling for months (or years) with constant proxy issues, only to have them disappear completely when they switch to Crawlera. 

Today we’re going to give you a behind the scenes look at Crawlera so you can see for yourself why it is the world’s smartest web scraping proxy network and the...

Why We Created Crawlera? The World’s Smartest Web Scraping Proxy Network

Let’s face it, managing your proxy pool can be an absolute pain and the biggest bottleneck to the reliability of your web scraping! 

Nothing annoys developers more than crawlers failing because their proxies are continuously being banned.

The Rise of Web Data in Hedge Fund Decision Making & The Importance of Data Quality

Over the past few years, there has been an explosion in the use of alternative data sources in investment decision making in hedge funds, investment banks and private equity firms.

These new data sources, collectively known as “alternative data”, have the potential to give firms a crucial informational edge in the market, enabling them to generate alpha.

The Predictive Power of Web Scraped Product Data For Institutional Investors: A GoPro Case Study

Investors understand the importance of high-quality information. It minimizes risk, empowers decision-making, and enables investors of all sizes to obtain alpha - like the old adage, knowing is often half the battle.

Knowing this, alternative data providers wield vast, untraditional datasets derived from hundreds of millions of sources, not only enabling asset managers to consistently obtain...

The Challenges E-Commerce Retailers Face Managing Their Web Scraping Proxies

These days web scraping amongst big e-commerce companies is ubiquitous due to the advantages that data-based decision making can bring to remain competitive in such a tight-margin business.

E-commerce companies are increasingly using web data to fuel their competitor research, dynamic pricing and new product research.

For these e-commerce sites, their most important considerations are: the ...

Looking Back at 2018

What a year 2018 has been for Scrapinghub!!

It’s hard to know where to start…

This year has seen tremendous growth at Scrapinghub, setting us up to have a great 2019.

Here are some of the highlights of 2018…

Do What is Right Not What is Easy!

I was recently invited to speak at the IAPP Europe Data Protection Congress in Brussels about web scraping and GDPR. The panel also included Claire François of Hunton Andrews Kurth and Peter Brown from the Information Commissioner’s Office (ICO). For more information you can check out my blog about this topic GDPR Compliance for Web Scrapers: The Step-by-Step Guide.

Shubber GetTogether 2018

It’s hard to believe our annual Shubber GetTogether is already over.

Data Quality Assurance for Enterprise Web Scraping

When it comes to web scraping, one key element is often overlooked until it becomes a big problem.

That is data quality.

Getting consistent high quality data when scraping the web is critical to the success of any web scraping project, particularly when scraping the web at scale or extracting mission critical data where accuracy is paramount.

Data quality can be the difference between a...

Sign up now

Be the first to know. Gain insights. Make better decisions.

Use web data to do all this and more. We’ve been crawling the web since 2010 and can provide you with web data as a service.

Tell me more

Welcome

Here we blog about all things related to web scraping and web data.

If you want to learn more about how you can use web data in your company, check out our Data as a Services page for inspiration.

Follow Us

Learn More

Recent Posts