How To Scrape The Web Without Getting Blocked

Web scraping is when you extract data from the web and put it in a structured format. Getting structured data from publicly available websites and pages should not be an issue as everybody with an internet connection can access these websites. You should be able to structure it as well. In reality though, it’s not that easy.

Free Web Scraping Webinar: How To Scale With Proxies

News Web Data Extraction to Predict Irish Election Results

Can pre-election news coverage of political parties predict the trend of the elections?

On February 9th, 2020, Ireland elected a new parliament. Prior to the elections, the political parties invested a lot of time, money and energy to get their political message to the people. A lot of research goes into selecting the right platform and the right medium.

Introducing Crawlera free trials & new plans

One of the biggest pain points we’ve heard from our Crawlera customers last year is the inconvenience of having to jump from one Crawlera plan to another, when more requests are needed in a month. For this reason, we have been working on rethinking our Crawlera plans to better accommodate these cases and be more flexible with customers that have variable crawling requirements from month to...

Best Recruitment Tips: How to Scout Top Talent

Attracting top talent is essential for the success and growth of a company. The majority of employers will agree that finding the best talent is just as hard as it is important. Which is why, rather than waiting for the right candidate to magically fall into your lap, it's time for employers to turn towards the untapped power of web scraped recruitment data.

Looking Back at 2019

2019 was an exciting year for Scrapinghub. We created things we have never created before and did things nobody in our industry had ever done before. Let’s revisit what happened in 2019!

How to use a proxy in Puppeteer

Puppeteer is a high-level API for headless chrome. It’s one of the most popular tools to use for web automation or web scraping in Node.js. In web scraping, many developers use it to handle javascript rendering and web data extraction. In this article, we are going to cover how to set up a proxy in Puppeteer and what your options are if you want to rotate proxies.

...

How to leverage alternative data in asset management

Whether you are managing a hedge fund trying to find innovative sources of alpha or are an analyst looking to future proof your company’s financial investments, as big data continues to disrupt the investment research landscape, getting on top of these alternative datasets as early as possible is the key to capturing the immense alpha left in this data.

Building Blocks of an Unstoppable Web Scraping Infrastructure

More and more businesses leverage the power of web scraping. Extracting data from the web is becoming popular. But it doesn't mean that the technical challenges are gone. Building a sustainable web scraping infrastructure takes expertise and experience. Here, at Scrapinghub we scrape 9 billion pages per month. In this article, we are going to summarize what the essential elements of web...

Backconnect Proxy: Explanation & Comparison To Other Proxies

Scaling up your web scraping project is not an easy task. Adding proxies is one of the first actions you will need to take. You will need to manage a healthy proxy pool to avoid bans. There are a lot of proxy services/providers, each having a whole host of different types of proxies. In this blog post, you are going to learn how backconnect proxies work and when you should use them.

...

4 Sectors That Benefited Most from Business Intelligence Software

Data moves around the marketplace. It can be sourced internally or externally and collected from vendors, manufacturers, retailers, wholesalers, consumers, and other players in the marketplace. This data is then processed and used by businesses in making insights and decisions regarding new business ventures, product ideas, conflict resolution, and process improvement.