Python 3 is Coming to Scrapy


Scrapy with beta Python 3 support is finally here! Released through Scrapy 1.1.0rc1, this is the result of several months of hard work on the part of the Scrapy community and Scrapinghub engineers.

This is a huge milestone for all you Scrapy users (and those who haven’t used Scrapy due to the lack of Python 3). Scrapy veterans and new adopters will soon be able to move their entire stack to Python 3 once the release becomes stable. Keep in mind that since this a release candidate, it is not ready to be used in production.


Benefits of Scrapy 1.1.0rc1

In addition to Python 3 support, Scrapy 1.1.0rc1 includes a lot of enhancements and bug fixes:

  • Scrapy now respects robots.txt by default
  • Item loaders now support nested loaders
  • Anonymous S3 connections are now allowed
  • XML node names containing dots are now accepted
  • A new response.text attribute to get the response body as unicode
  • Selectors were extracted from Scrapy into a separate library called parsel, allowing you to benefit from using the power of Scrapy selectors in your own projects

Python 3 Limitations

This beta release includes some limitations for Python 3 (which will not affect Python 2 users):

  • Scrapy doesn’t work yet in Windows with Python 3
  • S3 downloads are not supported yet
  • Sending emails is not supported yet
  • Telnet is not supported yet
  • FTP download handler is not supported yet
  • Some non-ASCII URLs may fail in Scrapy with Python 3

If you want the full details of all the benefits of this new release, check out the release notes.

Test it Out

And now it’s your turn, Scrapy users! The sooner this is battle-tested, the sooner a stable release comes your way, so please give it a try by installing or upgrading Scrapy via pip:

$ pip install scrapy==1.1.0rc1

Wrap Up

We appreciate any feedback or bug reports as they will help a lot in making Python 3 support with Scrapy stable. If you run into any problems, please let us know via the issue tracker. We’re looking forward to releasing the official Scrapy 1.1, so get to experimenting!


November 14, 2019 In "Crawlera" , "Scrapy" , "Scrapinghub"
November 07, 2019 In "Scrapy" , "SQL" , "Real Estate" , "Matlab"
October 17, 2019 In "Scrapinghub" , "Machine Learning" , "Extract Summit" , "Web Data Extraction Summit" , "AI"
Web Scraping, Releases, Scrapy, python 3, Scrapinghub, scrapy, web crawling
Sign up now

Be the first to know. Gain insights. Make better decisions.

Use web data to do all this and more. We’ve been crawling the web since 2010 and can provide you with web data as a service.

Tell me more


Here we blog about all things related to web scraping and web data.

If you want to learn more about how you can use web data in your company, check out our Data as a Services page for inspiration.

Follow Us

Learn More

Recent Posts