Scrapy 0.15 dropping support for Python 2.5

After a year considering it, we have decided to go ahead and drop support for Python 2.5 in Scrapy.

Starting from 0.15, Scrapy will require Python 2.6 or above.

The main reasons for dropping support for Python 2.5 are:

  1. most people have already moved to 2.6 or 2.7
  2. we are no longer testing it ourselves on Python 2.5, so it's quite possible we release changes incompatible with Python 2.5 by mistake. By the way, this has happened in the past and only disovered after a while, which proves our belief that Python 2.5 is barely used anymore.
  3. having to check compatibility with 2.5 has become a painful, time consuming and error-prone task. We would rather spend that time on improving the framework.
  4. there are some ugly bugs in the standard library of Python 2.5 (mainly in urlparse, urllib & cookielib modules) that we would like to leave behind
  5. there is some boilerplate code to keep backwards compatibiltiy with 2.5 (scrapy.utils.py26 module, optional dependency on simplejson) that we want to remove to keep a cleaner codebase

The code cleanup and the changes haven't been done yet, nor they will happen all at once in a single commit, so Scrapy 0.15 still works on Python 2.5. Not for long however, since we will be making these changes during the remaining development phase of 0.15.

November 14, 2019 In "Crawlera" , "Scrapy" , "Scrapinghub"
November 07, 2019 In "Scrapy" , "SQL" , "Real Estate" , "Matlab"
October 15, 2019 In "Scrapy" , "AutoExtract"
Sign up now

Be the first to know. Gain insights. Make better decisions.

Use web data to do all this and more. We’ve been crawling the web since 2010 and can provide you with web data as a service.

Tell me more


Here we blog about all things related to web scraping and web data.

If you want to learn more about how you can use web data in your company, check out our Data as a Services page for inspiration.

Follow Us

Learn More

Recent Posts