Issues with urllib and urllib2

In the early months of 2011, Niels Heinen of the Google Security Team reported a ‘Redirect vulnerability in urllib/urllib2 ( http://bugs.python.org/issue11662). The patch to fix this issue was committed within a month of it being reported.

Post-commit, more details were shared about the nature of the issue and the fix in the official blog, Python Inside (see http://bit.ly/urllibSecVulFix). Another bug in urllib was reported in 2013, ( http://bugs.python.org/issue17322); however, this bug’s category was ‘normal’ unlike ‘11662’ which was classed as a ‘release blocker’. However both are fixed now.

It should be noted that when using urllib or urllib2, SSL verification isn’t possible when ‘urlopen’ is invoked. For anyone serious about security this is surely a concern. This is where the ‘requests’ module scores and stands apart from the others. The list of supported features in the request module is quite comprehensive, such as HTTP(S) Proxy Support, Connection Timeouts and Basic/Digest Authentication etc. For the complete list and overview check the ‘requests’ module ( http://bit.ly/PythonRequests).

Enthusiastic members of the Python community have come up with solutions to overcome or fix the shortcomings of both the urllib and urllib2 modules. One group, in particular has tried to patch urllib2 to use CONNECT for https proxies ( https://pypi.python.org/pypi/httpsproxy_urllib2). Meanwhile, urllib3 (see https://pypi.python.org/pypi/urllib3) encompasses many critical features, such as thread safety, connection pooling and proxy support.