Python File Read Write with Urllib2

Update: Looking for how to download files using Python3 and urllib? Check out my post here .

While checking out my great stats on Lijit recently, I started to see a pattern. I was able to determine that a large part of my Re-Search(search engine) traffic was coming from a post I did about reading and writing to files in Python back in  2005. In an effort to shamelessly attract more traffic on this topic, I have decided to flesh this post out a bit.

A common task that I run into both in my work life as well as my personal life, revolves around programmatically downloading content from the interwebs. This little code example will illustrate how to use urllib to download a file, and write/save the file contents locally. You may be saying to yourself “Self, can’t I do this in my favorite web browser??” . The answer is “YES”, but it’s a pain in the ass if you have more than 5 files you want to download.

Assume there are a set of images on your favorite website, and they are all named  image1.jpg,image2.jpg,image3.jpg, etc. Now imagine there are 50 images using this naming convention.How do you download them all using python , without struggling to do it one image at a time in your browser? Look below!

python

# Let's create a function that downloads a file, and saves it locally.
# This function accepts a file name, a read/write mode(binary or text),
# and the base url.

def stealStuff(file_name,file_mode,base_url):
	from urllib2 import Request, urlopen, URLError, HTTPError
	
	#create the url and the request
	url = base_url + file_name
	req = Request(url)
	
	# Open the url
	try:
		f = urlopen(req)
		print "downloading " + url
		
		# Open our local file for writing
		local_file = open(file_name, "w" + file_mode)
		#Write to our local file
		local_file.write(f.read())
		local_file.close()
		
	#handle errors
	except HTTPError, e:
		print "HTTP Error:",e.code , url
	except URLError, e:
		print "URL Error:",e.reason , url


# Set the range of images to 1-50.It says 51 because the 
# range function never gets to the endpoint.
image_range = range(1,51)

# Iterate over image range
for index in image_range:
	
	base_url = 'http://www.techniqal.com/'
	#create file name based on known pattern 
	file_name =  str(index) + ".jpg"
	# Now download the image. If these were text files, 
	# or other ascii types, just pass an empty string 
	# for the second param ala stealStuff(file_name,'',base_url)
	stealStuff(file_name,"b",base_url)

That’s it. It not only reports on any errors it encountered while downloading, but think of all of the time you just saved… Really though, how important is your time to you if you’re reading this blog???

 

18 thoughts on “Python File Read Write with Urllib2

  1. Hey daonb,

    the only reason I could see an issue with large files, is if the socket connection times out.
    If you see this happening, check out the socket module. You can import it within this function, and set temporary timeout settings.

  2. Thanks very much for this script. I’ve downloaded the google logo based on your example; now I just need to tweak your example a bit to download a series of PDF documents instead of images.

  3. You need it install a basic auth handler to urlllib 2. There are 2 methods to do that.
    1. Detect a 401 error, and send back the appropriate headers.
    2. If you know the uri you are logging into you can install it with the appropriate credentials before you call urlopen.

    Great example here from http://docs.python.org/library/urllib2.html:
    import urllib2
    # Create an OpenerDirector with support for Basic HTTP Authentication…
    auth_handler = urllib2.HTTPBasicAuthHandler()
    auth_handler.add_password(realm=’PDQ Application’,
    uri=’https://mahler:8092/site-updates.py’,
    user=’klem’,
    passwd=’kadidd!ehopper’)
    opener = urllib2.build_opener(auth_handler)
    # …and install it globally so it can be used with urlopen.
    urllib2.install_opener(opener)
    urllib2.urlopen(‘http://www.example.com/login.html’)

  4. This is also a great introduction for someone to hit the ground running using Python in an applicable manner. Have you ever messed with multi-threaded processes in Python? In the case you REALLY want to speed some things up.. :-D

  5. Thank you for this post, it is really helpful for me as a beginner with python. I wonder how could we use it to download videos say from youtube. Any Ideas.

Leave a Reply