Category Archives: Python

The place for anything Py

Python 3 File Read Write with Urllib

This post is inspired by my previous post on utilizing urllib2 to download a sequence of files programatically. As you probably know, the transition from Python2 to Python3 has left many people struggling to port their code, so I thought I would re-hash some of my old posts and provide Python3 versions of my code examples. One resource I found recently that really helped me is the online version of Mark Pilgrim’s “Dive into Python3”, specifically the chapter on porting your 2.x code to Python3.

The example provided below outlines how to use the urllib library included within Python3 to download a sequence of image files along with comments to describe what is going on.

#import urllib request 
import urllib.request
#import urllib error handling
from urllib.error import HTTPError,URLError

#function that downloads a file
def downloadFile(file_name,file_mode,base_url):
    #create the url
    url = base_url + file_name
    
    # Open the url
    try:
            f = urllib.request.urlopen(url)
            print("downloading ", url)
    
            # Open our local file for writing
            local_file = open(file_name, "w" + file_mode)
            #Write to our local file
            local_file.write(f.read())
            local_file.close()
    
    #handle errors
    except HTTPError as e:
            print("HTTP Error:",e.code , url)
    except URLError as e:
            print("URL Error:",e.reason , url)

# Set the range of images to 1-50.It says 51 because the
# range function never gets to the endpoint.
image_range = list(range(1,51))

# Iterate over image range
for index in image_range:
    base_url = 'http ://www.techniqal.com/'
    #create file name based on known pattern
    file_name =  str(index) + ".jpg"
    # Now download the image. If these were text files,
    # or other ascii types, just pass an empty string
    # for the second param ala stealStuff(file_name,'',base_url)
    downloadFile(file_name,"b",base_url)

The key things to learn about converting my old example to the new are outlined below. This was a learning exercise for me, and will hopefully provide enough context for you to understand how to port your own code to Python3.

  • There are obvious changes on how to use Urllib vs the old Urllib2 methods. Take a peek at “Dive into Python3” for more details. He does a much better job describing it than I ever could.
  • Print statements are now called as a function.

    Python2:

    print "My Variable is equal to " + myVariable
    

    Python3:

    print("My Variable is equal to ", myVariable)
    
  • Except blocks are handled differently when using a try/except.

    Python2:

    except HTTPError, e:
    		print "HTTP Error:",e.code , url
    

    Python3:

    except HTTPError as e:
    		print("HTTP Error:",e.code , url)
    
  • The range() function used to return a list , but now returns an iterator object. If you still want to get a list from the range function, see below.

    Python2:

    myRangeList = range(1,100)
    

    Python3:

    myRangeList = list(range(1,100))
    

I’m not a software engineer by trade, so please excuse any syntax oddities. I appreciate any feedback, or more graceful ways to write this code. Leave them in the comments and I’ll happily update my example.

Python File Read Write with Urllib2

Update: Looking for how to download files using Python3 and urllib? Check out my post here .

While checking out my great stats on Lijit recently, I started to see a pattern. I was able to determine that a large part of my Re-Search(search engine) traffic was coming from a post I did about reading and writing to files in Python back in  2005. In an effort to shamelessly attract more traffic on this topic, I have decided to flesh this post out a bit.

A common task that I run into both in my work life as well as my personal life, revolves around programmatically downloading content from the interwebs. This little code example will illustrate how to use urllib to download a file, and write/save the file contents locally. You may be saying to yourself “Self, can’t I do this in my favorite web browser??” . The answer is “YES”, but it’s a pain in the ass if you have more than 5 files you want to download.

Assume there are a set of images on your favorite website, and they are all named  image1.jpg,image2.jpg,image3.jpg, etc. Now imagine there are 50 images using this naming convention.How do you download them all using python , without struggling to do it one image at a time in your browser? Look below!

python

# Let's create a function that downloads a file, and saves it locally.
# This function accepts a file name, a read/write mode(binary or text),
# and the base url.

def stealStuff(file_name,file_mode,base_url):
	from urllib2 import Request, urlopen, URLError, HTTPError
	
	#create the url and the request
	url = base_url + file_name
	req = Request(url)
	
	# Open the url
	try:
		f = urlopen(req)
		print "downloading " + url
		
		# Open our local file for writing
		local_file = open(file_name, "w" + file_mode)
		#Write to our local file
		local_file.write(f.read())
		local_file.close()
		
	#handle errors
	except HTTPError, e:
		print "HTTP Error:",e.code , url
	except URLError, e:
		print "URL Error:",e.reason , url


# Set the range of images to 1-50.It says 51 because the 
# range function never gets to the endpoint.
image_range = range(1,51)

# Iterate over image range
for index in image_range:
	
	base_url = 'http://www.techniqal.com/'
	#create file name based on known pattern 
	file_name =  str(index) + ".jpg"
	# Now download the image. If these were text files, 
	# or other ascii types, just pass an empty string 
	# for the second param ala stealStuff(file_name,'',base_url)
	stealStuff(file_name,"b",base_url)

That’s it. It not only reports on any errors it encountered while downloading, but think of all of the time you just saved… Really though, how important is your time to you if you’re reading this blog???

Ruby and Rails Float Python’s Boat

python_winner.gif
Antonio over at antoniocangiano.com has a great article about the effect the rise of Rails and Ruby has had on Python.

I totally agree with him regarding the meteoric rise in popularity of Rails buoying scripting languages such as python. I believe more and more businesses are choosing, or are open enough, to implement Rails and Django solutions for their webapps. On top of other MVC frameworks, both Django and Rails allow for fast and sometimes cheap development (the non-cheap comes in if you have to hire an over-priced rails consultant). The visibility is definitely changing the way businesses view scripting languages and allow them to be seen as a viable alternative to Java and other enterprise adopted technologies.

I have had a man-crush on python for quit some time, and try to spout off about it’s benefits whenever I can.
I have been so close to starting a series of blog posts surrounding Python and the ease of adoption for the non-programmer. As soon as I have more time…

My Squidoo Public Beta

squid
So, the word is out, and Seth Godins‘ latest project, Squidoo, is slightly open to the public. Those of us who were closed beta testers were given the go ahead to share our Squidoo Lenses with friends and family.

For those who don’t know, Squidoo, is a site that enhances the search engine ideology. It does this by allowing people to publish information/links and articles(Lenses) on subjects that a person(LensMaster) may have experience/expertise in.
The concept allows these self proclaimed experts to provide tightly filtered and relevant information on a topic, without all of the cruft that comes along with a normal search engine. The whole project abounds with the hottest buzzwords of the day,ie. longtail,web 2.0, ajax,etc.

So I decided to do a lense about the Python programming language, and it’s use on Windows. So without any further ado, the link to my Squidoo lense.
http://www.squidoo.com/python_on_windows/

I am currently ranked 1,777 , –so if you believe my lense has some valuable content, go ahead and rate it.

How to Run a Script from Email

Got a little idea to write this up after seeing a question over on Photo Matt. He asked about having a script that could take commands sent by email and execute them remotely, then return the results via email.
So I decided to hack something up in Python, and give it a try. Read the rest of the post to see how I did it.
Or you can cheat and just look at the script itself. Click Here for the script.

Continue reading How to Run a Script from Email