What is BeautifulSoup?
BeautifulSoup is a Python library from www.crummy.com
What can it do
On their website they write "Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. You can tell it to: "Find all the links" "Find all the links of class externalLink" "Find all the links whose urls match "foo.com" "Find the table heading that's got bold text, then give me that text.""
In this example, we will try and find a link (a tag) in a webpage. Before we start, we have to import two modules. (BeutifulSoup and urllib2). Urlib2 is used to open the URL we want. We will use the soup.findAll method to search through the soup object to match fortext and html tags within the page.
from BeautifulSoup import BeautifulSoup import urllib2 url = urllib2.urlopen("http://www.python.org") content = url.read() soup = BeautifulSoup(content) links = soup.findAll("a")
That will print out all the elements in python.org with an "a" tag. (The "a" tag defines a hyperlink, which is used to link from one page to another.)
BeautifulSoup Example 2
To make it a bit more useful, we can specify the URL's we want to return.
from BeautifulSoup import BeautifulSoup import urllib2 import re url = urllib2.urlopen("http://www.python.org") content = url.read() soup = BeautifulSoup(content) for a in soup.findAll('a',href=True): if re.findall('python', a['href']): print "Found the URL:", a['href']
I recommend that you head over to http://www.crummy.com to read more about what you can do with this awesome module.