Browsing with Mechanize

The mechanize module in Python is similar to perl WWW:Mechanize.

It gives you a browser like object to interact with web pages.

Here is an example on how to use it in a program.

import mechanize
br = mechanize.Browser()"")

Follow second link with element text matching regular expression

response1 = br.follow_link(text_regex=r"cheeses*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()
print  # headers
print  # body

To get the response code from a website, you can the response.code

from mechanize import Browser
browser = Browser()
response ='')
print response.code

Get all forms from a website

import mechanize
br = mechanize.Browser()"")
for f in br.forms():
    print f

I found this post at that very accurate describes how to emulate a browser in Python using mechanize.

Browsing with Python (written of Drew Stephens)

import re
from mechanize import Browser
br = Browser()

Ignore robots.txt

br.set_handle_robots( False )

Google demands a user-agent that isn’t a robot

br.addheaders = [('User-agent', 'Firefox')]

Retrieve the Google home page, saving the response "" )

Select the search box and search for ‘foo’

br.select_form( 'f' )
br.form[ 'q' ] = 'foo'

Find the link to; why did we run a search?

resp = None

for link in br.links():
    siteMatch = re.compile( '' ).search( link.url )

    if siteMatch:
        resp = br.follow_link( link )

Print the site

content = resp.get_data()
print content

The script above is split up to make it easier to read

Leave a Reply

Your email address will not be published. Required fields are marked *