Requests In Python

What is Requests

The Requests module is a an elegant and simple HTTP library for Python.

What can I do with Requests?

Requests allow you to send HTTP/1.1 requests.

You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way.

Note, the notes in this post are taken from Python-Requests.org :

Requests Installation

To install requests, simply:
$ pip install requests

Or, if you absolutely must:
$ easy_install requests

Make a Request

Begin by importing the Requests module:
>>> import requests

Now, let’s try to get a webpage.

For this example, let’s get GitHub’s public timeline

>>> r = requests.get('https://github.com/timeline.json')

# Now, we have a Response object called r. We can get all the information we need from this object.

To make a HTTP POST request

>>> r = requests.post("http://httpbin.org/post")

You can also use other HTTP request types, like PUT, DELETE, HEAD and OPTIONS

>>> r = requests.put("http://httpbin.org/put")

>>> r = requests.delete("http://httpbin.org/delete")

>>> r = requests.head("http://httpbin.org/get")

>>> r = requests.options("http://httpbin.org/get")

Response Content

We can read the content of the server’s response.

Consider the GitHub timeline again:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.text
'[{"repository":{"open_issues":0,"url":"https://github.com/...

Requests will automatically decode content from the server.

Most unicode charsets are seamlessly decoded.

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers.

The text encoding guessed by Requests is used when you access r.text.

You can find out what encoding Requests is using, and change it, using the r.encoding property:

>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

If you change the encoding, Requests will use the new value of r.encoding whenever you call r.text.

Binary Response Content

You can also access the response body as bytes, for non-text requests:

>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...

JSON Response Content

There’s also a builtin JSON decoder, in case you’re dealing with JSON data:

>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...

In case the JSON decoding fails, r.json simply returns None.

Custom Headers

If you’d like to add HTTP headers to a request, simply pass in a dict to the headers parameter.

For example, we didn’t specify our content-type in the previous example:

>>> import json
>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> headers = {'content-type': 'application/json'}

>>> r = requests.post(url, data=json.dumps(payload), headers=headers)

Response Status Codes

We can check the response status code:

>>> r = requests.get('http://httpbin.org/get')
>>> r.status_code
200

# Requests also comes with a built-in status code lookup object for easy reference:
>>> r.status_code == requests.codes.ok
True

# If we made a bad request (non-200 response), 
# we can raise it with Response.raise_for_status():
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404

Response Headers

We can view the server’s response headers using a Python dictionary:

>>> r.headers
{
    'status': '200 OK',
    'content-encoding': 'gzip',
    'transfer-encoding': 'chunked',
    'connection': 'close',
    'server': 'nginx/1.0.4',
    'x-runtime': '148ms',
    'etag': '"e1ca502697e5c9317743dc078f67693f"',
    'content-type': 'application/json; charset=utf-8'
}

HTTP Headers are case-insensitive, so we can access the headers using any capitalization we want:

>>> r.headers['Content-Type']
'application/json; charset=utf-8'

>>> r.headers.get('content-type')
'application/json; charset=utf-8'

# If a header doesn’t exist in the Response, its value defaults to None:
>>> r.headers['X-Random']
None

Cookies

If a response contains some Cookies, you can get quick access to them:

>>> url = 'http://httpbin.org/cookies/set/requests-is/awesome'
>>> r = requests.get(url)

>>> r.cookies['requests-is']
'awesome'

# To send your own cookies to the server, you can use the cookies parameter:

>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')

>>> r = requests.get(url, cookies=cookies)

>>> r.text
'{"cookies": {"cookies_are": "working"}}'

Basic Authentication

Many web services require authentication.

There are many different types of authentication, but the most common is HTTP Basic Auth.

Making requests with Basic Auth is extremely simple:

from requests.auth import HTTPBasicAuth
requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))


# Due to the prevalence of HTTP Basic Auth, 
# requests provides a shorthand for this authentication method:

requests.get('https://api.github.com/user', auth=('user', 'pass'))

Providing the credentials as a tuple in this fashion is functionally equivalent to the HTTPBasicAuth example above.

Digest Authentication

# Another popular form of web service protection is Digest Authentication:

>>> from requests.auth import HTTPDigestAuth

>>> url = 'http://httpbin.org/digest-auth/auth/user/pass'

>>> requests.get(url, auth=HTTPDigestAuth('user', 'pass'))

Redirection and History

Requests will automatically perform location redirection while using the GET and OPTIONS verbs.

GitHub redirects all HTTP requests to HTTPS.

We can use the history method of the Response object to track redirection.

Let’s see what Github does:

>>> import requests
>>> r = requests.get("http://github.com")
>>> r.url
u'https://github.com/'
>>> r.status_code
200
>>> r.history
[]
>>>

The Response.history list contains a list of the Request objects that were created in order to complete the request.

The list is sorted from the oldest to the most recent request.

If you’re using GET or OPTIONS, you can disable redirection handling with the allow_redirects parameter:

>>> r = requests.get('http://github.com', allow_redirects=False)
>>> r.status_code
301
>>> r.history
[]

Timeouts

You can tell requests to stop waiting for a response after a given number of seconds with the timeout parameter:

>>> requests.get('http://github.com', timeout=0.001)
Traceback (most recent call last):
  File "", line 1, in 
requests.exceptions.Timeout: Request timed out.

Errors and Exceptions

In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception.

In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.

If a request times out, a Timeout exception is raised.

If a request exceeds the configured number of maximum redirections, a TooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException.

You can refer to Configuration API Docs for immediate raising of HTTPError exceptions via the danger_mode option or have Requests catch the majority of requests.exceptions.

RequestException exceptions with the safe_mode option.