Regular Expressions in Python

What is a Regular Expression?

It’s a string pattern written in a compact syntax, that allows us to quickly check whether a given string matches or contains a given pattern.

The power of regular expressions is that they can specify patterns, not just fixed characters. Many examples in this article can be found on:
Googles Python Course

• Python Programming – Pattern Matching
• The Odds & Ends of Python Regular Expressions
• Introduction to Python Regular Expressions

Basic patterns

a, X, 9
ordinary characters just match themselves exactly.

. ^ $ * + ? { [ ] | ( )
meta-characters with special meanings (see below)

. (a period)
matches any single character except newline ‘n’

w
matches a “word” character: a letter or digit or underbar [a-zA-Z0-9_].
It only matches a single character not a whole word.

W
matches any non-word character.

w+
matches one or more words / characters

b
boundary between word and non-word

s
matches a single whitespace character, space, newline, return, tab, form

S
matches any non-whitespace character.

t, n, r
tab, newline, return

D
matches anything but a digit

d
matches a decimal digit [0-9]

d{1,5}
matches a digit between 1 and 5 in lengths.

{n} d{5}
matches for 5 digits in a row

^
match the start of the string

$
match the of the string end

*
matches 0 or more repetitions

?
matches 0 or 1 characters of whatever precedes it

use . to match a period or to match a slash.

If you are unsure if a character has special meaning, such as ‘@’, you can put a slash in front of it, @, to make sure it is treated just as a character.

re.findall

The findall() is probably the single most powerful function in the re module and we will use that function in this script.

In the example below we create a string that have a text with many email addresses.

We then create a variable (emails) that will contain a list of all the found email strings.

Lastly, we use a for loop that we can do something with for each email string that is found.

str = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher'

## Here re.findall() returns a list of all the found email strings
emails = re.findall(r'[w.-]+@[w.-]+', str) ## ['alice@google.com', 'bob@abc.com']
  
for email in emails:
    # do something with each found email string
    print email

We can also apply this for files. If you have a file and want to iterate over the lines of the file, just feed it into findall() and let it return a list of all the matches in a single step

read() returns the whole text of a file in a single string.

(If you want to read more about file handling in Python, we have written a ‘Cheat Sheet’ that you can find here)

# Open file
f = open('test.txt', 'r')

# Feed the file text into findall(); it returns a list of all the found strings
strings = re.findall(r'some pattern', f.read())

re.search

The re.search() method takes a regular expression pattern and a string and searches for that pattern within the string.

The syntax is re.search(pattern, string).

where:
pattern
regular expression to be matched.

string
the string which would be searched to match the pattern anywhere in the string.

It searches for first occurrence of RE pattern within string with optional flags.

If the search is successful, search() returns a match object or None otherwise.

Therefore, the search is usually immediately followed by an if-statement to test if the search succeeded.

It is common to use the ‘r’ at the start of the pattern string, that designates a python “raw” string which passes through backslashes without change which is very handy for regular expressions.

This example searches for the pattern ‘word:’ followed by a 3 letter word.

The code match = re.search(pat, str) stores the search result in a variable named “match”.

Then the if-statement tests the match, if true the search succeeded and match.group() is the matching text (e.g. ‘word:cat’).

If the match is false, the search did not succeed, and there is no matching text.

str = 'an example word:cat!!'

match = re.search(r'word:www', str)

# If-statement after search() tests if it succeeded
  if match:                      
    print 'found', match.group() ## 'found word:cat'

  else:
    print 'did not find'

As you can see in the example below, I have used the | operator, which search for either pattern I specify.

import re
programming = ["Python", "Perl", "PHP", "C++"]

pat = "^B|^P|i$|H$"

for lang in programming:
    
    if re.search(pat,lang,re.IGNORECASE):
        print lang , "FOUND"
    
    else:
        print lang, "NOT FOUND"

The output of above script will be:

Python FOUND
Perl FOUND
PHP FOUND
C++ NOT FOUND

re.sub

The re.sub() function in the re module can be used to replace substrings.

The syntax for re.sub() is re.sub(pattern,repl,string).

That will replace the matches in string with repl.

In this example, I will replace all occurrences of the re pattern (“cool”) in string (text) with repl (“good”).

import re
text = "Python for beginner is a very cool website"
pattern = re.sub("cool", "good", text)
print text2

Here is another example (taken from Googles Python class ) which searches for all the email addresses, and changes them to keep the user (1) but have yo-yo-dyne.com as the host.

str = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher'

## re.sub(pat, replacement, str) -- returns new string with all replacements,

## 1 is group(1), 2 group(2) in the replacement

print re.sub(r'([w.-]+)@([w.-]+)', r'1@yo-yo-dyne.com', str)

## purple alice@yo-yo-dyne.com, blah monkey bob@yo-yo-dyne.com blah dishwasher

re.compile

With the re.compile() function we can compile pattern into pattern objects, which have methods for various operations such as searching for pattern matches or performing string substitutions.

Let’s see two examples, using the re.compile() function.

The first example checks if the input from the user contains only letters, spaces or . (no digits)

Any other character is not allowed.

import re

name_check = re.compile(r"[^A-Za-zs.]")

name = raw_input ("Please, enter your name: ")

while name_check.search(name):
    print "Please enter your name correctly!"
    name = raw_input ("Please, enter your name: ")

The second example checks if the input from the user contains only numbers, parentheses, spaces or hyphen (no letters)

Any other character is not allowed

import re

phone_check = re.compile(r"[^0-9s-()]")

phone = raw_input ("Please, enter your phone: ")

while phone_check.search(phone):
    print "Please enter your phone correctly!"
    phone = raw_input ("Please, enter your phone: ")

The output of above script will be:

Please, enter your phone: s

Please enter your phone correctly!

It will continue to ask until you put in numbers only.

Find Email Domain in Address

Let’s end this article about regular expressions in Python with a neat script I found on stackoverflow.

@
scan till you see this character

[w.]
a set of characters to potentially match, so w is all alphanumeric characters, and the trailing period . adds to that set of characters.

+
one or more of the previous set.

Because this regex is matching the period character and every alphanumeric after an @, it’ll match email domains even in the middle of sentences.

import re

s = 'My name is Conrad, and blahblah@gmail.com is my email.'

domain = re.search("@[w.]+", s)

print domain.group()