Practice
Regular Expressions, Part I
Sergey Kolesnichenko
2004-03-23
Chapter 1. Sherlock Holmes Comes to the Web Programmer's Aid, or Regular
Expressions Made Simple.
Introduction.
Every web programmer has faced the task of finding some piece of
data in an arbitrary text according to some rule, validating data
submitted by a user, or subjecting found data to complex
modification. You can reinvent the wheel, or you can use the tools
that programmers all over the world use. Sometimes it seems that
professionals use some tools and techniques available only to them.
I will disappoint the reader: professionals use the same tools and
instruments as you do; the only difference is that they know how to
use them and know how to choose which tool is worth using in a
specific case.
This material is intended to help programmers solve everyday tasks
using regular expressions. I will try to describe the very basics of
using this tool, so that you no longer stare at a combination like
this: /^(?:http:\/\/)?[-0-9a-z._]*.\w{2,4}[:0-9]*$/ like a sheep at
a new gate.
The general task of the regular expression engine is to find or not
find matches between a string or part of it and a pattern. Let's
analyze the first sentence of this paragraph for any confusing or
scary words:
"Regular expression engine" and "pattern" - these are the two words
that plunged me into despair when I realized that I couldn't do
without regular expressions. We have some kind of engine that
searches for something and either finds it or doesn't; this
"something" is connected to concepts such as string and pattern.
These are exactly what we'll sort out, and end with, because once we
understand what this engine does with strings and patterns, we won't
need to dig through math textbooks looking for what the words
"regular expressions" mean.
Part 1.
Where have we seen a pattern before? Let's go ask a secretary we
know. Right, the answer is Microsoft Word templates! What's the
difference between the "Calendar" template and the "Elegant Resume"
template? The data and the way it's presented. A person who has seen
both at least once will have no trouble telling a calendar apart
from a resume. So why then do regular expressions scare a
programmer? After all, it's almost the same thing! What does a
person think about who sees a calendar and knows what it is, how do
they recognize it? A calendar is a document divided into blocks,
each block consists of numbers that correspond to the days of the
month. Each month corresponds to only one block, a month has no more
than 31 days, February has no more than 28 (except in a leap year),
days that correspond to Sunday or a public holiday are highlighted
in red, you can continue systematizing the data further, specifying
which months have exactly 30 and 31 days. What did we do? We created
a description of the calendar; in other words, we described the
data, and finding this data in an arbitrary text lets us say with a
certain degree of confidence that we're looking at a calendar. I
call this kind of description a pattern, in the context of talking
about regular expressions.
The day will come when a programmer will simply tell the computer
what program he wants written, and the computer will write it, but
for now the programmer still has to work himself. That is, if I say
that I'm looking for a piece of text that satisfies the description
of a calendar, the computer of today won't understand me. Get
comfortable, because if the mountain won't come to Muhammad, then
Muhammad must go to the mountain. Like all programmers, we'll go to
the mountain sitting in front of a monitor. Our task is to learn to
describe the data we want to find in a form the computer
understands. And what else don't you know how to do? Shame on you!
Even every secretary knows this. I hope I don't need to explain the
command line to a programmer. Start=>Run=>cmd.
Let's type the word dir on the command line.
Here's what I got:
C:\Documents and Settings\Administrator>dir Volume in drive C has no label. Volume Serial Number is 3CC6-6445 Directory of C:\Documents and Settings\Administrator 13.10.2003 18:03 <DIR> . 13.10.2003 18:03 <DIR> .. 18.07.2003 21:55 <DIR> .java 18.07.2003 21:54 <DIR> .javaws 18.07.2003 21:55 <DIR> .jpi_cache 15.10.2003 16:33 694 .plugin141.trace 05.10.2003 11:40 <DIR> Desktop 16.10.2003 13:08 <DIR> Favorites 08.10.2003 16:42 <DIR> My Documents 18.08.2003 20:51 <DIR> Start Menu 04.07.2003 21:24 <DIR> WINDOWS 1 File(s) 694 bytes 10 Dir(s) 2 162 040 832 bytes free
Looks familiar? Of course! Consider that you already know how to
use regular expressions, all that's left is to refine your skills.
What would you do if there are many files and directories, but you
only need to check the ones you're actually interested in? You
would try to reduce the amount of output data by specifying a
search condition, describing the data you want to get. Notice that
you need to describe the data, which means you're standing at the
threshold of creating a pattern. Suppose we're interested in all
files and directories whose name contains the word java. I'm sure
you're thinking the same way I am and you get a result like this:
C:\Documents and Settings\Administrator>dir *java* Volume in drive C has no label. Volume Serial Number is 3CC6-6445 Directory of C:\Documents and Settings\Administrator 18.07.2003 21:55 <DIR> .java 18.07.2003 21:54 <DIR> .javaws 0 File(s) 0 bytes 2 Dir(s) 2 161 618 944 bytes free
Let's try to translate the line dir *java* into plain English: Find
and show all files and directories in the current directory whose
name contains the word java.
Seems right, but no! Understanding regular expressions consists
primarily of correctly describing the rule of a match (or
mismatch), as well as knowing the tools with which this description
can be "told" to the computer. Most articles on the web deal with
solving the second point, and completely skip the first; they tell
the programmer what tools can be used to "tell" the computer his
description of the data he's interested in. Why did I say the
description is wrong? Because dir java* differs by one character
from dir *java*, but under the plain-English description it also
qualifies, since the word "java" is present in the name, right?
Let's make a second attempt at describing the string dir *java*
Find and show all files and directories in the current directory
(and up to this point everything is correct), whose name starts
with any character, there can be any number of such characters
(including none at all), but after them the characters "j", "a",
"v", "a" must follow in a row, after which any character may
follow, there can be any number of these characters (or there may
be none at all).
Different? You bet! Let's continue learning to describe the data we
want to find. We search using the dir command, working from the
command line. But let's make the task harder. Now I'll give a
description of the data I want to find, and you try to do it
yourself; all the tools are familiar to you.
Find and show all files and directories in the current directory,
whose name starts with any character, there can be any number of
such characters (or there may be none at all), but after them there
must be the character "j", after which any number of any characters
follows again, but at the end of the name there must be the
characters "w" and "s" in a row.
If you read carefully, you'll have no trouble entering: dir *j*ws
and getting the response:
C:\Documents and Settings\Administrator>dir *j*ws Volume in drive C has no label. Volume Serial Number is 3CC6-6445 Directory of C:\Documents and Settings\Administrator 18.07.2003 21:54 <DIR> .javaws 0 File(s) 0 bytes 1 Dir(s) 2 161 504 256 bytes free
Simple? Sure is! Now imagine that I want to find not any number,
but a specific number of arbitrary characters. For this the
character "?" (question mark) is used. And here's the task:
Find and show all files and directories in the current directory,
whose name starts with any character, there can be any number of
such characters, but after them there must be the character "j",
after which exactly four arbitrary characters follow, and after
them there must be the characters "w" and "s" in a row. Its
solution: dir *j??ws
C:\Documents and Settings\Administrator>dir *j?ws Volume in drive C has no label. Volume Serial Number is 3CC6-6445 Directory of C:\Documents and Settings\Administrator 18.07.2003 21:54 <DIR> .javaws 0 File(s) 0 bytes 1 Dir(s) 2 161 504 256 bytes free
The first part is finished. Brief summary:
* search data comes as input to, in our case, the dir command in
some unified form, which I called the data description or
pattern;
* the output mechanism of the dir command can be controlled by
changing the pattern;
* the dir command outputs data only if it satisfies the given
pattern;
* a pattern consists of both ordinary letters and special
characters.
Part 2.
A pattern consists of both ordinary letters and special characters;
in the previous part this was the last line, and it's also the
first one in this part. You didn't even notice that you're already
thinking abstractly while describing the data you want to find. For
now you still need a nudge to do this, but soon you'll learn to do
it on your own, and you won't need me anymore. It turns out that
ordinary characters and special characters have names. Each
individual character is called a literal, each special character is
called a metacharacter. Let's leave literals alone for now, they're
not interesting to us yet, and instead let's sort out
metacharacters. So far we know two of them, but that's enough for
us.
Look at what the character "*" and the character "?" mean. The
first means any number of any characters, the second means only one
arbitrary character. Well, if the quantity part is clear so far,
then what does "any character" mean? You ask - we answer! "Any
character" in our context of working with the dir command from the
command line means any of the literals. Which means the asterisk
"*" means "any number of literals", and "?" means one literal.
Clear? Sort of. Clear, but not quite - the definition isn't quite
precise. Why? Because when I was describing the search data for the
dir command, I wrote something like: starts with any character,
there can be any number of such characters (including none at all)
The asterisk "*" means any number of literals, including their
absence! What does this mean? It means that there can be any number
of literals, but for a match not even one is required! And this
needs to be understood and remembered.
The question mark "?" means that there can be only one literal!
That is, at this position there can be only one literal, no more
and no less!
Let's take a closer look at the concept of a literal, from a
slightly different angle. Which of these two characters is a
literal, and which is a metacharacter? "j" "*"
Elementary, my dear Watson, "j" is a literal, "*" is a
metacharacter.
A quite logical question arises: how did I tell them apart?
To a logical question we give a logical answer. We know that the
characters "*" and "?" have some kind of magical effect, at least
the dir command treats them differently from the characters "j",
"a", "v", "a". So we've divided all characters into two classes:
one - literals, the other - metacharacters. We know for certain
that the class of literals includes all Latin letters a, b, c, d
and so on up to z, as well as the digits 0, 1, 2 and so on up to 9.
That is, we've split the class of literals further into subclasses.
Don't you think we already know too much to still be fiddling with
the command line? It's time to try out our knowledge.
Brief summary of part two:
* ordinary characters are called "literals";
* special characters are called "metacharacters";
* literals mean themselves;
* metacharacters are meant to describe a range of literals, some
conditions applicable to literals, properties of literals,
their quantity;
* all literals can be classified by grouping them together
according to some feature.
Part 3.
Well, this author has finally stopped telling tales about the
command line, and started saying something about what the article
was actually written for - the use of regular expressions in web
programming. Stop and think, is everything clear in the previous
two parts? If not, read them again, work through all the examples,
experiment! I don't recommend reading part 3 without understanding
the previous parts, you'll just waste your time.
From the great and mighty PHP language we'll need only one
function: preg_match()
Its general format is as follows:
preg_match("search_pattern", "string_to_search_in",
array_with_search_results)
This function implements a call to the regular expression
processing engine, a search for a match in a string, and the return
of matches into an array. Now. Stop! What does "call to the
engine", "search" and "return" mean? We're used to operating with
somewhat different concepts. Let's go back to the dir command
again. It contains some mechanism that reads your pattern and
searches for files and directories in the current directory based
on it, right? For simplicity let's call this mechanism the regular
expression processing engine too. Imagine that the names of files
and directories are written into a string, the search pattern is
written into another string, and the results are output not to a
black screen, but into an array, and to search for information
about files and directories we use not the dir command, but the PHP
function preg_match(). Everything falls into place. The function
preg_match() passes the search pattern and the string to search in
to the match search engine (the regular expression processing
engine), and ensures the search results are output into an array.
Just like the dir command! Now it's clear why I said at the
beginning that you don't need to be taught how to use regular
expressions - you already know how it's done, you just don't
realize the fact. Well, there you go, congratulations, now you're
aware of the fact too. We could stop here, if PHP's search
capabilities were no different from those of the dir command. But
that's not the case, so we'll keep learning and read Chapter 2.
Chapter 2. Basics.
One more introduction
Regular expressions are a pattern language (actually this is a
mathematical term; if you're interested, read about deterministic
and non-deterministic finite automata)
In order to perform some action, you need to specify which one;
usually the action is specified by a function:
In PHP this would look, for example, like this:
* preg_replace - replace
* preg_match - find a match
* preg_split - split
the same goes for functions of the Posix standard like ereg, except
that they differ in that, unlike Preg, they have a different match
search engine.
Pattern processing happens character by character; how would you
search for the letter d in the word stadium? Right, the first thing
that comes to mind is to go through all the letters of the word
stadium and compare them with the letter you're looking for, so we
get a simple brute-force search. Consider that you've already
learned to use regexes, all that's left is to learn how to specify
the pattern of what you want to find in a string.
A pattern is a kind of pointer to what to search for in a string.
You can search for digits, letters, invisible characters (space,
tab)
How would you search in the word stadium for the letter d or the
letter m until one of the characters matches for the first time?
Right, also by brute force, you'd take each letter of the word and
compare it in turn with what you need to find, with d and with m.
But now each time you'll have to compare a letter of the word
(string) with two letters of the search condition. In this way,
you've created your first character class, which in the language of
regular expressions is written like this: [dm] - this means that
you're looking for either d or m
What's needed to specify that you're looking in the string for
something that can be any letter of the alphabet? You either need
to list them all [abc....xyz] or simply write a range [a-z]
Attention, a deadly stunt, this is possible with Russian letters
[а-я] as well as with digits [0-9], but there are also uppercase
[A-Z], i.e. to get a character class with all the letters of the
Latin alphabet you need to put [a-zA-Z] in the pattern
But such a character class only describes one character, while you
have quite a few of them in your string - this is solved using
quantifiers.
Quantifiers
So, let's continue, as I already said, that a search condition can
be described using character classes.
A single character class can match only one character! - this must
be understood!
How do you specify a search for two characters in the condition?
Let me give a simple example without character classes, which was
already touched on above:
We're looking for the sequence iu in the word stadium. How would
you do this? Naturally by brute force:
1. take the first character of the word and compare it with the
first character of the condition: 's' is not equal to 'u'
2. take the second character of the word and compare it with the
first character of the condition: 't' is not equal to 'i'
3. keep doing this until the j-th character in the word matches
the first character of the search condition. At this moment
'i' equals 'i' (we are at the highlighted position in the word
stadium, and the highlighted position of the search condition
iu), as soon as this condition is met, you need to take the
next character from the word and from the condition (having
first remembered where the first match occurred, a very
important step!)
4. we compare the next character of the word with the second
character of the condition: 'u' equals 'u'. Congratulations,
the original combination has been found!
5. What to do if condition 4 is not met, if instead of the word
stadium we were given the misspelled word stadiem?
Go back to step 3 and remember the important step you took. You
remembered where you found the first match. There was no match
between the next character of the word and the next character of
the condition, so you need to take the next character of the word;
in our misspelled word this will be stadiem, and compare it again
with the first character of the condition! and continue performing
step 3 until the end of the word; it's clear that no match will be
found.
We've already learned to search by two specific characters, now
let's learn to use character classes.
Suppose we have the strings:
abcd12345efg fghi56789qwe
Condition: Find in the strings segments that consist of four
arbitrary letters of the Latin alphabet, followed by five arbitrary
digits.
Above I explained how to describe a character that will match any
Latin letter; let me remind you, for this we use the character
class: [a-z] (for now we ignore the fact that there are also
uppercase letters); a condition character that will match any digit
is described by this character class: [0-9]
Let's go back to the original task. We need to find strings with
four letters at the beginning, immediately followed by five digits.
From what we already know, we can write:
[a-z][a-z][a-z][a-z][0-9][0-9][0-9][0-9][0-9]
See for yourself, this form of writing the search condition will
work! since we described each character in the search condition
with one character class. Don't you think the search condition is
getting too cumbersome? And that's where quantifiers come to the
rescue. Let's recall English: quantity. That is, a quantifier is
something that expresses a quantity of something, in our case the
number of characters in the search condition. Let's simplify the
search condition using quantifiers: [a-z]{4}[0-9]{5}
That's it! for those who haven't guessed, the curly braces specify
how many characters, described by the character class, can go in a
row in the string in which we're searching for a match.
So it turns out that after five arbitrary letters of the Latin
alphabet come 5 arbitrary digits. The search happens exactly as I
described above, only the characters in the search condition are
specified not explicitly, but using character classes.
Each character class describes only one character, the number of
similar characters following in a row is described by quantifiers.
Naturally, this kind of specifying the number of characters in the
search condition is not the only one. Quantifiers come in different
forms! [a-z]{1,3} means that one to three Latin letters can go in a
row. [a-z]{2,} means that at least 2 Latin letters can go in a row.
But the quantifier in curly braces is not the only way to specify
the number of characters in a row described by a character class.
[a-z]* means that any number of Latin letters can go in a row, or
there may be none at all, identical to [a-z]{0,}. [a-z]+ means that
at least one Latin letter must necessarily go in a row, but the
maximum number is not specified, identical to [a-z]{1,} [a-z]?
means that the number of Latin letters must not exceed 1, the
letter may also be entirely absent, identical to [a-z]{0,1}.
Applying quantifiers to literals
Let's go back to the very beginning of the article, where we
searched by brute force for the letter d in the word stadium. A
search condition that is described by a single non-special
character is called a literal.
There are strings:
abcdefg abcddefg abcdddefg abcddddefg
Requirement: Write a search condition matching all the strings.
Solution: abcd{1,4}efg
You just saw how I applied a quantifier to a literal. Any of the
above quantifiers can be applied to literals.
It's clear that applying the search condition avcd{1,4}efg to the
string abcefg, no match will be found, since the quantifier {1,4}
implies that after abc and before efg there must be at least one,
at most four letters d.
Character classes "for advanced users"
You've already seen how character classes make it easier to
describe a match condition. Let's finally sort them out completely.
What can be included in a character class? A character class can
include any literal, as well as ranges of literals. To describe
ranges of literals, the character '-' is used, placed between the
first character of the range and the last. Here are examples of
specifying various ranges in one character class: [1-5] - numbers
in the range from 1 to 5 [a-f] - letters of the Latin alphabet from
a to f [a-fq-x] - letters of the Latin alphabet from a to f and from
q to x, as you noticed, in the last character class I used two
ranges.
Imagine that you need to describe a condition where a certain
position in the string can contain the characters: either a, or g,
or 7, or 4. What to do? Write the character class: [ag47]
The explanation is simple: in a character class you can list the
literals allowed in the search condition. Listing literals can be
combined with specifying ranges: [14a-kz] - this means that the
character in the string can match 1, 4, letters of the Latin
alphabet from a to k, as well as the letter z. Naturally, literals
can be not only letters and digits, but also punctuation marks,
mathematical signs, for example ',' (comma), '!' (exclamation
mark), '+' (plus), '-' (minus). Wait, you'll say, what minus, if
it's used to describe ranges? Correct, if you put a minus between a
and z, it will be a range, but if you put a minus right after the
opening square bracket, it will be a minus! And here's an example:
[-,a-z] - means that the character class includes minus, comma, as
well as the letters of the Latin alphabet from a to z.
And how do you write a character class that includes all characters
except the given ones? For example, all except a, b, c? For this
there is a special negation metacharacter: ^ (caret) We write:
[^abc] - all characters (not letters, but specifically characters)
except the Latin letters a, b, c.
Remember everything!
Probably many people wonder: what if you need to check the whole
string against a condition, but return, using the language's
functions, only part of the string?
How do you use the language's functions to perform an operation not
on the whole string that satisfies the condition, but only on part
of it?
Specifically for such purposes there are grouping and simultaneously
capturing parentheses.
Example:
A string may consist of 5 Latin letters, followed by a minus sign,
followed by four digits from 1 to 8.
Task:
1. check strings for compliance with the condition
2. return to the program the four digits, which at this stage
means only that these four digits need to be somehow
highlighted in the search condition
If we just want to check a string for a match with the condition,
the search condition will look like this: [a-z]{5}[1-8]{4} but we
need to remember the digits, so we add a memorization instruction:
[a-z]{5}([1-8]{4})
As you can see, what needs to be remembered is set off by
parentheses. Inside the function that will perform the operation on
the string using the above condition, the match will be stored in
special variables; in PHP it can be accessed via \1, in Perl - $1.
A single search condition can contain several memorization
instructions: ([a-z]{5})([1-8]{4}) - will check the string for a
match with the condition, and in case of a successful match, will
remember five letters in \1 ($1), four digits in \2 ($2). If you
reference the variable \0, it turns out that it holds the entire
matched string that was described by the condition.
Closer to reality:
Whoever has had the patience to read this far already understands a
lot, tries to write their own search conditions for a string, but
it doesn't work out. Most likely this happens because you don't
know many of the finer details of working with regular expressions.
I'll tell you about some of them right now. A string has a
beginning and an end. You'll say that this is obvious to everyone,
but most likely you'll have to slightly change your notion of
strings. Consider that there is an invisible character at the
beginning of each string and at the end of each string. When
searching in a real program, this must be taken into account by
placing the character '^' (start of string) at the beginning of the
search condition, and '$' (end of string) at the end of the search
condition. So the example with five letters and four digits (like
all the other examples above) needs to be rewritten:
^[a-z]{5}[1-8]{4}$ now using this condition you can actually check
the strings
asbvc1234 sdwtsv1234
for compliance with the condition.
Don't confuse the '^' (caret) in a character class with the same
character used when describing a match condition. A caret placed at
the beginning of a condition corresponds to the start of a string;
a caret placed right after the square bracket that opens a
character class description will be a negation metacharacter
(everything except what's present in the character class); a caret
placed at any other position inside a character class, other than
the first, will simply be a literal.
Your search condition may not work because you're not aware of the
quantity, as well as the kind, of whitespace characters present in
the string. Spaces are described either by a literal - simply put a
space in the search condition - or by a visible combination of
characters: \s
Consider that \s matches the literal 'space', or the literal
'newline', or the literal 'tab'.
In programs, in order to show where the search condition (regular
expression) begins, delimiters are used.
Example:
preg_match("/^[a-z0-9]/", $string,$mathces);
look at what is specified in the quotes of the first argument of
the function: first comes a slash, then the start-of-string
character, then comes the character class, then comes the slash
again. It is exactly the first and last slash that signal that a
regular expression is enclosed between them. This came from Perl,
where right after the delimiter you can also specify modifiers,
which we'll get to later. At this stage you need to know that the
search condition record needs delimiters (two identical characters,
not necessarily a slash, many people use a tilde)
Examples for the chapter
All the examples are taken from questions people ask on various
forums. I will go through examples in the language I write in
myself: PHP.
1. A very common situation is when you need to write registration for a resource you maintain. For various reasons, requirements are placed on the entered information. For example, people often ask how to check that a user entered, as a nickname (login), a word consisting only of Latin letters and digits. I'll give the condition check right inside the code, so it's clear what to write and where:
<?php
$user = $_POST['username'];
if(!preg_match("/^[a-zA-Z0-9]+$/", $user)) {
echo "Username is in the wrong format";
} else {
echo "Username is in the correct format";
}
?>
Now let's analyze the regular expression itself.
Since the registration username must consist of Latin letters
as well as digits, we need to write a character class that
satisfies this condition: [a-zA-Z0-9] this character class
includes three ranges, the first range a-z (all characters from
the lowercase letter a to the lowercase letter z), the second
range A-Z (similarly, but with uppercase letters), the third
range 0-9 (digits from 0 to 9). We've described only one letter
that the registration name can consist of, but there can be...
and now we need to answer the question, how many such letters
can there be? As many as you like, you'll say, and I'll say
you're wrong.
The registration name must consist of at least one letter! and
I consider this a mandatory condition for registration, so this
fact needs to be described. Let's recall quantifiers:
[a-zA-Z0-9]+
The plus '+' is exactly the quantifier that says that the
string variable $user must have at least one character that
satisfies the condition.
Now we need to tell the regular expression that the entire
string, from beginning to end, must satisfy the condition, so
we add to the regular expression the start-of-string character
'^' at the beginning of the regular expression and the
end-of-string character '$' at the end:
^[a-zA-Z0-9]+$
Now we need to explain to the preg_match function that the
string ^[a-zA-Z0-9]+$ is a regular expression, we need to put
delimiters, I'll use the slash '/':
preg_match("/^[a-zA-Z0-9]+$/",$user)
2. There are people who make their own lives harder, these people are called programmers and system administrators, and often they make life harder not only for themselves, but also for each other, and then on top of that the boss comes along and sees that on the company site you can register, and the username is already checked in some fancy way; in 80% of cases the above task is made harder by the boss, either the character set needs to be expanded, which is done by simply adding characters to the character class, or the conditions are tightened. For example, a condition is set that the username can consist of both Latin letters and digits, but the first character of the username must necessarily be a Latin letter!
We add to the regular expression: ^[a-zA-Z][a-zA-Z0-9]*$
As has long been known, a character class describes only one
character. Our task here is precisely to describe one character
(the first): [a-zA-Z] here there are two ranges, describing the
first character and making it clear that there can be no digits
in the first character of the string being checked.
From the reasoning in the previous example it clearly followed
that the registration username must be at least one character
long. We've already described the first character of the
string, but it may also be the only one; after it there can be
any number of characters, or there may be none at all, but the
regular expression will only match when all subsequent
characters satisfy the condition of being either letters of the
Latin alphabet or digits. [a-zA-Z0-9] The first mandatory
character is described, the remaining characters are not
mandatory, so we change the quantifier from '+' to '*'. We put
the start and end of string characters: ^[a-zA-Z][a-zA-Z0-9]*$
Chapter 3.
After studying the basics of how it works, it's worth moving on to
the practical application of regular expressions. But if you look
at most regular expressions with the knowledge you have so far, it
turns out they still look like a set of squiggles, though some of
them are already recognizable. In this part, building on what I
explained in the previous chapter, I will broaden your horizons.
Special characters.
How do you show something invisible? How do you indicate, for
example, that a space is present in the search condition? Probably
many people guess that in order to show the invisible, you need to
make it visible, i.e., introduce some character, a set of
characters, that will be interpreted as invisible. With this
knowledge, we've arrived at studying special characters in regular
expressions.
* \s - if you put a backslash followed by the letter s in the search condition, this way you'll describe either a space or a tab character. Of course you can put a literal space in the search condition, the way you'd normally write it, but the notation [a-z\s] is much clearer and more readable than [a-z ] - at a glance you can see that the first character class includes a space, whereas with the second character class you have to look closely, and since regular expressions already look like a bunch of squiggles to a lot of people, it's very easy to overlook a space written this way. Use this special character carefully, because besides matching a space and a tab, it also matches a newline character. * \S - simply put, these are mostly visible characters, i.e. everything that does not match \s * \w - a special character meant to stand in for an entire character class; it includes all characters that can be part of a word, usually this is [a-zA-Z_], though a lot can depend on the locale set, unicode support, etc. * \W - everything not included in the definition of \w, i.e. [^a-zA-Z_] * \d - all digits, i.e. the character class you already know: [0-9] * \D - everything that is not a digit
As you can see, special characters often describe some frequently
used sets of characters that programmers use every day, but these
sets have boundaries, i.e. either digits, or letters and
underscore, or whitespace characters, but how do you describe all
characters? Simple! Put a dot!
And then you'll ask, what do you do if you need to describe exactly
a dot, and not all characters? Put a backslash before the dot: \.
But you won't calm down, and you want to know what to do if you're
searching in the text for a backslash followed by a dot. It's
simple, as you can see, the backslash is part of practically every
special character, and is also needed to turn a special character
into a literal, which means this backslash is itself a special
character, and in order to turn it into a literal, you need to
follow the same rules that were used when turning the dot from a
special character into a literal, namely, place another special
character in front of the special character; so as not to invent
any more special characters, it was decided to use the same slash.
So, to put a backslash as a literal in a search condition, you need
to double it, like this: \\
Similarly, to put two backslashes, you also need to double them,
like this: \\\\
Alternatives
To be or not to be... The eternal alternative! After reading this
part, you'll be able to write Hamlet's alternative yourself using
regular expressions. First, as always, let's sort out the data that
needs to be processed. Hamlet had a choice between being (be) and
not being (not to be). Either way he got some result out of it,
which was equal to true :) To describe a choice condition using
regular expressions, you first need to decide what you're choosing
between. In Hamlet's case, we're choosing between groups of
literals, one group consists of two literals in a row, b and e, the
second group consists of nine literals in a row: n, o, t, \s, t, o,
\s, b, e
It's clear that \s represents one space. I think I used a new word
that you didn't pay attention to: group of literals. A group of
literals is a sequence of characters described either by character
classes or by literals themselves. A group of literals is described
in parentheses, which also save the matched group of literals into
special variables, which I wrote about in the previous article.
Here are examples of groups of literals:
(be) (not\sto\sbe)
Now let's go back to choosing between two groups of literals:
(be)|(not\sto\sbe) this vertical bar | between the groups of
literals is exactly the choice condition, read as 'or'. Now imagine
that Hamlet's answer is processed by a computer, which knows that
there can only be two answers, either "to be" or "not to be". All
other answers simply aren't accepted as answers at all. We write a
regular expression to check Hamlet's answers:
preg_match("/^(be)|(not\sto\sbe)$/", $alternate, $answer);
The regular expression will match if
* $alternate equals "be"
* $alternate equals "not to be"
At the beginning of the explanation I asked what we'd be choosing
between. I think it's time now to figure out what you can choose
between at all. You can choose between literals and between groups
of literals. As already mentioned, groups of literals are combined
with parentheses; if you need to choose between single literals,
then the two literals with a vertical bar between them need to be
grouped with parentheses.
Example of choosing between two literals: s(o|u)n matches both son
and sun.
Example of choosing between two groups of literals: (son)|(sun)
similarly matches son and sun
In the case of choosing between groups of literals, or between
single literals, the literals are combined using parentheses, but
in the first article, in the "Remember everything" section, I said
that everything enclosed in parentheses is saved into special
variables. Saved means it uses the "memory" resource. What do you
do if we don't need to remember the highlighted groups of literals,
but do need to group them? You need to make the parentheses not
save! This is done using the sequence ?: like this:
(?:be)|(?:not\sto\sbe) now neither the group of literals "be" nor
the group of literals "not to be" will be stored in the variables
\1, \2 (Perl $1, $2), but they will still be grouped perfectly
well.
It's clear that if you need to put the literal "|", you need to put
a special character before it that will indicate that in this case
the vertical bar is a literal. This special character is the
backslash: \|
Example for the chapter
An administrator has a list of users generated by a program or
utility; it generates this list in the format:
last_name first_name middle_name
last_name f m
last_name f.m.
As always, the boss needs some statistics, but he's not at all
interested in either first names or middle names, he needs the last
name and initials in the form:
last_name fm.
Something needs to be done... I did it like this (the loop over all
the lines that the utility generates, you'll write yourself):
preg_match("/([^\s]+)\s+([^\s.])[^\s.]*(?:\s|\.)([^\s.])[^\s.]*/",$income_str,$out_arr); print_r($out_arr);
First let's sort out the data coming in, so that the regular
expression works 100% of the time. There's no need to validate the
data against the condition, since the data stored in the system has
already passed this check when it was entered. The last name, first
name and middle name can all consist of any characters, since we
don't know which characters the system allows and which characters
the user entered into the system, but it's known for certain that
the last name is separated from the first name by a space, the
first name and middle name can be given in full, can consist of a
single abbreviation letter, can be separated by a space, or a dot
can follow each letter.
If you learn to compose such data descriptions yourself, for what
you need to search, then most of your work is done, because all
that's left is to write the previous sentence in the language of
regular expressions:
* [^\s] - any character that is not a space! including the newline character; you could write \S without a caret at the start of the character class, but that's a matter of taste. * [^\s]+ - at least one character that is not a space, i.e. we've already described the last name. * \s+ - at least one space between the last name and the first name
Several ways are used to denote the first name:
* full first name followed by a space * first letter of the first name followed by a space * first letter of the first name followed by a dot
What do we need to search for? Everything after the space between
the last name and first name up to the next space or dot, and the
first name or its abbreviation consists of at least one letter.
* [^\s] - everything that is not a space, the first mandatory character of the first name (abbreviation) * [^\s.] - everything that is not a space and not a dot, in the character class the dot is a literal * [^\s.]* - in the case of a full first name, this will mean we need to find everything that is not a dot and not a space and follows the mandatory first character of the name we described above
It turns out that we described the first name with this
construction: [^\s.][^\s.]* but the first letter of the first name
(even if it's an abbreviation) needs to be remembered according to
the task condition: ([^\s.])[^\s.]*
What comes after the first name? In any case, there's either a
space or a dot there. This means we need to pass through this
segment so that the choice condition between two characters is
satisfied: (\s|\.) - the choice between two literals is achieved by
grouping them in parentheses and writing a vertical bar between the
literals. If the string is guaranteed to have only one space, you
can leave it like that; if you're not sure, put a '+' character
after \s: (\s+|\.)
Do you need to remember what stood between the last name and the
middle name, a space or a dot? I don't! So out of the capturing and
grouping parentheses, we make only grouping ones: (?:\s|\.)
And now look at the middle name from the point of view of regular
expressions, the characters it's made of follow the same rules as
the first name, so for the description of the rest of the regular
expression, right to the end, see above.
Regular Expressions, Part II
Using Positional Assertions
To many, the material in this article will seem unnecessary, since
most tasks using regular expressions are solved with the means I
described in the previous one.
First, let's digress a little. How would you describe something
inconspicuous, or something that's simply hard to describe? I think
you need to describe something conspicuous, or something easy to
describe, and then indicate where the sought-for "inconspicuous
something" is located relative to the described "conspicuous
something".
Example: My director asks how to find such-and-such bank in Kyiv. I
know that explaining the location of the bank to a Dutch director
using transliterated Russian names is impossible! But I'm a
programmer and I explain that the bank we need is on the central
square. At the same time I describe the square by saying that on
one side stands a woman on a huge column, and on the other side
there's a McDonald's. Even people who've never been to Kyiv guessed
that I meant Independence Square.
In regular expression syntax, there is a way to describe what comes
before, and what comes after, the part of the string that we need
to find. Consider that there's a way to describe the location of
the bank without naming the exact address. If you manage to
describe what the bank is located between, and also manage to
describe that it's specifically a bank and not a cafe, then
consider that you don't need to give the exact address.
Every time, using regular expressions, a search for matches takes
place. The search is carried out over a string, a substring is
searched for within the string. Now let's learn to specify a
condition for what must match before the substring and after it.
The match condition that comes before the substring is called a
lookbehind assertion, the match condition that comes after the
substring is called a lookahead assertion. It can also be explained
that a lookbehind matches to the left, a lookahead - to the right.
Positive and Negative Assertions
It's good if you can describe what comes before and what comes
after the substring you're looking for. But what if you need to
describe what does NOT come before or after the substring you're
looking for? For this, the lookahead and lookbehind assertions are
each further split into two:
* Positive lookahead, negative lookahead assertions.
* Positive lookbehind, negative lookbehind assertion.
A positive lookahead assertion specifies that the condition
describes what must necessarily be present after the substring
being searched for. A negative lookahead assertion describes what
must under no circumstances stand after the string being searched
for. Similarly with a lookbehind assertion, except the condition is
searched for before the substring being searched for.
Lookbehind Assertion
A positive lookbehind assertion, or a check to the left, is
described by a special sequence of characters: (?<=) the
parentheses group the condition, the question mark shows that the
contents of the parentheses should not be treated as a grouping of
characters to remember them, but as a grouping of characters to
perform some other action on the expression inside the parentheses.
Recall that plain parentheses describe a group of characters and
save everything inside them into variables, while parentheses with
a question mark and colon indicate a special action: grouping
without saving. The character(s) after the question mark are
interpreted as an instruction for exactly which action needs to be
performed. In total: the greater-than-or-equal sign following the
question mark, which stands inside parentheses right after the
opening parenthesis, shows that the following characters are a
positive lookbehind assertion, i.e. a condition for what must
necessarily precede the substring we're searching for.
A negative lookbehind assertion is described by a similar sequence
of characters: (?<!) Notice that the equals sign has changed to an
exclamation mark, and the assertion has immediately changed
"polarity", i.e. turned from positive to negative. The logic is
simple if you recall that to check the condition "not equal", a
negation of equals, many programming languages use the sequence of
characters: !=
Lookahead Assertion
If you understood the description of the lookbehind assertion, then
understanding the essence of the lookahead assertion won't be
difficult for you.
To describe a positive lookahead assertion, the sequence of
characters is used: (?=) everything as before, the parentheses
indicate that the characters are grouped for some action, the
question mark indicates that the next character will describe
exactly what action will be performed on the group of characters,
or how this group of characters will be interpreted. The '='
character, which comes after the question mark, indicates that the
expression in parentheses is a lookahead assertion, a check of what
must match to the right of the substring being searched for.
To describe a negative lookahead assertion, the sequence of
characters is used: (?!) the same thing, except the equals sign is
replaced by the negation - an exclamation mark.
Example No.1
A common problem is parsing data of interest to a programmer out of
HTML that isn't always of good quality; it would all be tolerable
if it weren't for the javascript inserts; here's an example of such
text:
<TD>20.02<BR>05:30 <TD class="l">Product 1<BR>Product 2 <TD><B>35</B> <TD><A href="http://link/" id=sfsd32dfs onclick="return m(this)">26.92</A><BR><A href="http://link/" id=r3_3143svsfd onclick="return m(this)">27.05</A> <TD><B>270.5</B> </TR>
The numbers written with a dot are prices. The task is to collect
all the prices located between the <a>... </a> tags
We see that, besides the prices between the given tags, there are
some that come right after the <TD> tag, as well as ones between
the <B>...</B> tags. It's clear that describing the contents of the
attributes of the <A> tag precisely enough is not the easiest task,
so we need to simplify it! Any tag has a closing character '>', our
task is to describe that this character comes before the price, but
since a <B> tag and a <TD> tag can also come before a price, and we
don't need those prices. How will we know that a price stands
between the <A>...</A> tags? By the tag that comes after the price:
if it's not the </B> tag, then it will be either the </A> tag or
<BR>, and also by the tag before the price, if that tag is <TD>.
Through this reasoning we've concluded what should be on the right,
and what should be on the left, of the string we're looking for,
which is described as digits separated by a dot: \d*\.\d*
What must match on the left, we described as the character '>', we
write: (?<=>) - looks a little strange, but a match on the right is
written like this (?<=), and inside it, after ?<=, comes the
character '>'
What must match on the right is described by (?=) inside which we
write </A>.
Now let's describe what must not come before the price:
(?<!<TD>) the <TD> tag must not come before the price, this is a
negative lookbehind assertion.
Using a negative lookahead assertion we describe what must not come
after the price: (?!<\/B>) the </B> tag must not come after the
price.
The resulting regular expression, which describes all the given
conditions, looks like this:
preg_match_all("/(?<!<TD>)(?<=>)\d*\.\d*(?!<\/B>)(?=<\/A>)/", $string, $matches); print_r($matches);
After looking at the first example, some remarks and clarifications
about using positional assertions are in order.
1. Assertions written one after another are applied independently of each other at one point, without moving it. Naturally, a match will be found if all the assertions match. In our example these were the checks before and after the price. From the point of view of the logic of applying the assertions, it makes no difference whether the check for the <TD> tag comes before the check for the '>' character. Though, from an optimization point of view, the first positional assertion should be the one with the highest probability of not matching. 2. The values matched by lookbehind assertions are not saved. That is, if in our example the lookahead assertion matches, which indicates that the </A> tag comes after the price, then the </A> tag itself, which is enclosed in the (?=) construct, will not be stored in the special variables \1, \2, etc. This is because a positional assertion doesn't match a string, but a position in the string (it describes the place where the match occurred, not the characters that matched). 3. It should be noted that PCRE does not allow checking for a match of text of arbitrary length. That is, you cannot, for example, do a check like this: /(?<=\d+)
The matching engine for lookbehind assertions is implemented in
such a way that during the search, the engine must be fed a string
of fixed length, so that in case of a mismatch, the engine can go
back a fixed number of characters and continue searching for
matches in other positional assertions. I think this is hard to
grasp right away, but imagine how the match search happens in the
part (?)(?<=>) of the regular expression described above. The
string being searched is taken, and starting from the beginning as
many characters are counted off as there will be in the match of
the positional assertion, in our case that's 4: <, T, D, > from
this point a "look back" happens (lookbehind assertions in English
sound like lookbehind assertions), i.e. all the previous 4
characters are checked for a match with the string <TD>, if the
engine didn't find a match, it needs to go back 4 characters, do
the same thing with the check (?<=>), i.e. count off one
character, "look" back, try to find a match of the previous
character with the character '>'. Imagine that the match condition
consists of a string of non-fixed length: (??) such a notation
should mean that before the price there must not be a <TD> tag, at
most one instance (or not at all).
So it turns out that after the engine counts off 4 characters from
the beginning, it will check for a match with <TD>, but the
condition states that the tag might not be there at all, then the
question arises, how many characters back should it go to check the
other assertions for a match. 4, or not go back at all?
The question immediately arises, why go forward just to then
"look" back? This is done so that, in case all the assertions
match, the check of the characters that come after the positional
assertions can begin right away.
Example No.2
At some point I needed to get all the images that were used on a
site. What do you need to do for this? Right, you need to click
"Save as" in the browser, specify where to save the page. A file
with the page's source code and a folder with images will appear.
But you will never get saved into that folder the images that are
specified in the styles of objects, at least in Explorer:
style="background-image:url(/editor/em/Unlink.gif);"
To carry out the above operation you need to:
1. ask the host owner for permission to use the content hosted on
their site.
2. find in the text all the strings similar to the one shown
above, and extract the relative path to the file from them
3. build a file in which images will be output using
<img src=full_path_to_image>
Let's do it: We get the page's source code into the $content
variable. Then, using regular expressions, we search for the
relative paths specified in the styles. Every time I describe how I
implemented an example, I first carefully describe what we're
looking for, and carefully describe in what context the search
takes place. Having analyzed the page's source code, it became
clear that, apart from in style descriptions, relative paths to
images are not used anywhere else. To the left of the relative path
comes the sequence of characters: url(
To the right of the relative path is a closing parenthesis.
Between these sequences of characters there can be letters of the
Latin alphabet, digits and slashes, as well as a dot before the
file extension.
Let's start simple. Latin alphabet characters, digits, dot and
slash are described by the character class: [a-z.\/] there can be
any number of them, in fact more than 3 (file name, at least one
character, dot, extension, at least one character), but in this
case, knowing the context, this isn't critical, so we specify the
quantifier * [a-z.\/]*
To the left there must be 'url(' and we describe this using a
positive lookbehind assertion: (?<=url()
But notice that the parenthesis is a grouping metacharacter in
regular expressions, so for it to become a character, we need to
put another metacharacter in front of it - a slash. (?<=url\()
To the right of the relative path there must be a closing
parenthesis. This condition is described using a positive lookahead
assertion: (?=\))
As you can see, before one of the parentheses there is a slash,
which means it is interpreted not as a metacharacter, but as a
literal.
Below is the complete PHP code that performs all the actions,
except for the matter of getting permission to use the content:
preg_match_all("/(?<=url\()[a-z.\/]*(?=\))/i", $content, $matches); foreach($matches[0] as $item) { echo "<img src = http://www.a???s.ru".$item.">"; }
When and Why You Should Not Use Regular Expressions
This note was written based on discussions on one of the forums and
is intended to warn programmers so that they don't feel the
omnipotence of regular expressions when working with string data.
I'll try to explain in which cases it's worth looking for a more
suitable solution to your problem than using regular expressions.
The description of the problem posed on the forum was this: There
is a certain page, created as a result of a template engine's work,
it contains elements like <input type=checkbox>, an authorized
user works with this page and checks some checkboxes <input
type=checkbox checked>, after the form is submitted to the server
the pages are generated again by the template engine and the
programmer arranges for this page to be saved in the cache, already
with the checkboxes checked.
The programmer's task: Based on the results of the submitted form,
he needs to remember the selected checkboxes and, on this user's
subsequent visit to this page, display his choice with the elements
already checked.
What he does: Instead of taking the template, selecting the data
from the database and marking the checkboxes anew based on it, by
running the template through the template engine, the programmer
takes the page saved in the cache and parses it using regular
expressions; depending on the data he selected for this user from
the database, he either removes data from the page or adds new
'checked' attributes in the right places in the page's text.
His argument: This works faster than processing the original
template.
Let me try to give my arguments against this: What's the point of
caching if afterwards the data from the cache has to be changed
using regular expressions? The cache exists precisely so that you
don't have to run resource-intensive processes an extra time, such
as, for example, processing the ENTIRE text with regular
expressions... multiple times.
Let's imagine a text with n characters, as well as a regular
expression in which
1. there are three choice conditions part1|part2|part3| 2. each of the parts parti consists of character classes, as well as given sequences of literals. 3. the maximum length of a match of the regular expression equals m characters. 4. the programmer used the preg_replace function; we know the operating principle of the regular expression engine, which means we can replicate its work.
Let me give these steps together with the arguments against the
approach the programmer wanted to apply, which follow from them:
1. we take a character from the text at index 1 and try to apply the regular expression to the text starting with this character. We go through every character starting from the first one and compare it to the pattern (does it fit). 2. following each choice instruction in the regular expression we can make at most three choices and go through m subsequent characters for EACH i-th character of the text. 3. it's not hard to calculate how many characters will be scanned in vain if no match is found. 4. the cached document is already the result of a regular expression's work (many template engines run on regular expressions), so an additional call to the regular expression engine is simply being made to parse a raw, half-finished document, instead of making it ready in one pass right away. 5. if you compare the regular expressions that template engines run on with the regular expression our programmer ends up with, you'll immediately see that the former are simpler! For this reason matches will be found faster. 6. often the work of regular expressions in a template engine will happen on small pieces of text (it's no secret that pages can be assembled from small chunks), rather than on the generated page all at once.
I hope I managed to explain why regular expressions, while a
powerful tool for working with strings, are not suited to solving
this kind of large-scale task; you need to learn to set tasks for
yourself and be able to solve them in several ways, then you'll
have a choice in favor of the better solution. Regular expressions
are just one of many methods, but by no means a panacea.
The note describes a case where a person knew how to use regular
expressions, and also knew how to use a template engine in their
work, i.e. they had a choice, they simply couldn't settle on one of
the options. There is a category of beginner programmers who,
having seen a bunch of symbols with which many string problems can
be solved, build themselves an idol in the form of a statue to this
"magic tool" and don't even realize that their task of replacing
one word with another in a string is solved by the simple function
str_replace(). Not understanding how regular expressions work, they
come to a forum and ask how to replace abcd with asdf using
preg_replace, and get very offended when given a link to
str_replace in the manual at http://www.php.net/. This is probably
the most common case of misuse of regular expressions, when they
could be replaced using string-handling functions.
Comments