Regular expressions ("regexps," from the English Regular Expressions) are a powerful tool for building patterns that can be used to search and match characters of any complexity within a given text.

How is such a pattern constructed? Special characters, metacharacters, and character classes (sets) are used for this. A regular expression is a simple string, and any characters in this string that are not special (reserved) are treated as ordinary characters.

Regular expressions (regex or regexp) are very effective for extracting information from text. To do this, you need to search for one or more matches against a specific pattern (i.e., a specific sequence of ASCII or Unicode characters).

The areas of application for regex are diverse, ranging from validation to parsing/replacing strings (essentially one-dimensional pattern recognition), converting data into other formats, and Web Scraping.

One of the curious features of regular expressions is their universality — once you learn the syntax, you can apply it in (almost) any programming language (JavaScript, Java, VB, C #, C / C++, Python, Perl, Ruby, Delphi, R, Tcl, and many others). The small differences concern only the most advanced features and syntax versions supported by the engine.

Special (reserved) characters are divided into three classes:

• the first denotes some class (set) of characters (for example, \w means any letter)
• the second, unlike the first, has no length (for example, ^ - beginning of the string, \b - beginning of a word)
• the third class consists of operators. Operators are applied to metacharacters, ordinary characters, or other operators.

Any expression can be grouped (enclosed in parentheses) and an operator applied to the whole group.

The regular expression syntax used in nnCron matches the regular expression syntax of the Perl language. Small differences exist only in some extended, specific operators.

• Syntax
• Special characters
• Extended special characters
• Operators
• Extended operators
• Optimizing regular expressions
• Regular expressions cheat sheet
• Comparing regular expression capabilities in Perl, Python, and Emacs
• Conclusion, areas of use for regexes

Syntax

All regexps must be enclosed in forward slashes (/.../). After the final slash, parameters (flags) may follow:

Примеры:

\ совпадет только со словом 'Valery'
/Valery/
\ совпадет со словами 'VALERY', 'valery', 'Valery' и т. д.
/Valery/i 
\ совпадет с 'foobar', 'foobar barfoo'
/foobar/
\ совпадет с 'foobar', 'FOOBAR', 'foobar and two other foos'
/ FOO bar /ix
\ совпадет с 'Valery%crlf%Kondakoff'
/Valery.*Kondakoff/s

Each character of a regular expression is compared sequentially against the string being tested. Anything that is not one of the special characters or operators listed below is treated as an ordinary character, checked for a simple match.

Special characters (metacharacters)

Examples:

\ matches the word 'help' with a period
/help\./
\ matches the words 'cats', 'cars', etc.
/ca.s/
\ matches the words 'testing', 'tester', but not 'the test'
/^test/
\ matches the phrase 'see me', but not 'meter' or 'me and you'
/me$/
\ matches a single (Latin) vowel letter
/[aeiou]/
\ matches a single letter or digit
/[a-z0-9]/
\ matches 'footer', 'footing', 'a foot', but not 'afoot'
/\bfoot/
\ matches 'afoot', 'foot.' (the period is not considered part of the word)
\ does not match 'footing'
/foot\B/
\ matches the whole word 'foot'
/\bfoot\B/
\ matches the words 'q2w', 'r5t', etc.
/\D\d\D/

Extended special characters

Unlike the regular characters, these classes are not compatible with Perl's:

Example:

\ matches the phrases 'man to man', '
\ hand to hand', '100 to 100', etc.
(\b\w+\B) to \1

Operators

Operators cannot be used on their own, without specifying the character they act upon. An operator acts on the character (meta or ordinary) that precedes it. If an expression is enclosed in parentheses followed by an operator, the operator acts on the entire parenthesized group.

Examples:

\ matches the words 'cat' or 'mouse'
/(cat)|(mouse)/
\ matches the words 'dogs', 'doggie'
/dog(s|gie)/
\ matches 'ma', 'maaa', 'maaaaaaa'
/ma+/
\ matches 'm', 'maaa'
/ma*/
\ matches 'yada yada yada'
/(yada ){2,}/
\ matches 'fooandbar', 'foobar'
/foo(and)?bar/

If you add ? after an operator, it turns from greedy into lazy. For example, the greedy * becomes lazy after being replaced with *?. Greedy operators capture the maximum amount in a string, while lazy ones capture the minimum.

Extended operators

Note that although the last two operators also exist in Perl, there they are written as (?=foobar). In nnCron, the operator looks like (foobar)?=.

Examples:

\ matches any word followed by a tab character
\ the tab character itself will not be included among the matched characters
/\w+(\t)?=/
\ matches any occurrence of 'foo' that is not followed by 'bar'
/foo(bar)?!/
\ matches any occurrence of 'bar' preceded by 'foo'
/(foo)?#3bar/ 

A few more examples:

\ matches "foobar", "bar"
/(foo)?bar/
\ matches _only_ "foobar"
/^foobar$/
\ matches "foobar", "for", "far"
/f[obar]+r/
\ defines any number with a decimal comma
/([\d\.])+/
\ matches "foofoofoobarfoobar", "bar"
/((foo)|(bar))+/

Basic regular expression constructs

Anchors — ^ and $

^Hello        matches a string starting with Hello  bye$          matches a string ending with bye^Hello bye$  exact match (starts and ends with Hello bye)sparrows      matches any string that contains the text sparrows

Quantifiers — * + ? and {}

abc*       matches a string where, after ab there are 0 or more occurrences of c abc+       matches a string where, after ab there is one or more occurrence of cabc?       matches a string where, after ab there are 0 or 1 occurrences of cabc{2}     matches a string where, after ab there are 2 occurrences of cabc{2,}    matches a string where, after ab there are 2 or more occurrences of cabc{2,5}   matches a string where, after ab there are from 2 to 5 occurrences of ca(bc)*     matches a string where, after ab there are 0 or more sequences of the characters bca(bc){2,5} matches a string where, after ab there are from 2 to 5 sequences of the characters bc

The OR operator — | or []

a(b|c) matches a string where a is followed by b or c a[bc]  same as in the previous example

Character classes — \d \w \s and .

\d matches a single character that is a digit  \w matches a word character (can consist of letters, digits, and underscore) \s matches a whitespace character (including tab and line break). matches any character 

Use the . operator with caution, since a character class or a negated character class (which we will look at later) is often faster and more precise.

The operators \d, \w, and \s also have negations ― \D, \W, and \S respectively.

For example, the \D operator will search for matches opposite to \d.

\D matches a single character that is not a digit 

Certain characters, such as ^.[$()|*+?{\ , must be escaped with a backslash \ .

\$\d matches a string in which the character $ is followed by a single digit  

Non-printable characters can also be searched for, such as tab \t, newline \n, carriage return \r.

Flags

We have learned how to build regular expressions, but we forgot about a fundamental concept ― flags.

A regular expression is usually written in the form /abc/, where the pattern to match is delimited by two slashes /. At the end of the expression, we specify the flag value(s) (these values can be combined):

• g (global) ― does not return a result after the first match, but continues searching from the end of the previous match.
• m (multi line) ― with this flag, the operators ^ and $ will match at the beginning and end of an input line, instead of the entire string.
• i (insensitive) ― makes the expression case-insensitive (for example, /aBc/i matches AbC).

Intermediate level

Capturing groups ― ()

a(bc)       creates a group with the value bc  a(?:bc)*    the operator ?: disables the group  a(?bc) this way, we can assign a name to the group  

This operator is very useful when you need to extract information from strings or data using your favorite programming language. Any multiple matches, across several groups, will be represented as a classic array: their values can be accessed using an index from the match results.

If you assign names to the groups (using (?...)), you can get their values by using the match result as a dictionary, where the keys are the names of each group.

Bracket expressions ― []

[abc]       matches a string that contains either the character a or a b or a c -> the same effect as a|b|c  [a-c]       same as above[a-fA-F0–9] a string representing a single hexadecimal digit, case-insensitive  [0–9]%      a string containing a character from 0 to 9 before the % sign[^a-zA-Z]   a string that does not have a letter from a to z or from A to Z. In this case ^ is used as negation in the expression - 

Remember that inside bracket expressions, all special characters (including the backslash \) lose their special meaning, so we do not need to escape them.

Greedy and lazy matching

Quantifiers ( * + {}) are «greedy» operators, because they keep searching for matches as deep as possible ― through the entire text.

For example, the expression <.+> matches

in This is a

test. To find only the div tag ― you can use the ? operator, making the expression «lazy»:

<.+?> matches any character, found one or more times between < and >, expanding as needed  

Note that it is considered good practice not to use the . operator, in favor of a more strict expression:

<[^<>]+> matches any character except < or >, occurring one or more times between < and >  

Advanced level

Word boundaries ― \b and \B

\babc\b searches for the whole word  

\b ― matches a word boundary, like an anchor (similar to $ and ^), where the preceding character is a word character (e.g., \w) and the following one is not, or vice versa (for example, this can be the beginning of a string or a space).

\B ― matches a non-word boundary. A match must not be found at a \b boundary.

\Babc\B matches only if the pattern is completely surrounded by word characters 

Backreferences — \1

([abc])\1 \1          matches the text from the first capturing group -> ([abc])([de])\2\1     you can use \2 (\3, \4, etc.) to specify the sequential number of the capturing group (?[abc])\k  we assigned the name foo to the group, and now we reference it using ― (\k). The result is the same as in the first expression  

Lookahead and lookbehind assertions — (?=) and (?<=)

d(?=r) matches d, only if it is followed by r, but r will not be included in the expression's match ->  (?<=r)d matches d, only if it is preceded by r, but r will not be included in the expression's match  

You can use the negation operator !

d(?!r) matches d, only if it is not followed by r, but r will not be included in the expression's match ->  (?r)d matches d, only if it is not preceded by r, but r will not be included in the expression's match 

Regular expression examples

The examples below show how to use and construct simple regular expressions. Each example contains the text being searched for, one or more regular expressions that match it, and notes explaining the use of special characters and formats.

• Searching for an exact phrase
• Searching for a word or phrase from a list
• Searching for a word with different spellings or special characters
• Searching for any email address in a specific domain
• Searching for any IP address in a specific range
• Searching for an alphanumeric string

For more information, review the tips on creating content filters using regular expressions and visit the GitHub page dedicated to RE2 syntax. Also read the article on how to configure content matching rules.

Important! Only RE2 syntax is supported, which differs slightly from PCRE. Note that regular expressions are case-sensitive by default.

Note. Based on the examples given below, you can build more complex regular expressions. However, for searching individual words we recommend using the Content Match and Unwanted Content settings.

Password strength validation

Password strength is a rather subjective concept, so there is no universal solution for validating it. However, the regular expression example given above can serve as a good starting point if you don't want to come up with a password validation expression from scratch.

Hex color code

Hexadecimal color codes are used very often in web development. This regular expression can help check whether any given string matches the hexadecimal code pattern.

Email address validation

One of the most common tasks in development is validating that a user-entered string matches the format of an email address. There are many different variations of expressions for solving this task; the author of this article offers his own original variant.

IP address (v4)

Just as an email can be used to identify a visitor, an IP address is an identifier for a specific computer on a network. The regular expression given checks whether a string matches the IPv4 address format.

IP address (v6)

You can also check whether a string matches the newer, sixth-version IP address format using this more advanced regular expression.

Thousands separator in large numbers

Traditional separators in large numbers are commas, periods, or other marks, repeating in the number every 3 digits. The regular expression code given works with any number and any character you define for separating the three-digit groups in large numbers: thousands, millions, etc.

Adding a protocol before a hyperlink

Regardless of which language you work with: JavaScript, Ruby, or PHP, this regular expression can be very useful. It is used to check any URL for the presence of a protocol in the string, and if the protocol is missing, the given code adds it to the beginning of the string.

Getting the host or domain from a URL.

As is known, any URL consists of several parts: first the protocol is specified (HTTP or HTTPS), sometimes followed by a subdomain, and finally the path to the page is added. You can use this expression to return only the domain name, excluding all other parts of the address.

Sorting keyword phrases by word count

These are really useful expressions for users of Google Analytics and Webmaster Tools. With them, you can sort the keyword phrases used by visitors in searches by the number of words they contain.

These expressions can check phrases containing a specific number of words (for example, 5), as well as phrases where the number of words is more than two, three, etc. This is one of the most powerful expressions used for sorting analytics data.

Finding a valid Base64 string in PHP

If you are a PHP developer, you may sometimes need to find an object encoded in Base64 format. The expression given above can be used to search for encoded strings in any PHP code.

Phone number validation

This regular expression is used to validate any phone number, primarily in the American phone number format.

To validate Russian phone numbers, use the following expression:

However, in reality a phone number is simply a set of up to 15 digits with a plus sign at the beginning, WITHOUT hyphens or parentheses. In this format, numbers cannot begin with 0; if they begin with zero or 8, then this is possibly some other local or outdated format

Leading and trailing whitespace

Use this regular expression to get rid of leading and trailing spaces in a string. This is not a particularly common task, but this expression can sometimes be useful. For example, when retrieving data from a database or passing a string to a script in a different encoding.

Getting the HTML code of an image

If for some reason you need to «pull out» the HTML code of an image directly from the page's code, this regular expression will be an ideal solution for you. Although it can work without problems on the server side, for front-end developers it would be preferable to use jQuery's attr() method instead of this regular expression.

Checking a date against the DD/MM/YYYY format

Validating dates is difficult because they can be presented in various formats, including ones containing both numbers and text.

PHP has an excellent date() function, but it's not always suitable, since an unprocessed string may be passed to it. Therefore, to validate the specified date format, you need to use the regular expression given above.

Matching a string to a YouTube video URL

For several years now, the structure of YouTube's URLs has not changed. YouTube is the most popular video hosting site on the Internet, and thanks to this, videos from YouTube generate the most traffic.

If you need to get the ID of any YouTube video, use the regular expression given above. This is the best expression, suitable for all variants of URLs on this video hosting site.

ISBN validation

Information about all printed publications is stored in a system known as ISBN, which consists of 2 systems: ISBN-10 and ISBN-13. It is very difficult for a non-specialist to see the differences between these systems. However, the regular expression presented above allows you to check ISBN code compliance with both systems at once: whether it's ISBN-10 or ISBN-13. The code is written in PHP, so this solution is suitable exclusively for web developers.

Zip code validation

Please note that this expression is only suitable for validating American zip codes. For postal codes of other countries, adjustment is required.

To validate Russian postal codes, use the following expression:

Validating a Twitter username

This small regular expression helps find a Twitter username within text. It checks for the presence of a name in tweets matching the pattern: @username.

Credit card number validation

Credit card number validation is very often performed when making payments in various online payment systems. However, the regular expression provides only minimal validation of a standard credit card. Some closed countries have non-standard card numbers, for example in China or Russia.

You can review a more complete list of codes for detailed card validation. The list includes systems such as Visa, MasterCard, Discover, and many others.

Finding CSS attributes

A situation where you have to use this regular expression may arise very rarely, but it's not a given that it will never arise

This code can be used when you need to «pull out» some CSS rule from the list of rules for a given selector.

Removing comments in HTML

If you need to remove all comments from a block of HTML code, use this regular expression. To get the desired result, you can use the PHP function preg_replace().

Checking for a match against a Facebook account link

If you need to find out your site visitor's Facebook page address, try this regular expression. It will help you verify the correctness of the URL entered by the user. This code is great for checking links on this social network.

Checking the Internet Explorer version

Despite the fact that Microsoft released the new Edge browser, many users still use Internet Explorer. Web developers often have to check the version of this browser in order to account for the features of different versions when working on their projects.

You can use this regular expression in JavaScript code to find out which version of IE (5-11) is being used.

Parsing a price from a string

The price of a product may be specified in various formats: it may contain commas, decimal digits, and currency symbols.

The regular expression given above takes into account various price display formats; with its help you can «pull out» the price from any character string.

Parsing headers in email

Using this small expression, you can parse an email message header to extract the list of recipients from it. The expression can also be used in cases where there are multiple recipients.

Instead of regular expressions, you can use a PHP library to parse email headers.

Matching a filename to a specific type

If your application has the ability to upload files to the server, this regular expression can help you validate files before a visitor uploads them.

With this code, you can get the extension of the uploaded file and check whether it is present in the list of allowed extensions for upload.

Matching a string to the URL format

The regular expression can check URLs with HTTP and HTTPS protocols specified for compliance with TLD domain syntax.

There is a simple way to validate this using JavaScript RegExp.

Adding the rel=”nofollow” attribute to a link tag

If you work a lot with HTML code, you will want to automate frequently repeated actions. Regular expressions are great for solving this task and will save you a lot of time.

Using the code given, for example together with PHP, you can «pull out» link code from blocks of HTML code and add the rel=”nofollow” attribute to each of them.

Working with media queries

You can split strings containing media queries into parts consisting of parameters and properties. This expression can be useful for analyzing third-party CSS code. Using it, you can, for example, understand in more detail how someone else's code is structured.

Google search query syntax

You can build your own regular expressions to manipulate the search results of your queries in the Google search engine. For example, the plus sign (+) adds additional keywords, while the minus sign (-) means that words should be ignored and removed from the results.

This is a fairly complex expression, but if you figure out how to use it properly, the code given can become the basis for building your own search algorithm.

Unused characters in regular expressions. Optimizing regular expressions

Regular expressions with the following characters are not supported, as they can lead to delays in processing your message:

• * (asterisk)
• + (plus sign)

You can optimize a regular expression by using atomic grouping (Atomic Grouping) or by using possessive quantifiers (Possessive Quantifiers) where possible.

In addition, if you have things like .* or .+ in your regular expression, which can be real memory/runtime problems, replace them with (possessive) character classes/sets (Character Classes/Sets) (again, if possible).

What is optimization?

Here we're talking about both «micro-optimizations» and powerful optimization. On this page I took one trick from the syntax tricks page and a few optimizations that I found in Jeffrey Friedl's book Mastering Regular Expressions, and tested how one particular version of one particular engine (PCRE 8.12), used in one particular version of one particular language (PHP), responds to each of them. At some point I would like to run the same tests in .NET, Python, Java, JavaScript, and Ruby.

Unrolling alternation quantified by an asterisk

In the trick of simulating alternation quantified by a star, we see that can be unrolled into . Is there a time benefit from giving up alternation?

Using pcretest, I compared two patterns against this string: 44e223eae7e7e7e9876e14ou
Patterns: and
The original pattern (with alternation) compiles faster: 1.6 millionths of a second versus 2.2 for the unrolled version. However, it runs much slower: 1.7 millionths of a second versus 0.8.

This seems like a potentially useful optimization to implement at the engine level (it's a bit difficult to maintain in your own code).

When alternating, put the most likely cases first

The engine reads left to right. In the world of web addresses, dot-com occurs more often than dot-net and dot-biz, so if you're checking these three domains in a pool of random names, in theory you should write
\. (?: com | net | biz) \ b rather than
\. (?: biz | net | com) \ b
In practice, in PCRE there seems to be no difference - I ran the two patterns two million times and compared the results. If you want to run your own tests, here is the (very simple) code I used for this test.

Exposing literal characters

Regular expression engines match fastest when anchors and literal characters are placed directly in the main pattern, rather than hidden inside subexpressions. Hence the advice - «expose» literal characters whenever you can pull them out of an alternation or a quantified expression. Let's look at two examples.

Example 1: should be faster than . They mean the same thing: at least one A, possibly followed by more A characters.

I ran these two patterns two million times on the string BBBCCC. Both took the same amount of time. This tells me that the PHP PCRE engine must be «polite» regarding this optimization, i.e., it does it for you. Just use A +.

Example 2: should be faster than .

I ran these two patterns two million times on the string that. The second («less optimal») pattern was actually eight percent faster, earning me half a millionth of a second per run - nothing special. Again, the optimization must be built into the PHP PCRE regular expression engine. Perhaps, in theory, the «more optimal» pattern loses out somewhere in compilation.

Is it worth using?
As far as speed is concerned, the answer depends on the engine you are using. For me, regardless of which engine I use, the deciding factor is readability, and hence maintainability.

For example, if I'm matching all numbers from 10 to 19, I always factor out the 1 and use 1 [0-9]. For me this is easier to read and maintain than spelling out each number, especially when working with a more complex range of numbers.

On the other hand, if I were building a regular expression for all two-letter US state abbreviations, I would spell them out: \ b (?: AL | AK | AZ |…) \ b. I would do this even if I had tools on hand that automatically compress long alternations into their optimized equivalents (regex-opt in C, Regexp :: Assemble and Regexp :: Assemble :: Compressed in Perl).

That's because on any given day I would rather debug this:


than this:

Using and exposing anchors

Anchors at the beginning and end of a string, ^ and $, can save your regular expression a lot of backtracking in cases where a match is bound to fail.

In theory, ^. * abc fails faster than . * abc. I ran this two million times on a «failure string» (forty z characters in a row). As in the last example, the «less optimal» pattern was eight percent faster, earning me half a millionth of a second per run. Again, PCRE sounds polite. The lost time may be related to processing the extra anchor.

It's also recommended to expose anchors, meaning, whenever possible, to pull anchors out of alternation parentheses. For example,
^ (?: abc | def) is preferable to ^ abc | ^ def.

I ran each of these two preg_match functions two million times.

The first saved me one second (out of fourteen). On the one hand, that's a seven percent improvement.

On the other hand, per run, that's only an improvement of half a millionth of a second.

Is it worth using?
I use anchors everywhere I can, for the sake of good style - and to avoid unnecessary backtracking. As for whether to place them outside the alternation, I usually do that too, not because it's faster, but because it tends to be more readable.

Distributing into alternation

The last method I tried is what Jeffrey calls «distributing into alternation»:



This method really did speed up the script by seven percent, saving a millionth of a second per run.

Will I use it? Probably not. I like exposing boundaries.

Automatic optimization: study mode

PCRE has a «Study» modifier that can be added at the end of a pattern. For patterns that do not start with a fixed character and are not anchored, this modifier makes the regular expression engine study the string a bit more before applying the pattern, just in case some optimizations can be found.

To use this mode, add a capital S after the closing delimiter, for example:

Apparently, this study mode can be useful when analyzing long documents, such as web pages. It might not help, but it costs less than a hundred thousandths of a second.

You might think that the «micro-optimizations» in Mastering Regular Expressions don't speed up the code. Does that mean they're bad? On the contrary. However, I suggest you forget about «micro-optimizations» like the ones presented in this section. Good style matters more.

Just write a working regular expression, focusing on the big picture, to avoid patterns that slow you down by orders of magnitude. This mainly means the sensible use of anchors, quantifiers (lazy or greedy), groups (atomic or backtracking), and anything that can make your regular expression more specific than an overly generic soup of dots and asterisks - such as literal characters and negated classes.

Regular expressions cheat sheet

Comparing regular expressions in Perl, Python, and Emacs

Regular expressions are very convenient, but unfortunately they are implemented at least a little differently everywhere you use them. If I went into all the details, I'd have a book, not a page of notes.

Conclusion

As you have seen, the areas of application for regular expressions are diverse. I'm sure you've encountered similar tasks in your work (at least one of them), such as these:

• Data validation (for example, whether a time string is filled in correctly)
• Data collection, analysis, and recognition (especially web scraping, finding pages that contain a specific set of words in a specific order)
• Data processing (converting raw data into the required format)
• Parsing (for example, extracting all GET parameters from a URL or text inside parentheses)
• String replacement (even while writing code in an IDE, you can, for example, convert a Java or C# class into a corresponding JSON object, replace “;” with “,”, change letter case, avoid type declarations, etc.)
• Syntax highlighting, file renaming, package analysis, and many other tasks where you need to work with strings (where the data does not have to be text).

The path to understanding regular expressions is quite difficult, but if you stick with it, the result will not disappoint you. Try using the regular expressions given in the article when creating your own web application. This way you will be able to understand how the expressions from the examples given in the article work in reality.

If you have your own examples of useful regular expressions, you can add them as a comment to this article.

See also

• [[b9358]]
• [[b7771]]
• [[b6853]]
• [[b5822]]
• [[b11563]]
• [[b11597]]
• [[b11582]]
• [[b11563]]

See also

• Regular expressions ("regexps," from the English Regular Expressions), optimizing regular expressions
• Regular expressions in php and other languages. Regular expressions: search with inversion
• Regular expressions in mysql RLike REGEXP, operation or, and, escaping special characters
• Regular expressions. in php
• Verbal expressions instead of regular ones invariably simplify coding in PHP and Javascript
• PHP regular expression to detect that there are no Russian letters
• Verbal expressions instead of regular ones invariably simplify coding in PHP and Javascript