Practice
In this article I want to look at several well-known methods for protecting a website's content from automated parsing. Each of them has its own advantages and disadvantages, so the choice should be based on the specific situation. In addition, none of these methods is a silver bullet, and there is a way around almost every one of them, which I will also mention.
The simplest and most common way of detecting scraping attempts on a website is to analyze the frequency and regularity of requests to the server. If requests from a given IP address come too often or there are too many of them, that address gets blocked, and to unblock it the user is usually asked to enter a CAPTCHA.
The most important thing in this protection method is finding the boundary between a natural request frequency/volume and scraping attempts, so as not to block innocent users. This is usually determined by analyzing the behavior of normal site visitors.
An example of this method in use is Google, which monitors the number of requests from a given address and issues an appropriate warning, blocking the IP address and prompting for a CAPTCHA.
There are services (such as distilnetworks.com) that let you automate the process of tracking suspicious activity on your site and even enable user verification via CAPTCHA themselves.
This protection is bypassed by using several proxy servers that hide the parser's real IP address. For example, services like BestProxyAndVPN provide inexpensive proxies, while the SwitchProxy service, though more expensive, is specifically designed for automated parsers and can withstand heavy loads.
In this protection method, access to data is granted only to authorized users. This makes it easier to monitor user behavior and block suspicious accounts regardless of which IP address the client is working from.
An example of this is Facebook, which actively monitors user actions and blocks suspicious ones.
This protection is bypassed by creating (including automatically) a large number of accounts (there are even services that sell ready-made accounts for well-known social networks, for example buyaccs.com and bulkaccounts.com). A significant complication for automated account creation can be the requirement to verify the account by phone, checking its uniqueness (so-called PVA — Phone Verified Account). But, in principle, this too can be bypassed by buying a batch of disposable SIM cards.
This is also a common method of protecting data from parsing. Here the user is asked to enter a CAPTCHA in order to access the site's data. A significant drawback of this method is the inconvenience to the user of having to enter a CAPTCHA. Therefore, this method is best applied in systems where access to data happens via individual, relatively infrequent requests.
An example of using a CAPTCHA to protect against automated request generation is services that check a site's position in search results (for example http://smallseotools.com/keyword-position/).
CAPTCHAs are bypassed using programs and services designed to recognize them. These fall into two main categories: automatic recognition without human involvement (OCR, for example the program GSA Captcha Breaker) and recognition with human help (where people, often somewhere in India, sit and process image-recognition requests online in real time — an example being the Bypass CAPTCHA service). Human-based recognition is usually more effective, but in this case payment is made per CAPTCHA rather than a one-time purchase, as with software.
Here, in the request to the server, the browser sends a special code (or several codes) generated by complex logic written in JavaScript. Often, the code implementing this logic is obfuscated and placed in one or more loaded JavaScript files.
A typical example of using this method of protection against parsing is Facebook.
This is bypassed by using real browsers for parsing (for example, with the help of libraries such as Selenium or Mechanize). But this gives this method an additional advantage: by executing JavaScript, the parser will show up in site traffic analytics (for example, Google Analytics), which allows the webmaster to immediately notice something is wrong.
One of the effective ways to protect against automated parsing is frequently changing the page structure. This can involve not only changing the names of identifiers and classes, but even the hierarchy of elements. This greatly complicates writing a parser, but on the other hand it also complicates the code of the system itself.
On the other hand, these changes can be made manually, roughly once a month (or every few months). This too will substantially spoil life for parsers.
To get around such protection requires building a more flexible and "smart" parser, or, if the changes are made infrequently, simply manually fixing the parser whenever those changes occur.
This makes parsing large amounts of data very slow and therefore impractical. At the same time, the limits need to be chosen based on the needs of a typical user, so as not to reduce the overall usability of the site.
This is bypassed by accessing the site from different IP addresses or accounts (simulating many users).
This method of protecting content makes automated data collection more difficult while preserving visual access to it for regular users. Email addresses and phone numbers are often replaced with images, but some sites even manage to replace random letters in the text with images. Although nothing prevents rendering the entire content of a site as graphics (whether Flash or HTML 5), doing so can significantly hurt its indexability by search engines.
The downside of this method is not only that not all content will be indexed by search engines, but also that it prevents the user from being able to copy the data to the clipboard.
This protection is difficult to bypass; most likely you need to apply automatic or manual image recognition, as with CAPTCHAs.
Comments