Lecture
Modern Data Mining technology is based on the concept of patterns that reflect fragments of multidimensional relationships in data.
Definition
Patterns are stable, recurring combinations of data that reflect regularities inherent in data subsamples, which can be compactly expressed in a form understandable to humans.
The search for patterns is carried out using methods that are not limited by a priori assumptions about the structure of the sample and the type of distribution of the values of the indicators being analyzed. The patterns found may reflect non-obvious, unexpected regularities in the data, revealing so-called hidden knowledge. Just as with the extraction of mineral resources, «raw» data may contain a deep layer of knowledge, the skillful excavation of which can uncover real nuggets that provide tangible competitive advantages. Data Mining methods were of interest first of all to commercial enterprises deploying projects based on data warehouses. Their experience shows that the profit from using Data Mining can reach 1000%.
Examples of search tasks for solving the same problem using Data Mining technology and OLAP technology are given in Table 4.5'.
Comparison of OLAP and Data Mining technologies
Table 4.5
|
OLAP |
Data Mining |
|
What are the average injury rates for smokers and non-smokers? |
Are there precise patterns in the description of people prone to increased injury rates? |
|
What are the average sizes of the phone bills of existing customers compared with the bills of former customers who have discontinued the phone company's services? |
Are there characteristic profiles of customers who are, in all likelihood, about to discontinue the phone company's services? |
|
What is the average amount of daily purchases on a stolen credit card versus a non-stolen one? |
Are there stereotypical purchasing patterns for cases of credit card fraud? |
Overall, Data Mining technology is quite accurately defined by one of the founders of this field, G. Piatetsky-Shapiro .
Definition
Data Mining — is the process of discovering in raw data previously unknown, non-trivial, practically useful, and interpretable knowledge necessary for decision-making in various spheres of human activity.
Data Mining is an essentially multidisciplinary field that arose and is developing on the basis of the achievements of applied statistics, pattern recognition, artificial intelligence methods, database theory, and others. Hence the abundance of methods and algorithms implemented in various operational Data Mining , systems, many of which integrate several approaches at once. Nevertheless, every system has a certain key component that solves the specific task at hand.
Note!
There are five standard types of regularities identified using Data Mining methods: association, sequence, classification, clustering, and forecasting.
Association occurs when several events are related to one another. For example, a study conducted in a computer supermarket may show that 55% of those who buy a computer also buy a printer, and when a discount is offered for such a bundle, the printer is purchased in 80% of cases.
A chain of events linked in time forms a sequence. Thus, for example, after the purchase of an apartment a new kitchen stove is bought within a month in 45% of cases, and within two weeks 60% of new homeowners acquire a refrigerator.
With the help of classification, the features characterizing the class to which a given object belongs are identified. This is done by analyzing objects that have already been classified and formulating a certain set of rules.
Clustering differs from classification in that the classes themselves are not specified in advance; Data Mining tools independently identify various homogeneous groups of data.
The basis for all kinds of forecasting systems is historical information stored in the database in the form of time series. If it is possible to find patterns that adequately reflect the dynamics of the behavior of the target indicators, they can be used to predict the behavior of the system in the future.
Comments