Lecture
Machine Learning — a subfield of artificial intelligence that studies methods for building algorithms capable of learning. Two types of learning are distinguished. Learning from examples, or inductive learning, is based on identifying general patterns from specific empirical data.
The search for the Master Algorithm is difficult, but it is enlivened by the rivalry between different scientific schools working in the field of machine learning. The most important of them are the symbolists, connectionists, evolutionaries, Bayesians, and analogizers.
Symbolism - search for logical patterns: Decision Tree, Rule Induction, Association Rules
For symbolists, intelligence comes down to manipulating symbols — this is how mathematicians solve equations, replacing some expressions with others
Connectionism - artificial neural networks: BackPropagation, Deep Belief Nets, Deep Learning
For connectionists, learning — is what the brain does, and so they believe that this organ needs to be reproduced through reverse engineering
Evolutionism (genetic algorithms ) Genetic Algorithms, Genetic Programming
Evolutionists believe that the mother of learning — is natural selection
Bayesianism( estimation of posterior distributions ) Naive Bayes, Bayesian Networks, Graphical Models
Bayesians are concerned above all with uncertainty
Analogism (continuity and compactness hypothesis)kNN, RBF, SVM, Kernel Regression, Kernel Density Estimation
For analogizers, the key to learning — is finding similarities between different situations and thereby logically inferring other similarities
Compositionism Voting, Boosting, Bagging, Stacking, RF, MatrixNet, CatBoost
Thanks to machine learning, a programmer does not have to write instructions that account for every possible problem and contain every solution. Instead, the computer (or a separate program) is given an algorithm for independently finding solutions through the comprehensive use of statistical data, from which patterns are derived and on the basis of which predictions are made.
Machine learning technology based on data analysis dates back to 1950, when the first programs for playing checkers began to be developed. Over the decades that have passed, the general principle has not changed. However, thanks to the explosive growth in computer processing power, the patterns and predictions they generate have become far more sophisticated, and the range of problems and tasks solved using machine learning has expanded.
To start the machine learning process, you first need to load a dataset (a certain amount of source data) into the computer, on which the algorithm will learn to process queries. For example, these could be photos of dogs and cats that already have labels indicating which animal they belong to. After the training process, the program will be able to recognize dogs and cats in new images on its own, without label information. The learning process continues even after predictions have been produced: the more data we have processed with the program, the more accurately it recognizes the required images.
Thanks to machine learning, computers are learning to recognize not only faces in photos and pictures, but also landscapes, objects, text and numbers. As for text, machine learning is indispensable here too: a grammar-checking function is now present in every text editor and even in phones. Moreover, not only the spelling of words is taken into account, but also context, shades of meaning and other subtle linguistic aspects. Furthermore, software already exists that can write news articles (on topics such as economics or, for example, sports) without human involvement.
All tasks solved using ML fall into one of the following categories.
1) Regression task – a prediction based on a sample of objects with various features. The output should be a real number (2, 35, 76.454, etc.), for example the price of an apartment, the value of a security after six months, a store's expected income for the next month, or the quality of a wine in a blind tasting.
2) Classification task – obtaining a categorical answer based on a set of features. It has a finite number of possible answers (usually in the format «yes» or «no»): is there a cat in the photo, is the image a human face, does the patient have cancer.
3) Clustering task – grouping data: dividing all of a mobile operator's customers by solvency level, assigning cosmic objects to one category or another (planet, star, black hole, etc.).
4) Dimensionality reduction task – reducing a large number of features to a smaller number (usually 2–3) for convenient subsequent visualization (for example, data compression).
5) Anomaly detection task – separating anomalies from standard cases. At first glance it coincides with a classification task, but there is one essential difference: anomalies – are a rare phenomenon, and training examples on which a machine learning model could be trained to detect such objects are either vanishingly few or simply nonexistent, which is why classification methods do not work here. In practice, an example of such a task is detecting fraudulent activity with bank cards.
How do you determine which task is needed?
Figure: Types of machine learning
The bulk of tasks solved using machine learning methods belong to two different types: supervised learning or unsupervised learning. However, this teacher is not necessarily the programmer standing over the computer and monitoring every action in the program. «Teacher» in machine learning terms – is human intervention itself in the process of processing information. In both types of learning, the machine is provided with source data that it must analyze and find patterns in. The only difference is that in supervised learning there are a number of hypotheses that need to be refuted or confirmed. This difference is easy to understand with examples.
Supervised machine learning
Suppose we have data on ten thousand apartments in Moscow: area, floor, district, presence or absence of parking near the building, distance from the metro, apartment price, etc. We need to build a model that predicts the market value of an apartment based on its parameters. This is an ideal example of supervised machine learning: we have source data (the number of apartments and their properties, which are called features) and a ready-made answer for each apartment – its price. The program has to solve a regression task.
Another example from practice: confirming or ruling out the presence of cancer in a patient, knowing all their medical indicators. Determining whether an incoming email is spam by analyzing its text. These are all classification tasks.
Unsupervised machine learning
In the case of unsupervised learning, when the system is not provided with ready-made «correct answers», things get even more interesting. For example, we have information about the weight and height of a certain number of people, and this data needs to be divided into three groups, for each of which shirts of suitable sizes will be sewn. This is a clustering task. In this case, all the data must be divided into 3 clusters (although, as a rule, there is no such strict and uniquely possible division).
If we take another situation, where each of the objects in the sample has a hundred different features, the main difficulty will be graphically displaying such a sample. Therefore, the number of features is reduced to two or three, making it possible to visualize them on a plane or in 3D. This – is a dimensionality reduction task.
1. Decision tree
This is a decision support method based on using a tree-like graph: a decision-making model that takes into account their potential consequences (with calculation of the probability of one or another event occurring), efficiency, and resource intensity.
For business processes, this tree consists of a minimal number of questions that assume an unambiguous answer — «yes» or «no». By sequentially answering all these questions, we arrive at the correct choice. The methodological advantage of a decision tree – is that it structures and systematizes the problem, and the final decision is made on the basis of logical conclusions.
Let's take another example of useful classification. Say you take out a loan from a bank. How can the bank make sure you'll pay it back or not? There's really no way to know for certain, but the bank has thousands of profiles of other people who have already taken out loans before you. These include their age, education, job title, salary level, and most importantly — who among them repaid the loan and who had problems.
Yes, everyone has guessed where the data is here and what result needs to be predicted. We train the machine, find the patterns, get the answer — that's not the question. The problem is that the bank cannot blindly trust the machine's answer without explanations. What if there's a glitch, malicious hackers, or a drunk admin decided to fix a little script.
For this task, Decision Trees were invented. The machine automatically splits all the data by questions whose answers are «yes» or «no». The questions may not be entirely reasonable from a human point of view, for example «is the borrower's salary greater than $2,300?», but the machine devises them so that at each step the split is as accurate as possible.
This is how a tree of questions is formed. The higher the level, the more general the question. Afterward, you can even hand them over to analysts, and they'll come up with reasons why it's that way.
Trees have found their niche in areas of high responsibility: diagnostics, medicine, finance.
2. Naive Bayes classification
Naive Bayes classifiers belong to the family of simple probabilistic classifiers and originate from Bayes' theorem, which, as applied to this case, treats features as independent (this is called a strict, or naive, assumption). In practice, it is used in the following areas of machine learning:
3. Least squares method
Anyone who has studied statistics at least a little is familiar with the concept of linear regression. Least squares is one of the ways to implement it. Linear regression is usually used to solve the problem of fitting a straight line that passes through a set of points. Here is how it's done using the least squares method: draw a line, measure the distance from it to each of the points (the points and the line are connected by vertical segments), and carry the resulting sum upward. As a result, the line for which the sum of distances is the smallest is the one we're looking for (this line will pass through points with a normally distributed deviation from the true value).
A linear function is usually used when fitting data for machine learning, and the least squares method – is used to minimize errors by creating an error metric.
4. K-Means method
The only problem is what to do with colors like Cyan — is it closer to green or blue? This is where the popular clustering algorithm — the K-Means method helps us. We randomly scatter our 32 points on the color palette, calling them centroids. We assign all other points to the nearest centroid — forming what looks like constellations of the closest colors. Then we move each centroid to the center of its constellation and repeat until the centroids stop moving. The clusters are found, stable, and there are exactly 32 of them, as required.
5. Logistic regression
Logistic regression – is a way of determining the relationship between variables, one of which is categorically dependent while the others are independent. A logistic function (a cumulative logistic distribution) is used for this. The practical significance of logistic regression is that it is a powerful statistical method for predicting events that involves one or more independent variables. It is in demand in the following situations:
5. Support vector machine (SVM)
This is a whole set of algorithms needed to solve classification and regression analysis tasks. Based on the fact that an object located in N-dimensional space belongs to one of two classes, the support vector machine builds a hyperplane with dimensionality (N – 1) so that all objects end up in one of two groups. On paper this can be depicted as follows: there are points of two different kinds, and they can be linearly separated. In addition to separating the points, this method generates the hyperplane in such a way that it is maximally distant from the closest point of each group.
SVM and its modifications help solve such complex machine learning tasks as DNA splicing, determining a person's sex from a photograph, and displaying advertising banners on websites.
6. Ensemble method
It is based on machine learning algorithms that generate a multitude of classifiers and split all objects from newly arriving data based on their averaging or voting results.
It turns out that if you take several not very effective learning methods and train them to correct each other's mistakes, the quality of such a system will be significantly higher than that of each of the methods individually.
Moreover, it's even better when the algorithms used are as unstable as possible and fluctuate strongly with the input data. That's why Regression and Decision Trees are used more often, since a single strong anomaly in the data is enough to throw off the whole model. But Bayes and K-NN are never used — although they are dumb, they are very stable.
Initially, the ensemble method was a special case of Bayesian averaging, but later it became more complex and acquired additional algorithms:
Ensemble methods are a more powerful tool compared to standalone predictive models, because:
7. Clustering algorithms
Clustering consists in distributing a set of objects into categories so that each category – cluster – ends up containing elements that are most similar to each other.
Objects can be clustered using different algorithms. The following are most often used:
Clustering algorithms are used in biology (studying the interaction of genes in a genome containing up to several thousand elements), sociology (processing the results of sociological research using Ward's method, which produces clusters with minimal variance and roughly equal size), and information technology.
8. Principal Component Analysis (PCA)
Principal Component Analysis, or PCA, is a statistical operation involving an orthogonal transformation, whose purpose is to convert observations of variables that may be somehow interrelated into a set of principal components – values that are linearly uncorrelated.
Practical tasks in which PCA is applied include visualization and most procedures for compressing, simplifying, and minimizing data in order to make the training process easier. However, the principal component method is not suitable for situations where the source data is poorly structured (that is, all components of the method are characterized by high variance). So its applicability is determined by how well the domain has been studied and described.
9. Singular value decomposition
In linear algebra, singular value decomposition, or SVD, is defined as the decomposition of a rectangular matrix consisting of complex or real numbers. Thus, a matrix M of dimension [m*n] can be decomposed such that M = UΣV, where U and V are unitary matrices, and Σ is diagonal.
One special case of singular value decomposition is the principal component method. The very first computer vision technologies were developed based on SVD and PCA and worked as follows: first, faces (or other patterns to be found) were represented as a sum of basis components, then their dimensionality was reduced, after which they were matched against images from the sample. Modern singular value decomposition algorithms in machine learning are, of course, significantly more complex and sophisticated than their predecessors, but their essence has largely remained unchanged.
10. Independent Component Analysis (ICA)
This is one of the statistical methods that reveals hidden factors influencing random variables, signals, etc. ICA forms a generative model for bases of multifactor data. The variables in the model contain some hidden variables, and there is no information at all about the rules by which they are mixed. These hidden variables are the independent components of the sample and are considered to be non-Gaussian signals.
Unlike principal component analysis, to which this method is related, independent component analysis is more effective, especially in cases where classical approaches turn out to be powerless. It uncovers the hidden causes of phenomena, and thanks to this it has found wide application in a great variety of fields – from astronomy and medicine to speech recognition, automatic testing, and the analysis of financial indicator dynamics.
The goal of machine learning is the partial or complete automation of solving complex professional tasks across the most diverse areas of human activity.
Machine learning has a wide range of applications
The scope of machine learning applications is constantly expanding. The pervasive spread of information technology leads to the accumulation of huge volumes of data in science, manufacturing, business, transportation, and healthcare. The resulting forecasting, control, and decision-making tasks often come down to learning from examples. Previously, when such data did not exist, these tasks either were not posed at all, or were solved using completely different methods.
Example 1. Disease diagnosis
In this case, patients are the objects, and the features are all the symptoms observed in them, their medical history, test results, treatment measures already taken (essentially the entire medical history, formalized and broken down into individual criteria). Some features – sex, the presence or absence of headache, cough, rash, and others – are treated as binary. The assessment of condition severity (extremely severe, moderate, etc.) is an ordinal feature, while many others are quantitative: the volume of a medication, the level of hemoglobin in the blood, blood pressure and pulse readings, age, weight. Having collected information about the patient's condition containing many such features, it can be loaded into a computer and, using a program capable of machine learning, the following tasks can be solved:
No doctor is able to instantly process the entire volume of information for each patient, generalize a large number of other similar medical histories, and immediately produce a clear result. That is why machine learning is becoming an indispensable assistant for doctors.
Example 2. Finding mineral deposit locations
Here, the features are data obtained through geological exploration: the presence of certain rock types in the area (which would be a binary-type feature), and their physical and chemical properties (which break down into a number of quantitative and qualitative features).
For the training set, 2 types of examples are taken: areas where mineral deposits are definitely present, and areas with similar characteristics where these minerals were not found. But the extraction of rare minerals has its own specifics: in many cases the number of features significantly exceeds the number of objects, and traditional statistical methods are poorly suited to such situations. That's why machine learning focuses on discovering patterns in the already-collected data. To do this, small and highly informative sets of features are identified that are as indicative as possible for answering the research question – whether a given mineral is present in the specified area or not. An analogy can be drawn with medicine: deposits, too, can have their own syndromes identified. The value of applying machine learning in this field lies in the fact that the results obtained are not only practical in nature, but also of serious scientific interest to geologists and geophysicists.
Example 3. Assessing the reliability and creditworthiness of loan applicants
Every bank that issues loans faces this task daily. The need to automate this process has been apparent for a long time, back in the 1960–1970s, when the credit card boom began in the USA and other countries.
People applying to a bank for a loan are the objects, and the features will differ depending on whether it is an individual or a legal entity. The feature description of a private individual applying for a loan is formed based on the data from the application form they fill out. The form is then supplemented with some other information about the potential client, which the bank obtains through its own channels. Some of these are binary features (sex, having a phone number), others are ordinal (education, job position), while most are quantitative (loan amount, total outstanding debts to other banks, age, number of family members, income, length of employment) or nominal (name, employer company name, occupation, address).
For machine learning, a sample is compiled that includes borrowers whose credit history is known. All borrowers are divided into classes; in the simplest case there are 2 – «good» borrowers and «bad» ones, and a positive decision to grant a loan is made only in favor of the «good» ones.
A more complex machine learning algorithm, called credit scoring, involves assigning each borrower a certain number of conditional points for each feature, and the decision on granting the loan will depend on the total number of points earned. During the machine learning process, credit scoring systems first assign a certain number of points to each feature, and then determine the loan terms (duration, interest rate, and other parameters reflected in the loan agreement). But there is also another algorithm for training the system – based on precedents.
Tests
1.1 The test set is characterized by its unavailability during algorithm training(+)
1.2 For correct testing of the algorithm, the test set must be contained within the training set
1.3 A laptop's battery life is a categorical feature for a machine learning task
1.4 The estimate closest to the truth of a model's performance is its average quality score on the training and test sets
Choose all suitable answers from the list
2.1 Finding “toxic” comments on a social network
2.2 Predicting a bank client's salary(+)
2.3 Finding fraudulent transactions
2.4 A spam filtering algorithm
2.5 Predicting monthly precipitation amount(+)
Choose one option from the list
3.1 model_selection(+)
3.2 datasets
3.3 neighborspre
3.4 processing
Choose one option from the list
4.1 Higher on the test set
4.2 The same
4.3 Higher on the training set(+)
4.4 Impossible to determine
In general, in classification you have a set of predefined classes and you want to know which class a new object belongs to.
Clustering tries to group a set of objects and determine whether some relationship exists between the objects.
In the context of machine learning, classification is supervised learning and clustering is unsupervised learning.
Also see the definitions of Classification and Clustering.
+ Classification: you are given new data, and you must assign a new label to it.
For example, a company wants to classify its potential customers. When a new customer comes in, they need to determine whether this is a customer who is going to buy their product or not.
«When a new customer comes in, they need to determine whether this is a customer who is going to buy their product or not». is the best candidate for logistic regression. An example of classification could be predicting whether a buyer is going to buy the «premium», «standard», or «economy» model. An airline example: coach, coach with early boarding, coach with extra legroom. I would say that «When a new customer comes in, they need to determine whether this is a customer who is going to buy their product or not». is the best candidate for logistic regression. An example of classification could be predicting whether a buyer is going to buy the «premium», «standard», or «economy» model. An airline example: coach, coach with early boarding, coach with extra legroom.
+ Clustering: you are given a set of historical transactions recording who bought what.
Using clustering methods, you can describe the segmentation of your customers.
Comments