Lecture
Neural networks are computer models inspired by the workings of the brain's neural system. They are networks of interconnected artificial neurons that process and transmit information among themselves. Neural networks are used to solve a variety of tasks, including pattern recognition, data classification, forecasting, natural language processing, and much more.
The structure of a neural network consists of an input layer, hidden layers (if present), and an output layer. Each layer consists of neurons that exchange signals with one another through connections called weights. Each neuron receives input signals, computes their weighted sum, and applies a nonlinear activation function to generate an output signal. This process is repeated through the layers of the neural network until a final output is obtained.
Training a neural network is usually carried out by feeding it training data, during which the network adjusts its weights and parameters so as to minimize the error between predicted and expected results. This process is called backpropagation of error and uses optimization methods such as gradient descent to update the network's weights.
Neural networks are a powerful tool for solving complex information-processing tasks. They demonstrate the ability to learn from large volumes of data and to detect complex patterns and dependencies. This allows them to achieve high accuracy on a variety of tasks and to become a key technology in the fields of artificial intelligence, machine learning, and deep learning.
Diagram of a simple neural network. Input neurons are shown in green, hidden neurons in blue, and the output neuron in yellow
Many people have probably heard the term «neural networks,» associating it with artificial intelligence, androids, robots capable of learning by observing people. Some feel a fear of the unknown, a fear of possible consequences; others become interested, recognizing its potentially useful applications and seeing it as a way to help investigate problems that are currently unsolvable; still others look at what is happening with skepticism, having had an unsuccessful experience designing artificial neural networks (ANNs), or simply lacking the desire to look into it. In any case, over more than half a century of existence, ANNs have experienced no shortage of rises and falls. This article proposes to examine their possible purpose, the problems associated with their implementation, the current state of affairs, and to assess the factors influencing their development.
Historical overview
Interest in artificial neural networks arose from the work of pioneers in the field — W. McCulloch and W. Pitts. In 1943, public attention was drawn to a paper titled «A Logical Calculus of the Ideas Immanent in Nervous Activity,» in which they proposed a mathematical model of the neuron and formulated the principles for constructing artificial neural networks, based on the model of brain function they had developed. Many scientists enthusiastically began proposing their own solutions and new ANN architectures. For example, D. Hebb, in 1949, put forward ideas about the nature of connections between brain neurons and their interaction, presenting the first methods for training ANNs, while the prominent neurophysiologist Frank Rosenblatt proposed, in 1957, his own neural network model — the perceptron — along with the architecture of the world's first neurocomputer model, the Mark. However, his work was subjected to harsh criticism (some believe unjustified) from many respected scientific authorities of the time. Among them were early proponents of ANN theory, Minsky (M. Minsky) and Papert (S. Papert), who demonstrated the shortcomings of Rosenblatt's theory in 1969. Their work cooled the scientific enthusiasm of many researchers for almost two decades. A certain lull set in, associated with disappointment in the capabilities of ANNs, caused not least by the psychological factor manifested in a person's inability to describe in words how they think. In my view, the theory of ANNs developed very quickly — people were creating something that was not yet in demand at the time; in other words, the theory was ahead of its time. And although many noteworthy developments and studies were proposed during this period, the scientific world regarded them with skepticism. This continued until 1982, when the American biophysicist J. Hopfield proposed a neural network architecture (now named after him) that used an energy-based approach modeling associative memory. This gave new impetus to the development of the field, and in 1986 J. Hinton and his colleagues published a paper describing a new neural network model and its training algorithm. At the same time, the first commercial neurocomputer projects appeared (the Mark III, by TRW, USA).
Minsky returned to this research as well, despite his earlier work on the limitations of perceptrons.
Further development was supported by the establishment of specialized neurocomputer centers and by a large number of international conferences and forums. Sales of commercial neural-network products grew more than fourteenfold between 1991 and 1997.
Now, with the shift to nanotechnology and the emergence of new knowledge about the workings of the human brain, fundamentally new architectures and technological solutions have appeared, research directions have been defined, and a wide range of tasks has emerged that even modern PCs cannot solve algorithmically.
The Neuron Model
Let us determine what actually drove the pioneers of the field to construct a model of neural networks. Naturally, it stems from humanity's desire to reproduce the workings of the brain, to make a machine think.
The human cerebral cortex contains about 100 billion neurons, each of which is connected to 1000–10000 other neurons, giving approximately 1014 to 1015 interconnections.
Let me offer a brief overview. It is known that a neuron cannot change its state more than 100 times per second, i.e., at a frequency of about 100 hertz. A person visually perceives surrounding information within 0.7–1 second, recognizing images, identifying people and objects, and one must also account for the processing of information from other sense organs and the control of bodily functions. However, implementing these actions using algorithmic methods would require enormous computational power and databases. For this class of tasks, it turns out that slow neurons work far faster than the most advanced computers, thanks to an unknown principle of resource allocation and distributed data transmission. According to calculations by ITC specialists, modeling the human brain would require a computer with a clock speed of 2000 terahertz and 200 terabytes of RAM — or more than a million average-power computers with an ideal algorithm for distributing computations without any overhead. Below is a table comparing the architecture of the von Neumann machine (the standard PC architecture) and a biological neural network (from the work Artificial Neural Networks, by Anil K. Jain, Jianchang Mao, K. M. Mohiuddin, USA).
The diagram shows a biological neuron. Its mechanism of operation is fairly simple. A neuron receives impulses from the axons of other neurons through dendrites (receivers) and transmits signals, generated by the cell body, along its own axon (transmitter), which branches into fibers at its end. At the tips of these fibers are special structures — synapses — which affect the magnitude of the impulses.
By training an ANN we mean adjusting its coefficients so that it performs a specialized task effectively. Most often a training set (input–output pairs) is provided, and through iterative approximation the ANN adjusts its weights as needed, which improves the network's performance.
This is precisely the advantage of ANN-based systems over systems whose operation follows strictly defined rules. How does the creation and design of an ANN take place? In simplified
form, this process can be described as follows. A model of the process for which the ANN is being created is defined, along with the acting factors and available information. Then, based on certain conditions,
the rules, methods, and modifications of the weighting coefficients are determined — this produces a learning algorithm. Such a network is now capable of learning and of functioning.
It is quite natural to ask: what methods, exactly, are used in training? Let us briefly address this question without delving into mathematical and specialized theories. There are three learning paradigms: «supervised,» «unsupervised,» and mixed.
In the first case, values whose required results are already known are fed to the inputs of the neural network. The input coefficients are modified so that the network's output matches the results known in advance. A modified version of supervised learning assumes that only an assessment of the correctness of the neural network's output is known, not the correct output values themselves. In the second case, the data is structured and distributed into categories based on certain relationships between the inputs. The mixed variant of learning involves training some of the neurons with a teacher, while the rest are trained using the second method.
Typical Tasks for ANNs
In this part of the article we will look at typical tasks in which the results obtained using ANNs are more robust compared to results obtained by traditional methods. We will then review a number of real-world applications and current research.
According to Anil K. Jain of Michigan State University and specialists from the IBM Research Center, Jianchang Mao and K. M. Mohiuddin, the list of tasks can be classified as follows.
Once again, let us note that we have listed only typical tasks; in general, the situations most commonly encountered represent a combination of those defined earlier.
For example, ensuring an aircraft is loaded, minimizing the required amount of fuel, flight time, and distance. In addition to the optimization task, a control task and an approximation task are added (noise suppression while maintaining communication between pilots and control signals).
I propose to dwell in more detail on the solution of a problem faced by a great many people — the recognition of character images, or, more simply, text recognition. Today there are numerous processing programs that make it possible to solve the OCR problem. In solving this problem one has to deal with a whole range of issues: noise suppression, character and block segmentation, and their further processing. It is fine if the document was scanned from a well-printed source, but more often than not a whole range of problems arises. For example, you may need to recognize an old document that cannot be aligned properly, or, say, a handwritten one (even one written in calligraphic handwriting) — programs struggle greatly with such tasks.
Implementing a text-recognition algorithm is fairly complex and is made up of a number of interrelated blocks, including preprocessing, segmentation, feature extraction, classification, and contextual processing. The paper document is scanned and an image is created in grayscale or as a binary (black-and-white) image. At the preprocessing stage, filtering is applied to remove noise, the text region is localized, and it is converted to a binary image using a global and local adaptive threshold converter. At the segmentation step, the text image is divided into individual characters. This task is especially difficult for handwritten text, which contains connections between neighboring characters. One effective technique consists of breaking a composite sample into small samples (intermediate segmentation) and finding the points of correct segmentation using the output of a pattern classifier. Due to varying slant, distortions, noise, and handwriting styles, recognizing segmented characters is not a simple task.
The solution to this problem using ANNs works as follows: implicit feature extraction takes place inside the neural network itself, so that in this case feature extraction and classification are combined, proceeding together with the network's training. In this way, the network can «get used to» the style of the text, making it possible to obtain optimal results. But in any case, a human's own ability to recognize text (with complete independence from the factors mentioned above) is in no way comparable even to the most advanced OCR systems.
However, one should not lose sight of the fact that ANNs are not a panacea for problems of this kind — they too can «make mistakes.» ANNs can only help in solving hard-to-formalize problems, requiring a highly qualified network-architecture specialist capable of handling issues related to design, the use of specialized software, and so on.
Features of Implementing New Technologies
The development of new technologies and the emergence of new types of tasks, including hard-to-formalize ones, requires a number of firms to implement new technologies, use new types of software, and bring in qualified specialists. Naturally, each organization will have its own variation of the update process; however, it will be based (with one modification or another) on the methods described below, each of which has its own advantages and disadvantages.
One approach is to assemble a group of expert specialists. The advantages of this approach include the possibility of direct human interaction with the end «information processor,» and the ability to account for factors that cannot be formalized or presented in a form understandable to software. However, the disadvantages of this method include equally important features, among them the high material costs of an expert, associated with the need for the employee's periodic professional development and a high level of pay for their work. Human subjectivity, disagreements, and conflicts within the group also cannot be counted as advantages, along with the lack of any guarantee of the specialist's continued employment, due to various factors.
Another approach is to order a ready-made system from a specialized firm. In this case, the customer receives a finished product created with the specifics of the company's work in mind (bearing in mind that this applies only at the time of creation), at a relatively low cost of operation. Among the drawbacks are the fairly high cost of developing such products, as well as their limited flexibility for future use, due to the need to bring in outside specialists for modernization and correction of the system. And all of these factors mean disclosing the specifics of the company's work and its trade secrets, since otherwise it is impossible to create an even reasonably adequate system. There is also the possibility that programmers may misunderstand the necessary features or interpret them subjectively, which will affect the final effectiveness of the product.
Now let us consider the following approach, which has practically all the merits of those described above, but its drawbacks include practically all of their downsides as well. Building a system «from scratch» allows you to control the development process, and later its modernization, while maintaining full confidentiality. The drawbacks are obvious — a staff of programmers and ANN development specialists is required (and the human factor is present everywhere), which means paying high salaries, and the product must also be adapted for end users.
The fourth method — organizing a system based on ready-made software products — combines professional, ready-made architectures, training algorithms, typically technical support, and relatively high flexibility due to the large number of available components. When a product is widely used, the occurrence of errors is minimal, and their detection is more effective. However, there is no guarantee of full adaptation to the end user's tasks, and on top of that, preliminary data preparation is required, which is not always acceptable.
As we have already noted, the first three methods are quite costly, which is not always acceptable for large companies where the area being modernized is not a priority, nor for medium and small companies. The fourth method is suitable for companies of any size, provided that the field of application of the off-the-shelf package has been thoroughly studied, though the methods of using it are unique to each individual organization.
Practical applications
Monitoring credit card transactions
This is a case where the ANN's ability to classify is used. For example, a product developed by HNC Software Inc. (now acquired by Fair, Isaac and Com-pany), HNC Falcon, is designed to detect a wide range of fraudulent bank card transactions in real time. Falcon tracks correlations between transactions, identifying suspicious ones. The Falcon system
uses a patented Unified Decision Technology that combines an extensive database of transaction-processing rules, statistical analysis, and a neural network. The Falcon system also includes a special component that allows experienced specialists to add rules to the system's database, enabling it to determine, with a high degree of confidence, potential cases of bank card fraud based on the cardholder's geographic location or postal code. Practical use of this system has shown an improvement in detection quality of 20–60 %, along with a significant reduction in false positives. Falcon is also used to detect fraud in other fields, such as healthcare and telecommunications.
The PRISM family of systems, developed by Nestor, is based on the use of neural networks, expert systems, and statistical methods to detect credit and debit card fraud in real time, as well as to detect other types of fraud in financial or commercial transactions. The neural network used in the PRISM family of systems was trained on more than half a million transactions involving various types of cards. According to some estimates, using the PRISM family of systems can reduce the number of fraud cases by 50 %.
Medical diagnostics
The company «NeuroProject» has created a system for objective hearing diagnostics in infants. The generally accepted method of objective diagnosis involves recording «evoked potentials» (brain responses) to a sound stimulus during the examination, which appear as bursts on an electroencephalogram. For a sufficiently reliable diagnosis of a child's hearing, an experienced audiologist needs to conduct about 2000 tests, which takes about an hour. A neural network can determine the hearing level with the same reliability from just 200 observations within a few minutes, without the involvement of qualified personnel.
Another example of ANN use in medical diagnostic programs is a cardiac diagnostics package developed by RES Informatica together with the Cardiology Research Center in Milan. For diseases such as myocardial ischemia and arterial hypertension, it achieves a diagnostic accuracy of more than 95 %. In addition, one of the promising directions in this research is online diagnosis. The patient fills out a form specifying the necessary parameters, and the system determines the diagnosis based on a training set and an accumulated database. According to some data, this could increase productivity more than 1000-fold, reduce the cost of research by at least 500-fold, and process data from a virtually unlimited number of people, all while increasing diagnostic accuracy.
Pattern recognition
To give an example in this area of ANN application, let us turn to a demonstration product from «NeuroProject» designed for voice control of the built-in Windows calculator. The system reliably recognizes any of 36 words spoken into a microphone by any person. Pronunciation variants from 19 speakers were used to train the ANN.
Voice dialing of a telephone number is another example of speech recognition. Dr. Ortner from the USA trained a neural network to dial numbers, using 2500 examples for each of 28 words, including the digits zero through nine, as well as the names of departments within the company. This achieves a dialing accuracy of over 90 %. He has now created a «Voice Training Center», where recordings of words spoken by various people are made.
Further research plans to organize «voice communication» for deaf-mute people. Thus, the calling party's words will be converted into text (on the called party's phone screen), while a reverse converter will pronounce the text typed by the called party.
Fein-Marquart As-sociates Inc has developed a program for recognizing postal codes with automatic subsequent sorting. The system recognizes both typeset and handwritten digits. According to Joe McDaniel, it provides 98 % recognition accuracy.
Consumer market analysis
Dr. Al Behrens, an employee of Northern Natural Gas in the state of Nebraska, trained a neural network that predicts changes in gas prices for the following month with an average accuracy of 97 %. The monthly price is sometimes tied to price indexing in trade publications (Inside FERC and Natural Gas Week), taking into account the company's recent market activity, seasonal factors, weather, and so on.
Professor Awad Hanna of the University of Wisconsin built a neural network that, based on road surface data for a specific location at different times of the year, indicates the best type of surface for that area. To build this network, he used information obtained from specialists in the field, but the lack of sponsorship support has prevented testing from being carried out.
The well-known Microsoft corporation also uses neural-network-based software products in its marketing policy. Every year Microsoft sends out more than 40 million promotional offers to buy its products to more than 8.5 million registered users. Most of these are aimed at software upgrades or the purchase of additional packages. The purpose of using ANN-based software is obvious — increasing sales volume. According to Jim Minervino, demand increased from 4.9 % to 8.2 %.
And the difference in advertising costs decreased by 35 %. What changed? The database accumulated data on each recipient: what and when they bought, taking into account factors that included registration data — place of work, income, number of employees at the firm, and job position. The system, analyzing the relationships between these factors and the reply letters agreeing to a purchase, subsequently sent letters only to recipients with a similar set of factors.
GoalAssist Corporation built a neural network that predicts how many prizes, and in what ratio, need to be kept in stock when running an advertising campaign. Simply put, someone who sends in three coffee lids receives a certain prize, and someone who sends in more — a different prize. The system, based on various factors, determines with a high degree of confidence (an error of less than 4 %) which incentives will be highly popular, and which — will not.
Sports forecasting
Don Emmons of Detroit built a neural network for predicting the winning horse in races. The ANN, analyzing previous races, finish times, track records, and other factors, estimated the likely winner with 75 % accuracy. Using the results of this ANN, Don Emmons won 17 out of 22 bets. Mr. Derek Anderson (Lakewood, CO) trained a neural network to predict the winner in dog races. He took similar factors into account, resulting in an ANN with 504 inputs, which gives him 94 % accuracy, though only on a third of the tracks.
Estimating product value
For example, estimating the value of a car depends on many factors, such as the make, year of manufacture, engine size, interior «fittings», condition, and so on. Since the nature of the dependency between these factors is unknown, standard analysis methods are ineffective. The Attrasoft company gives an example of estimating a car's value based on 7 key factors. Home value estimation is carried out using 13 factors. A large number of databases on various topics can be downloaded from the company's website for conducting one's own research using their Decision Maker product.
Forecasting service consumption
Let us consider the example of electricity consumption. ZSolutions carried out an order from an energy company to analyze energy consumption data. Based on the data obtained on the amount of energy consumed at a particular time of day, the pricing policy changed, and, accordingly, so did the profit. A similar task can also be considered for Internet access providers.
Scientific research
Forecasting magnetic storms
Dr. Henrik Lundstedt of Lund Observatory, Sweden, trained neural networks to predict the effects of solar flares, such as disturbances in the Earth's magnetic field. As is known, so-called magnetic storms affect not only the condition and health of humans — power plants are also affected, along with disruptions in radio and television broadcasting, failures in geological equipment, satellites, and so on. The neural network, taking into account 37 known influencing factors once every four days and analyzing their changes, is able to determine «space weather» with a high degree of accuracy. A case is described in which the standard forecasting method failed to detect any severe magnetic storms at all, while the ANN-based method, at the same time, accurately predicted two out of three storms.
Determining the sex of mosquitoes
Aubrey Moore of the Maui Agricultural Research Station, University of Hawaii, trained a neural network to determine the sex of mosquitoes in flight. Taking into account statistical data on the average number of wingbeats for each sex, the neural network processes information from a special photosensor and determines a mosquito's sex with an average accuracy of 98 %. Aubrey Moore conducted the tests on 57 insects, while 403 specimens were used in training.
Forecasting heavy rainfall
There is no need to mention the necessity of this process; we will simply note that such information makes it possible to take the necessary safety measures. Computer systems that process information rely on specific data, whereas ANNs — on learning and using values from an earlier time in a specific area, which makes it possible to produce forecasts with an accuracy of up to 85%. The ANN model developed by Tony Hall uses 19 meteorological variables.
Manufacturing
Determining beer quality
Anheuser-Busch uses neural networks to identify organic compounds contained in competitors’ beer, with 96% accuracy. This allows it to maintain the quality of its own product and keep pace with the development of new ones. Anheuser-Busch keeps the methods and details of these operations a closely guarded secret.
Chip manufacturing
A neural network deployed at an Intel plant is able to identify defects in production. Initially, the experimental system was given electrical test data from finished chips along with the corresponding process control variables. The relationship between these two parameters was established through numerical experimentation and CMOS process modeling. A Response Surface Model (RSM) was used to store the results of a sufficient number of numerical experiments. As a result, the neural network was able to reject non-functioning chips with 99.5% accuracy.
Applications in construction
A large number of methods are used when testing concrete quality. One of them is drilling in search of voids that have formed. However, neural networks make it possible to check the entire material and also determine the depth at which a void is located. By sending sound waves and receiving the reflected signal, then processing it with an ANN, specialists at the National Institute of Standards and Technology (NIST) are able to check concrete quality at material thicknesses of up to half a meter.
Patterns can be objects of various natures: text characters, images, sound samples, and so on. During training, the network is presented with various pattern samples along with an indication of which class they belong to. A sample is typically represented as a vector of feature values. The complete set of features must uniquely determine the class to which the sample belongs. If the features are insufficient, the network may associate the same sample with several classes, which is incorrect. Once training is complete, the network can be presented with previously unknown patterns and will return an answer about which class they belong to.
The topology of such a network is characterized by the fact that the number of neurons in the output layer is typically equal to the number of classes to be determined. A correspondence is established between each output of the neural network and the class it represents. When the network is presented with some pattern, one of its outputs should show a sign that the pattern belongs to that class. At the same time, the other outputs should show a sign that the pattern does not belong to those classes. If two or more outputs show a sign of belonging to a class, the network is considered to be «unsure» of its answer.
This task is similar to the classification problem. The situations to be classified have their characteristics fed into the input of the neural network. At the output, a sign of the decision the network has made should appear. In this case, various criteria describing the state of the controlled system are used as input signals.
Clustering refers to partitioning a set of input signals into classes, where neither the number nor the features of the classes are known in advance. After training, such a network is able to determine which class an input signal belongs to. The network can also signal that an input signal does not belong to any of the identified classes — which is a sign of new data absent from the training set. Thus, such a network can detect new, previously unknown classes of signals. The correspondence between the classes identified by the network and the classes that exist in the domain is established by a human. Clustering is performed, for example, by Kohonen neural networks.
Neural networks in the simple Kohonen variant cannot be huge, so they are divided into hyperlayers (hypercolumns) and cores (microcolumns). Compared with the human brain, the ideal number of parallel layers should not exceed 112. These layers, in turn, make up a hyperlayer (hypercolumn), containing from 500 to 2000 microcolumns (cores). Each layer is divided into a set of hypercolumns that pass through these layers. Microcolumns are encoded with digits and units to produce the output result. If needed, excess layers and neurons are removed or added. Ideally, a supercomputer should be used to select the number of neurons and layers. Such a system allows neural networks to be plastic.
A neural network’s ability to forecast follows directly from its ability to generalize and to identify hidden dependencies between input and output data. After training, the network is able to predict the future value of some sequence based on several preceding values and/or some currently existing factors. It should be noted that forecasting is only possible when previous changes actually predetermine future ones to some degree. For example, forecasting stock prices based on last week’s prices may succeed (or may not), whereas forecasting the results of tomorrow’s lottery draw based on data from the last 50 years is almost certain to yield no results.
Neural networks can approximate continuous functions. A generalized approximation theorem has been proven: using linear operations and cascade connection, it is possible to obtain, from an arbitrary nonlinear element, a device that computes any continuous function to a predetermined degree of accuracy. This means that the nonlinear characteristic of a neuron can be arbitrary: from sigmoidal to an arbitrary wave packet or wavelet, sine, or polynomial. The complexity of a particular network may depend on the choice of nonlinear function, but with any nonlinearity the network remains a universal approximator and, with the right choice of structure, can approximate the behavior of any continuous automaton with sufficient accuracy.
The ability of neural networks to identify relationships between different parameters makes it possible to express high-dimensional data more compactly if the data are closely interrelated. The reverse process — reconstructing the original data set from part of the information — is called (auto)associative memory. Associative memory also makes it possible to restore the original signal/image from noisy/corrupted input data. Solving the problem of heteroassociative memory makes it possible to implement content-addressable memory[15
Some of these stages deserve a closer look.
Choosing data for network training and processing it is the most difficult stage in solving the problem. The training data set must satisfy several criteria:
The source data is converted into a form that can be fed to the network's inputs. Each record in the data file is called a training pair or training vector. A training vector contains one value for each input of the network and, depending on the type of training (supervised or unsupervised), one value for each output of the network. Training the network on a «raw» data set typically does not produce quality results. There are a number of ways to improve the network's «perception».
In addition, the representation of both the input and output data itself plays a major role. Suppose a network is trained to recognize letters in images and has a single numeric output — the letter's number in the alphabet. In this case, the network will form the false impression that the letters numbered 1 and 2 are more similar than the letters numbered 1 and 3, which is generally incorrect. To avoid this situation, a network topology with a larger number of outputs is used, where each output has its own meaning. The more outputs a network has, the greater the distance between classes and the harder they are to confuse.
The type of network should be chosen based on the problem statement and the training data available. Supervised training requires an «expert» assessment for each element of the sample. Sometimes obtaining such an assessment for a large data set is simply impossible. In these cases, the natural choice is a network that trains without a teacher (for example, a Kohonen self-organizing map or a Hopfield neural network). When solving other problems (such as time series forecasting), the expert assessment is already contained in the source data and can be extracted during processing. In this case, a multilayer perceptron[clarify] or a Ward network can be used.
After choosing the overall structure, the network's parameters need to be selected experimentally. For perceptron-like networks, this means the number of layers, the number of blocks in the hidden layers (for Ward networks), the presence or absence of bypass connections, and the neurons' transfer functions. When choosing the number of layers and neurons within them, one should proceed from the fact that a network's ability to generalize is greater the larger the total number of connections between neurons. On the other hand, the number of connections is bounded above by the number of records in the training data.
After choosing a specific topology, it is necessary to select the neural network's training parameters. This stage is especially important for networks trained with a teacher. The correct choice of parameters affects not only how quickly the network's responses converge to the correct answers. For example, choosing a low learning rate will increase convergence time, but sometimes helps avoid network paralysis. Increasing the learning momentum can either increase or decrease convergence time, depending on the shape of the error surface. Given this contradictory influence of the parameters, it can be concluded that their values must be chosen experimentally, guided by a criterion for terminating training (for example, minimizing the error or limiting the training time).
During training, the network scans through the training set in a certain order. The order of scanning can be sequential, random, and so on. Some networks trained without a teacher (for example, Hopfield networks) scan the sample only once. Others (for example, Kohonen networks), as well as networks trained with a teacher, scan the sample many times, with one complete pass through the sample called a training epoch. In supervised training, the source data set is divided into two parts — the actual training set and the test data; the principle of division can be arbitrary. The training data is fed to the network for training, while the verification data is used to calculate the network's error (verification data is never used for training the network). Thus, if the error on the verification data decreases, the network is truly performing generalization. If the error on the training data keeps decreasing while the error on the test data increases, this means the network has stopped generalizing and is simply «memorizing» the training data. This phenomenon is called network overtraining, or overfitting. In such cases, training is usually stopped. Other problems can arise during training, such as paralysis or the network falling into a local minimum of the error surface. It is impossible to predict in advance which problem will occur, nor is it possible to give unambiguous recommendations for resolving them.
Everything said above applies only to iterative algorithms for finding neural network solutions. For these, indeed, nothing can be guaranteed and the training of neural networks cannot be fully automated. However, alongside iterative training algorithms, there are also non-iterative algorithms that possess very high stability and allow the training process to be fully automated.
Even in the case of apparently successful training, a network does not always learn exactly what its creator wanted from it. There is a known case where a network was trained to recognize images of tanks from photographs, but it later turned out that all the tanks had been photographed against the same background. As a result, the network «learned» to recognize this type of landscape instead of learning to recognize tanks. Thus, the network «understands» not what was required of it, but whatever is easiest to generalize.
Testing the quality of a neural network's training must be carried out on examples that did not participate in its training. Moreover, the number of test examples must be greater the higher the quality of training. If a neural network's errors have a probability close to one in a billion, then confirming that probability requires a billion test examples. It turns out that testing well-trained neural networks becomes a very difficult task.
, where W — the network's weighting coefficients);
, where W — the network's weighting coefficients).In a number of neural networks, the activation function may depend not only on the connection weighting coefficients 
, but also on the time it takes for the pulse (signal) to travel along the connection channels 
. Therefore, in general form, the activating (transfer) function of the connection 
from element 
to element 
has the form: 
. Then a synchronous network is one in which the transmission time of each connection is equal either to zero or to a fixed constant 
. An asynchronous network is one in which the transmission time for each connection between elements and is its own, but also constant.
All connections are directed strictly from the input neurons to the output neurons. Examples of such networks are the Rosenblatt perceptron, the multilayer perceptron, and Ward networks.
The signal from the output neurons or the hidden layer neurons is partially fed back to the inputs of the input layer neurons (feedback). The Hopfield recurrent network «filters» input data by returning to a stable state, thereby making it possible to solve problems of data compression and building associative memory. A special case of recurrent networks is bidirectional networks. In such networks, connections exist between layers both in the direction from the input layer to the output layer and in reverse. A classic example is the Kosko neural network.
Artificial neural networks have been developed that use radial basis functions as activation functions (also called RBF networks). The general form of a radial basis function is:
, for example, 
where 
— the neuron's input signal vector, 
— the function's window width, 
— a decreasing function (most often equal to zero outside a certain interval).
A radial basis network is characterized by three features:
Such networks are a competitive neural network with unsupervised training that performs the task of visualization and clustering. It is a method of projecting a multidimensional space into a space of lower dimensionality (most often two-dimensional), and is also used for solving problems of modeling, forecasting, and so on. It is one of the versions of Kohonen neural networks. Kohonen self-organizing maps serve primarily for visualization and initial («exploratory») data analysis.
The signal enters a Kohonen network at all neurons simultaneously; the weights of the corresponding synapses are interpreted as the coordinates of a node's position, and the output signal is formed according to the «winner-take-all» principle — that is, the neuron nearest (in terms of synapse weights) to the object presented at the input has a nonzero output signal. During training, the synapse weights are adjusted so that the lattice nodes «position themselves» at locations of local data density, that is, they describe the cluster structure of the data cloud; on the other hand, the connections between neurons correspond to neighborhood relationships between the corresponding clusters in the feature space.
It is convenient to think of such maps as two-dimensional grids of nodes placed in a multidimensional space. Initially, a self-organizing map is a grid of nodes connected to one another by links. Kohonen considered two variants of connecting nodes — into a rectangular and a hexagonal grid — the difference being that in a rectangular grid each node is connected to 4 neighbors, while in a hexagonal grid it is connected to the six nearest nodes. For these two grids, the process of building a Kohonen network differs only in the way the neighbors nearest to a given node are enumerated.
The initial embedding of the grid into the data space is chosen arbitrarily. The original SOM_PAK package offers options for a random initial placement of nodes in space and an option for placing nodes in a plane. After this, the nodes begin to move through space according to the following algorithm:
is chosen at random. is determined (BMU — Best Matching Unit).. However, it does not move alone, but drags along a certain number of nearby nodes from some neighborhood on the map. Of all the moving nodes, the central one — the one nearest to the data point — shifts the most, while the others undergo smaller shifts the farther they are from the BMU. Two stages are distinguished in tuning the map — a coarse (ordering) stage and a fine-tuning (fine-tuning) stage. In the first stage, large neighborhood values are chosen and the movement of nodes is collective in nature — as a result, the map «unfolds» and roughly reflects the structure of the data; in the fine-tuning stage, the neighborhood radius equals 1-2 and individual node positions are already being adjusted. In addition, the magnitude of the displacement decays uniformly over time, that is, it is large at the beginning of each training stage and close to zero at the end.Computing systems based on artificial neural networks possess a number of qualities absent in machines with von Neumann architecture (but present in the human brain):
Input data — a stock's price over a year. The task — to determine tomorrow's price. The following transformation is carried out — the prices for today, yesterday, and the day before yesterday are arranged in a row. The next row is shifted by one day in date, and so on. A network with 3 inputs and one output is trained on the resulting data set — that is, the output: the price on a given date; the inputs: the price on that date minus 1 day, minus 2 days, minus 3 days. We feed the trained network today's, yesterday's, and the day before yesterday's price and get the answer for tomorrow. It is easy to see that in this case the network simply derives the dependence of one parameter on the three preceding ones. If it is desirable to also take into account some other parameter (for example, an overall industry index), it should be added as an input (and included in the examples), the network retrained, and new results obtained. For the most accurate training, it is worth using the backpropagation method, as the most predictable and easiest to implement.
A series of works by M. G. Dorrer and co-authors is devoted to studying the possibility of developing psychological intuition in neural-network expert systems. The results obtained provide an approach to uncovering the mechanism of intuition in neural networks as it manifests itself when they solve psychodiagnostic problems. A non-standard intuitive approach to psychodiagnostics has been created for computer-based methods, one that consists in dispensing with the construction of a described reality. It makes it possible to shorten and simplify work on psychodiagnostic methods.
Neural networks are widely used in chemical and biochemical research. Neural networks are currently one of the most common methods in cheminformatics for finding quantitative structure-property relationships, which is why they are actively used both for predicting the physicochemical properties and biological activity of chemical compounds, and for the targeted design of chemical compounds and materials with predetermined properties, including in the development of new drugs.
Neural networks are successfully used for synthesizing control systems for dynamic objects. Neural networks possess a number of unique properties that make them a powerful tool for building control systems: the ability to learn from examples and generalize data, the ability to adapt to changes in the properties of the controlled object and the external environment, suitability for synthesizing nonlinear controllers, and high resistance to damage to their elements due to the parallelism inherently built into the neural network architecture.
Artificial neural network algorithms have found wide application in economics. Neural networks are used to solve the problem of developing algorithms for finding an analytical description of the patterns governing the functioning of economic entities (an enterprise, an industry, a region). These algorithms are applied to forecasting certain «output» indicators of these entities. The use of neural network methods makes it possible to solve certain problems of economic-statistical modeling, to increase the adequacy of mathematical models, and to bring them closer to economic reality. Since economic, financial, and social systems are very complex and are the result of the actions and counteractions of various people, it is very difficult (if not impossible) to build a complete mathematical model that accounts for all possible actions and counteractions. In systems of this kind of complexity, it is natural and most effective to use models that directly imitate the behavior of society and the economy. And that is exactly what the methodology of neural networks is able to offer[
In conclusion, it should be noted that the article certainly does not present a complete list of examples of neural network use. We have tried to show how broad the areas of application and the problems being solved are. However, in my view, widespread adoption of ANNs in Russia is still fairly far off, which is due, on one hand, to the cost of such products, and on the other hand, to the skeptical attitude of potential end users.
Comments