Deep Learning

Deep learning (deep machine learning) — a set of machine learning methods (supervised, semi-supervised, unsupervised, and reinforcement learning) based on learning data representations (feature/representation learning) rather than on algorithms specialized for specific tasks. Many deep learning methods had already been known back in the 1980s (and even earlier ), but the results were unimpressive , until advances in the theory of artificial neural networks (pretraining neural networks using a special case of an undirected graphical model known as the restricted Boltzmann machine) and the computing power available by the mid-2000s (including the use of graphics accelerators, field-programmable gate arrays, and various forms of neural processors) made it possible to build complex neural network architectures with performance sufficient to solve a wide range of problems that had previously resisted effective solutions — for example, in computer vision, machine translation, and speech recognition — with quality that in many cases is now comparable to, and in some cases exceeds, human performance.

History

Although the term «deep learning» first appeared in the machine learning research community only in 1986, after the work of Rina Dechter , the first general working algorithm for deep, feed-forward multilayer perceptrons was published in the book by Soviet scientists Alexey Grigorievich Ivakhnenko and Valentin Grigorievich Lapa, «Cybernetic Predicting Devices» .

Other deep architectures, in particular those specializing in pattern recognition, trace their origin to the neocognitron, developed by Kunihiko Fukushima in 1980. In 1989, Yann LeCun succeeded in using the backpropagation algorithm to train deep neural networks for the task of recognizing handwritten ZIP codes . Despite the successful experiment, training the model took three days, which severely limited the applicability of this method. The low training speed was due to many factors, including the vanishing gradient problem caused by the large disparity in the scale of the trainable parameters, which was analyzed in 1991 by Jürgen Schmidhuber and Sepp Hochreiter. Because of these problems, neural networks lost ground to support vector machines during the 1990s.

By 1991, such systems were being used to recognize isolated two-dimensional handwritten digits, while recognition of three-dimensional objects was performed by matching two-dimensional images to a hand-built three-dimensional object model. In 1992, the cresceptron model was created for recognizing three-dimensional objects in cluttered scenes.

In 1994, André de Carvalho, together with Mike Fairhurst and David Bisset, published experimental results for a multilayer Boolean neural network, also known as a weightless neural network, consisting of a three-layer self-organizing feature-extraction neural network module (SOFT), followed by a multilayer classification neural network module (GSN). Each module was trained independently of the other. Each layer within a module extracted features of increasing complexity relative to the previous layer.

In 1995, Brendan Frey demonstrated that it was possible to train (over the course of two days) a network containing six fully connected layers and several hundred hidden units, using the wake-sleep algorithm developed jointly with Peter Dayan and Hinton^[11]. Many factors contribute to the slow speed, including the vanishing gradient problem analyzed in 1991 by Sepp Hochreiter.

Simpler models that rely on hand-crafted, task-specific engineering, such as Gabor filters and support vector machines (SVM), were a popular choice in the 1990s and 2000s because of the computational cost of artificial neural networks (ANN) and the lack of understanding of how the brain wires its biological networks.

Both shallow and deep learning (e.g., recurrent networks) of ANNs have been studied for many years. These methods never outperformed the mixture-of-Gaussians and hidden Markov model, based on generative models of speech trained discriminatively^[17]. Key difficulties were analyzed, including the vanishing gradient^[12] and the weak temporal correlation structure in neural predictive models. Additional difficulties were the lack of training data and limited computational power.

Deep learning gained popularity in the mid-2000s, when everything came together: computers became powerful enough to train large neural networks (computations were delegated to graphics processing units, which sped up training by an order of magnitude), datasets became large enough that training large networks made sense, and another advance occurred in the theory of artificial neural networks — papers by Hinton, Osindero, and Teh^[20], as well as by Bengio^[21], in which the authors showed that a multilayer neural network can be effectively pretrained by training each layer separately using a restricted Boltzmann machine, and then fine-tuning it using the backpropagation algorithm.

The deep learning revolution

In 2012, a team led by George E. Dahl won the «Merck Molecular Activity Challenge» using multi-task deep neural networks to predict the biomolecular target of a single drug.^[22] In 2014, Hochreiter's group used deep learning to identify off-target and toxic effects of chemical substances present in the environment, in nutrients, household products, and medications, and won the «Tox21 Data Challenge» from the U.S. National Institutes of Health, the Food and Drug Administration, and NCATS.^[23]

Conceptual illustration of deep learning as a subset of machine learning, with machine learning as a subset of artificial intelligence (AI)

A significant advance in image or object recognition was felt in the period from 2011 to 2012. Although convolutional neural networks (CNNs) trained by backpropagation had existed for decades, and GPUs had been used to implement neural networks for many years, including CNNs, fast GPU implementations of CNNs were used to advance computer vision. In 2011, this approach for the first time made it possible to achieve superhuman performance in a visual pattern recognition competition. Also in 2011, it won the ICDAR handwriting recognition competition, and in May 2012 — the ISBI image segmentation competition.^[24] Before 2011, CNNs did not play a major role at computer vision conferences, but in June 2012, a paper by Ciresan^[25] at the leading CVPR conference showed how max-pooling CNNs on GPUs could significantly improve many benchmark results. In October 2012, a similar system was developed by Krizhevsky , whose team won the large-scale ImageNet competition by a wide margin over shallow machine learning methods. In November 2012, Ciresan's team also won the ICPR contest on analysis of large medical images for cancer detection, and the following year the MICCAI Grand Challenge on the same topic. ^] In 2013 and 2014, the error rate on the ImageNet task using deep learning was reduced further, following a similar trend in large-scale speech recognition. Stephen Wolfram published these improvements as part of an image identification project.

Image classification was then extended to the more complex task of generating descriptions (captions) for images, often using a combination of a CNN and an LSTM

Some researchers believe that ImageNet's victory in October 2012 marked the beginning of the «deep learning revolution», which transformed the artificial intelligence industry.

In March 2019, Yoshua Bengio, Geoffrey Hinton, and Yann LeCun were awarded the Turing Award for conceptual and engineering breakthroughs that made deep neural networks a critical component of computing.

Neural networks

Artificial neural networks (ANNs) — are computing systems based on the principles of the biological neural networks that make up animal brains. Such systems learn (progressively improve their ability) to perform tasks, generally without being programmed for specific tasks. For example, in recognizing images of cats, they can learn to identify images containing cats by analyzing example images that have been manually labeled as «cat» or «no cat», and using the results of that analysis to identify cats in other images. ANNs have found their greatest application in software applications that are difficult to express with a traditional computer algorithm using rule-based programming.

ANNs are based on a collection of connected units called artificial neurons (an analog of biological neurons in a biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal(s) and then signal the neurons connected to it. Neurons can have a state, usually represented by real numbers, typically between 0 and 1. Neurons and synapses may also have a weight that changes during learning, which can increase or decrease the strength of the signal it sends onward.

Typically, neurons are organized into layers. Different layers may perform different kinds of transformations. Signals travel from the first (input) layer to the last (output) layer, possibly after passing through the layers multiple times.

The original goal of the neural network approach was to solve problems the way the human brain does. Over time, attention shifted to matching specific intellectual abilities, which led to deviations from biology, such as backpropagation, or transmitting information in the reverse direction and tuning the network to reflect that information.

Neural networks are used to solve a variety of tasks, including machine vision, speech recognition, machine translation, social network filtering, video games, and medical diagnosis.

As of 2017, neural networks typically have from a few thousand to a few million units and millions of connections. Although this number is several orders of magnitude smaller than the number of neurons in the human brain, these networks can perform many tasks at a level exceeding that of humans (for example, face recognition, playing «Go»).

Deep neural networks

A deep neural network (DNN — deep neural network) — is an artificial neural network (ANN) with multiple layers between the input and output layers. A DNN finds the correct method of mathematical transformations to turn the incoming data into the output, regardless of linear or nonlinear correlation. The network moves through layers, calculating the probability of each output. For example, a DNN trained to recognize dog breeds will process a given image and compute the probability that the dog in the image belongs to a particular breed. The user can review the results and select the probabilities the network should display (above a certain threshold, for example) and return the suggested label to the network. Each mathematical transformation is considered a layer, and complex DNNs have many layers, hence the name «deep» or «deep-layered» networks.

DNNs can model complex nonlinear relationships. DNN architectures generate compositional models in which an object is expressed as a layered composition of primitives. The additional layers enable composition of features from lower layers, potentially modeling complex data with fewer units than a shallow network with similar performance.

Deep architecture encompasses many variants of a few basic approaches. Each architecture has found success in certain areas. It is not always possible to compare the performance of several architectures unless they have been evaluated on the same datasets.

DNNs are typically feed-forward networks, in which data flows from the input layer to the output layer without feedback. First, the DNN builds a map of virtual neurons and assigns random numerical values, or «weights», to the connections between them. The weights and the input data are multiplied and return an output signal from 0 to 1. If the network fails to accurately recognize a particular pattern, the algorithm will adjust the weighting coefficients^[38]. In this way, the algorithm can make certain parameters more significant until it determines the correct mathematical manipulations to fully process the data.

Definitions

Machine learning algorithms

Deep learning is characterized as a class of machine learning algorithms that:

• use a cascade of multiple layers of nonlinear filters for feature extraction with transformations. Each successive layer takes as input the output of the previous layer. A deep learning system can combine supervised and unsupervised learning algorithms, whereby sample analysis constitutes unsupervised learning, and classification — supervised learning.
• have several layers for extracting features or parameters representing the data (unsupervised learning). Here, features are organized hierarchically, with higher-level features being derived from lower-level features.
• are part of the broader field of machine learning devoted to learning data representations.
• form, during training, layers at multiple levels of representation, corresponding to different levels of abstraction; the layers form a hierarchy of concepts.

All definitions state

1. the presence of multiple layers of nonlinear processing
2. supervised or unsupervised learning of the features of each layer, forming a hierarchy from low to high level^[39].

The specific composition of nonlinear layers depends on the problem being solved. Both hidden layers of a neural network and layers of complex logical transformations are used^[40]. The system may include hidden variables organized layer by layer in deep generative models, such as the nodes in a deep belief network and a deep restricted Boltzmann machine.

Deep learning algorithms are contrasted with shallow learning algorithms by the number of parameterized transformations that a signal encounters as it propagates from the input layer to the output layer, where a parameterized transformation is a data-processing unit that has trainable parameters, such as weights or thresholds^[41]. The chain of transformations from input to output is called the CAP — credit assignment path (CAP). The CAP describes the potential causal connections along the network from input to output, and the path length may differ across different branches. For a feedforward neural network, the depth of the CAP is the same as the depth of the network and equals the number of hidden layers plus one (the output layer is also parameterized). For recurrent neural networks, in which a signal can skip across layers bypassing intermediate ones, the CAP is potentially unlimited in length because of feedback. There is no universally agreed threshold of depth separating shallow learning from deep learning, but deep learning is generally considered to be characterized by several nonlinear layers (CAP > 2). Jürgen Schmidhuber also distinguishes «very deep learning», where CAP > 10^[41].

Areas of application of machine learning

1.Oil and gas

Deep learning makes it possible to determine the best way to optimize operations. For example, it can convert seismic maps into three-dimensional maps to improve the accuracy of reservoir predictions. Deep learning helps analyze huge volumes of data, including pump pressure, temperature, and flow rate.

2.Construction

Many construction companies use deep learning models similar to AlphaGo to model projects and find the optimal sequence for laying pipes and pouring concrete. Deep learning makes it possible to find the fastest and most efficient way to complete projects.

3.Finance

The largest banks use deep learning for electronic discovery. For example, JPMorgan Chase leverages deep learning to ensure compliance with government regulatory requirements and to detect insider trading.

4.Information security

When it comes to detecting malware, malicious URLs, and malicious code, deep learning has no equal.

5.Healthcare

Teams at the University of California, Los Angeles created an intelligent microscope that generates multidimensional datasets to train deep learning systems to detect cancer cells with a detection probability of 99.9%.

Social media

Pinterest uses a visual search tool to find visually similar objects. The company took advantage of deep learning to teach the system to recognize images and then recommend them to users.

Contents

Deep learning — is a set of machine learning algorithms for modeling high-level abstractions using numerous nonlinear transformations.

The following methods and their variations belong primarily to deep learning:

• Certain unsupervised learning systems, such as the restricted Boltzmann machine for pretraining, the autoencoder, the deep belief network, the generative adversarial network,
• Certain supervised learning systems, such as the convolutional neural network, which brought pattern recognition technology to a new level,
• Recurrent neural networks, which allow learning on processes over time,
• Recursive neural networks, which allow feedback to be included between the elements of a circuit and chains.

By combining these methods, complex systems are created that correspond to various artificial intelligence tasks.

Deep learning is a proven sample from a broad family of machine learning methods for data representations best suited to the nature of the task. An image, for example, can be represented in many ways, such as a vector of intensity values per pixel, or (in a more abstract form) as a set of primitives, regions of a certain shape, and so on. Good data representations make it easier to solve specific tasks — for example, recognizing faces and facial expressions^[46]). Deep learning systems automate the very process of selecting and tuning features, carrying out feature learning^[en] in an unsupervised or semi-supervised manner, using efficient algorithms and hierarchical feature extraction for this purpose.

Research in this area has made it possible to improve models for working with large volumes of unlabeled data. Some approaches emerged from advances in neuroscience, successes in interpreting information processing, and building communication models in the nervous system, such as neural coding, related to determining the relationship between a stimulus and neural responses, and the interrelation of electrical activity between neurons in the brain^[.

Deep learning systems have found application in areas such as computer vision, speech recognition, natural language processing, audio recognition, and bioinformatics, where significantly better results than before have been demonstrated for a number of tasks.

Despite the successes of deep learning, it still has a fundamental limitation: deep learning models are limited in what they can represent, and most programs cannot be expressed as a continuous geometric morphing of a data manifold^[49].

Nevertheless, a skeptical view remains that deep learning — is nothing more than a buzzword or a rebranding of neural networks^[50][51].

Overview and comparison of deep learning software