Analog and Physical Neural Networks

Physical (analog) neural network is a type of artificial neural network in which an electrically adjustable material is used to emulate the function of a neural synapse. The term «physical» neural network is used to emphasize the dependence on the physical hardware used to simulate neurons, as opposed to software approaches that simulate neural networks. More generally, the term also applies to other artificial neural networks in which a memristor or another electrically adjustable resistive material is used to emulate a neural synapse.

Classification of neural networks by type of input information

• Analog neural networks (use information in the form of real numbers);
• Binary neural networks (operate on information represented in binary form);
• Image-based neural networks (operate on information represented in the form of images: signs, hieroglyphs, symbols).

Let us consider analog neural networks.

Two kinds of signals are suitable for exchanging data in a neural network: analog and digital. Using analog signals, neurons exchange gradient data, while using digital signals they exchange logical data. By gradient data we mean integer numerical values lying within a given range, for example, from -127 to 127 or from 0 to 255. One boundary of this range corresponds to full confirmation, the other — to full negation. It is convenient to take the minimum value as false and the maximum as true. While the neural network is running, neurons receive such data, process it, and then pass it further along the axon. All neurons operate in parallel. The result produced by a neuron depends on the operation it performs, as well as on whether it is synchronized or unsynchronized. Unsynchronized neurons continuously output the result of the operation being continuously performed on the input gradient data. Synchronized neurons output the result of the operation performed on the input gradient data at the moment the synchronizing signal arrives. Thus, an analog neural network is a parallel analog computer with a dynamically changing structure.

Although analog neural networks are a “superset” of digital networks, in many practical cases their use is not justified. This is because a sufficiently developed component base for organizing widely parallelized analog computing processes does not yet exist today. In most cases, neural networks have to be implemented using modern digital computing technology. Moreover, there is a huge number of problems for whose solution two logical values, True and False, are more than sufficient. In digital neural networks, the logical values “0” and “1” circulate through the dendrites and axons. While the neural network is running, neurons receive such data, process it, and then pass it further along the axon. All neurons operate in parallel. The result produced by a neuron depends on the operation it performs, as well as on whether it is synchronized or unsynchronized. Unsynchronized neurons continuously output the result of the operation being continuously performed on the input logical data. Synchronized neurons output the result of the operation performed on the input logical data at the moment the synchronizing signal arrives. A digital neural network reduces to an analog neural network with the range of processed gradient data from 0 to 1. Thus, a digital neural network is a parallel digital computer with a dynamically changing structure.

Types of physical neural networks

ADALINE

In the 1960s, Bernard Widrow and Ted Hoff developed ADALINE (adaptive linear neuron), which used electrochemical cells called memistors (memory resistors) to simulate the synapses of an artificial neuron. Memistors were implemented as 3-terminal devices operating on the basis of reversible electroplating of copper, so that the resistance between two leads is regulated by the integral of the current supplied through the third lead. The ADALINE circuit was briefly commercialized by the Memistor Corporation in the 1960s, enabling some pattern recognition applications. However, because memistors were not manufactured using integrated circuit fabrication technologies, the technology was not scalable and was eventually abandoned as solid-state electronics matured.

Analog VLSI

In 1989, Carver Mead published his book Analog VLSI and Neural Systems, which spun off perhaps the most common variant of analog neural networks. The physical implementation is realized in analog VLSI. This is often implemented as field-effect transistors operating in weak inversion. Such devices can be modeled as translinear circuits. This is a method described by Barry Gilbert in several papers around the mid-1970s, and in particular in his 1981 Translinear Circuits. Using this method, circuits can be analyzed as a set of well-defined steady-state functions, and such circuits are assembled into complex networks.

Physical neural network

Alex Nugent describes a physical neural network as one or more nonlinear neuron-like nodes used to sum signals, and nanoconnections formed from nanoparticles, nanowires, or nanotubes, which determine the strength of the signal fed into the nodes. The alignment or self-assembly of the nanoconnections is determined by the history of the applied electric field, performing a function analogous to neural synapses. Numerous applications are possible for such physical neural networks. For example, a temporal summation device may consist of one or more nanoconnections having an input and an output, whereby an input signal applied to the input causes one or more nanoconnections to experience an increase in connection strength over time. Another example of a physical neural network is described in U.S. Patent No. 7,039,619, entitled «Nanotechnology neural network device utilizing solution and disconnection of connections», which was issued to Alex Nugent by the United States Patent and Trademark Office on May 2, 2006. [10]

A further application of a physical neural network is shown in U.S. Patent No. 7,412,428, entitled «Application of Hebbian and anti-Hebbian learning to nanotechnology-based physical neural networks», which was issued on August 12, 2008. [11]

Nugent and Molter showed that universal computation and universal machine learning are possible using the operations available through simple memristive circuits operating under the AHaH plasticity rule. [12] More recently, it has been argued that complex networks of purely memristive circuits can serve as neural networks. [13] [14]

Phase-change neural network

In 2002, Stanford Ovshinsky described an analog neural computing medium in which a phase-change material has the ability to respond cumulatively to multiple input signals. [15] The electrical change in resistance of the phase-change material is used to control the weighting of the input signals.

Memristive neural network

Greg Snyder of HP Labs describes a system of cortical computing with memristive nanodevices. [16] Memristors (memory resistors) are implemented using thin-film materials in which the resistance is electrically tuned via the transport of ions or oxygen vacancies within the film. DARPA«s SyNAPSE project funded IBM Research and HP Labs in collaboration with Boston University's Department of Cognitive and Neural Systems (CNS) to develop neuromorphic architectures that may be based on memristive systems [17].

Optical neural network

A trained multilayer phase mask (handwritten character classifier). On the right is a physical model of a D²NN optical neural network, printed on a 3D printer: layers 8×8 cm spaced 3 cm apart

A team of researchers from the University of California, Los Angeles has developed a new type of neural network that uses light instead of electricity to operate. Science published an article describing the idea, the working device, its performance, and the types of applications that, in the authors' opinion, this new type of neural network is well suited to computing.

The researchers explain that the structure of the optical neural network is organized according to Huygens' principle, according to which each element of a wavefront can be regarded as the center of a secondary disturbance generating secondary spherical waves, and the resulting light field at each point in space is determined by the interference of these waves. Thus, an artificial neuron in a D²NN is connected to other neurons of the following layer via a secondary wave, which is modulated in amplitude and phase both by the input interference pattern created by earlier layers and by the local transmission/reflection coefficient at that point.

By analogy with standard deep neural networks, the transmission/reflection coefficient of each point/neuron can be regarded as a multiplicative «bias» term, which is iteratively adjusted during training of the diffractive network using the backpropagation algorithm. After numerical training, the design of the D2NN is fixed and the transmission/reflection coefficients of the neurons in all layers are determined. The calculated layers can then be fabricated using any method: 3D printing, lithography, etc.

The scientists emphasize that the optical neural network performs its function at the speed of light and requires no energy. Thus, it represents an efficient and fast way of implementing machine learning tasks.

See also

• AI accelerator
• Brain simulation
• Neuromorphic engineering
• Optical neural network
• Quantum neural network