Impulse neural network

Pulsed neural network (ImNS, Eng. Pulsed neural networks , PNN) or Spiking neural network (SNN, Eng. Spiking neural network , SNN) - the third generation of artificial neural networks (INS) [1] , which differs from binary (first generation) and frequency / speed (second generation) of the ANN by the fact that neurons exchange short (in biological neurons - about 1-2 ms) pulses of the same amplitude (in biological neurons - about 100 mV). It is the most realistic, from the point of view of physiology, model of ANN [2] [3] [4] .

Content

• 1. History
  • 1.1 Source
  • 1.2Modern studies
• 2Device
  • 2.1 Principle of work
  • 2.2Neuron models
  • 2.3 Information submission
  • 2.4Device
  • 2.5 Training Methods
• 3Quality features
  • 3.1 Benefits
  • 3.2 Disadvantages
• 4Realizations
  • 4.1Program
  • 4.2 Hardware
• 5Application
  • 5.1 Prosthetics
  • 5.2Robotics
  • 5.3 Computer vision
  • 5.4 Telecommunications
• 6Notes

Story

Origins

The first scientific model of the impulse neural network was proposed by Alan Hodgkin and Andrew Huxley in 1952. This model described how action potentials arise and spread. Impulses, however, are usually not transmitted directly between neurons. Communication requires the exchange of chemicals called neurotransmitters in the synaptic cleft [5] .

From the point of view of information theory, the problem lies in the absence of a model that would explain how information is encoded and a series of impulse sequences decoded, that is, action potentials. For neurobiology, the answer to the question is still open: are neurons connected using frequency or time coding [6] ? Using temporal coding, one pulse neuron can replace hundreds of hidden elements of the frequency neural network [1] .

Current research

Currently, there are two areas of research IMHC [2] :

1. creation of computer models that faithfully reproduce the functioning models of real brain neurons, due to which it will be possible both to explain the mechanisms of its operation and to diagnose / treat diseases and injuries of the central nervous system;
2. creation of computer models that abstractly reproduce models of the functioning of real brain neurons, which will allow to use all the advantages of a real brain, such as noise immunity and energy efficiency, when analyzing large amounts of data.

Insect animation is managed by IMNS to search for targets in unknown areas.

Classification of complex images coming from two vision systems, in real time by the IMNS deep learning.

Device

Pulsed neural network with one hidden layer

Principle of operation

The network receives a series of pulses at the inputs and generates pulses at the output. At each instant, each neuron has a certain value (the analogue of the electric potential of biological neurons) and, if this value exceeds the threshold, the neuron sends a single pulse, after which its own value falls to a level below the average value (similar to the rehabilitation process of biological neurons, called the refractory period) for 2-30 ms. When out of equilibrium, the potential of the neuron begins to gradually move towards the average value. There are only two parameters of the weight connections of the pulsed neuron - the delay time and the weight value [7] .

Neuron models

Main article: Biological neuron model

Methods of modeling neurons of the IMNS can be divided into two groups [4] [8] :

• conduction models are similar to the process of ion channels;

1. Hodgkin's model - Huxley;
2. Izhikevich model;
3. FitzHugh model - Nagumo;
4. Hindmarsh-Rose Model;
5. The Morris - Lecar Model [ru] ;
6. Model Wilson - Cowan [ru] ;
7. Galves-Lecherbach model;
8. Multi-camera model [en] ;
9. Cable theory of dendrites.

• threshold value models - generate a pulse at a certain voltage level.

1. Integrate-and-work method;
2. Integrate-and-work method with leaks.

Presentation of information

The way of presenting information in pulse neural networks: phase.

The way of presenting information in the pulse neural networks: synchronous.

The way of presenting information in IOMS: the time until the first impulse.

The way of presenting information in IOMS: ordinal.

The way of presenting information in IOMS: interval.

Frequency ANNs use a signal that takes a value that depends on the frequency of generation of pulses by a certain group of neurons (the weights of the neurons, in fact, are a form of representation of this frequency) [3] . However, the average frequency of the pulses in the sequence is a rather poor way of presenting information, since different types of stimulation can lead to the same average frequency of the pulses [9] .

To get rid of this shortcoming in pulsed INS, the following types of information representation are used [3] [8] [10] [4] :

1. phase (time) - information about the signal is set by the exact (or within a certain window) position of the pulses in time (relative to any common reference rhythm of the brain);
2. synchronous (positional / spatial / population) - information about the signal is given by the synchronous activity of different groups of neurons, and, as a result, the synchronous (or within a certain window) appearance of pulses at certain network outputs (for example, cochlear auditory receptors that react to high and low ears are in different zones);
3. time before the first pulse appears - information about the signal is set by the time of the first pulse appearance at any output;
4. sequential - information about the signal is set in the order of receiving pulses at the network outputs;
5. interval (delay) - information about the signal is set by the distance between the pulses received at the network outputs;
6. resonant - information about the signal is given by a dense sequence of pulses (queue), which leads to the appearance of resonance (single pulses fade out and make no contribution to the transmission of information).

In addition, there are types of information presentation, which are a mixed form of several simple types of information presentation, for example:

1. space-time - information is not only defined by a specific sequence of pulses in time, but they must also come from a certain group of neurons;
2. population-frequency - information is given by increasing the frequency of generation of impulses by a certain group of neurons.

Device

ImNS architecture can be divided into the following groups [8] :

1. Direct propagation network [en] (NPR) - data is transmitted strictly in one direction: from inputs to outputs, there are no feedbacks, and processing can take place on multiple layers;
2. Recurrent neural network (RNS) - individual neurons / populations of neurons interact with each other, that is, there is feedback. ANNs of this type have their own dynamics and high computational ability;
3. Mixed neural network [en] - inside the ANN, some populations of neurons belong to the type of CPD, and some - to the PHC. The interaction between populations can be both unidirectional and reciprocal.

1. Synchronous excitation of a circuit [en] - is a multilayer circuit in which impulse activity can propagate in the form of a synchronous wave of transmission of packets of pulses from one population to the next;
2. Reservoir calculations [ru] - reservoir ANN consists of a reservoir, executed by a recurrent form, and output neurons.

Teaching methods

Methods of teaching IOMS are divided into three groups [10] [8] [11] :

• Teaching without a teacher:

1. Plasticity model, time-dependent impulse [en] (English Spike-timing-dependent plasticity (STDP) );
2. Growing pulse neural networks (English Growing spiking neural networks );
3. ABS rule (eng. Artola, Bröcher, Singer (ABS) rule );
4. BCM rule (English Bienenstock, Cooper, Munro (BCM) rule );
5. The relationship between the rules of BCM and STDP (English Relationship between BCM and STDP rules );
6. General education without a teacher (English General unsupervised learning ).

• Teaching with a teacher:

1. SpikeProp (English SpikeProp );
2. Deep learning methods;
3. Remote method of teaching with a teacher (English Remote Supervised Method (ReSuMe) );
4. FreqProp (English FreqProp );
5. Local error-controlled associative biological realistic algorithm [en] (Eng. Local error-driven associative biologically realistic algorithm (LEABRA) );
6. Hebb training with a teacher (English Supervised Hebbian Learning ).

• Training with reinforcements:

1. The impulse method of the actor and critic (English Spiking actor-critic );
2. Reinforcement training with the help of the STDP ( RL through reward-modulated STDP ) measurement-promoting model.

Qualitative characteristics

Benefits

Pulse ANNs have several advantages over previous-generation neural networks [12] :

1. HMSS are dynamic, and therefore are excellent for working with dynamic processes (speech recognition and dynamic images) [2] ;
2. ImNS have multitasking, because the input data is processed in a neural network with feedback, and different groups of reading neurons can be trained to solve different problems;
3. IMNs are capable of performing recognition with foresight (that is, it is not necessary to have complete information about the object or to know the result of the process);
4. ImNS is easy to teach, since it is enough to train only the output counting neurons;
5. IOMS have increased productivity of information processing and noise immunity, as they use temporal information representation;
6. HMNS requires a smaller number of neurons, since each neuron of a pulsed neural network replaces two neurons (exciting and inhibiting) of the classical ANN;
7. ImNS have a high operation speed and a large potential for parallelization, since for transmitting a pulse it is necessary to send 1 bit, and not a continuous value, as in frequency INS [13] ;
8. IOMS can be trained in the process of work [14] .

disadvantages

1. DIVS is impractical to use in systems with a small number of neurons;
2. There is no perfect learning algorithm.

Implementations

Red dots mark the first supercomputers in power from TOP500 for the respective years. The blue and green horizontal lines reflect the estimated (according to Raymond Kurzweil) performance requirements (expressed in FLOPS) supercomputers for emulating the corresponding levels of the human brain [15] . Thus, with a doubling of power every 1.1 years, in 2019 the first supercomputer will appear (10 18 FLOPS CPUs and 10 4 Tb of RAM), capable of fully emulating the activity of a pulsed neural network.

Software

Software that is used mainly to simulate impulse neural networks and used by biologists to study their work, properties and characteristics. Allows you to simulate with a high level of detail and accuracy, but require a lot of modeling time.

• GENESIS [ru] ;
• Neuron [ru] ;
• Brian [ru] ;
• NEST [en] .

Software that can be used to solve real, not theoretical problems. Modeling in them is very fast, but it does not allow modeling of complex, that is, biologically realistic, models of neurons.

• Spikenet.

Software that works effectively enough to simulate quickly, sometimes even in real time, but at the same time, it is able to use neural models, which are described in detail and biologically plausible. All this is very convenient for information processing tasks.

• EDLUT [ru] .

Hardware

Using existing user architecture:

• Neurogrid [ru] ;
• SpiNNaker.

Using its own specialized architecture:

• TrueNorth.

Application

Prosthetics

Visual and auditory neuroprostheses that use a sequence of pulses to send signals to the visual cortex, and return patients the ability to navigate in space, already exist, and work on mechanical motor prostheses is actively underway. Also, pulse sequences can be supplied to the brain via electrodes implanted into it and, thus, eliminate the symptoms of Parkinson's disease, dystonia, chronic pain, MDP and schizophrenia [2] .

Robotics

The San Diego Brain Corporation develops robots using ImNS, and SyNAPSE creates neuromorphic systems and processors [2] [8] .

Computer vision

There are prospects for the use of IMNS in computer vision (automatic analysis of video information). The IBM TrueNorth digital neurochip includes a million programmable neurons and 256 million programmable synapses, which makes it possible to simulate the operation of neurons in the visual cortex. This neurochip consists of 4096 cores, contains 5.4 billion transistors, but at the same time it has serious energy efficiency - only 70 milliwatts [2] .

Telecommunications

Qualcomm is engaged in research into the possibility of using the HPS in telecommunication devices [2] .