Lecture
«We have a network with a thousand layers, dozens of GPUs, but we still haven't figured out where this could be useful. Let it draw kittens!»
Today it's used for:
Popular architectures: Perceptron, Convolutional Networks (CNN), Recurrent Networks (RNN), Autoencoders
If no one has ever tried to explain a neural network to you using the supposed workings of the brain, please tell me how you managed to hide? I will avoid these analogies and explain it the way I like it.
Any neural network is a set of neurons and connections between them. A neuron is best thought of simply as a function with a bunch of inputs and one output. The neuron's job is to take the numbers from its inputs, perform a function on them, and give the result to the output. A simple example of a useful neuron: sum up all the numbers from the inputs, and if their sum is greater than N — output one, otherwise — zero.
Connections are the channels through which neurons send numbers to each other. Each connection has its own weight — its only parameter, which can be loosely thought of as the strength of the connection. When the number 10 passes through a connection with a weight of 0.5, it turns into 5. The neuron itself doesn't care what comes to it and sums everything up indiscriminately — that's what the weights are for, to control which inputs the neuron should react to and which it shouldn't.
So that the network doesn't turn into anarchy, it was decided to connect neurons not however one likes, but layer by layer. Within one layer, neurons are not connected to each other in any way, but are connected to the neurons of the next and previous layers. Data in such a network flows strictly in one direction — from the inputs of the first layer to the outputs of the last.
If you cram in a sufficient number of layers and set the weights correctly in such a network, you get the following — feeding in, say, an image of a handwritten digit 4, the black pixels activate the neurons connected to them, those activate the next layers, and so on and so forth, until finally the very output responsible for the four lights up. The result is achieved.
In real programming, of course, no one writes out neurons and connections by hand — everything is represented as matrices and computed via matrix products, because speed is needed. I have two favorite videos where this whole process I described is clearly illustrated using the example of handwritten digit recognition. Watch them if you want to get a deeper understanding.
A network like this, with several layers where all the neurons between them are connected, is called a perceptron (MLP) and is considered the simplest architecture for beginners. In real-world production tasks, I personally have never come across it.
Once we've built the network, our task is to set the weights correctly so that the neurons respond to the right signals. Here we need to recall that we have data — examples of «inputs» and the correct «outputs». Let's show the neural network a picture of the same digit 4 and say «adjust your weights so that with this input your output always lights up as a four».
At first all the weights are simply set randomly, we show the network a digit, it gives some random answer (there are no real weights yet), and we compare how far the result is from what we need. Then we go through the network in the reverse direction, from the outputs to the inputs, and tell each neuron — hey, you activated here for some reason, and because of you everything went wrong, so react a little less to this connection and a little more to that one, okay?
After about a hundred thousand of these «run-check-punish» cycles, there is hope that the weights in the network will adjust the way we wanted. Scientifically, this approach is called Backpropagation, or the «Method of Backward Error Propagation». Funnily enough, it took twenty years to discover this method. Before it, neural networks were trained however people could manage.
My second favorite video explains the whole process in more detail, but still just as simply, step by step.
A well-trained neural network could mimic any algorithm from this article, and often even work more accurately. This universality made them wildly popular. Finally we have an architecture of the human brain, they said, we just need to stack a lot of layers and train them on any data, they hoped. Then the first AI Winter began, then a thaw, then a second wave of disappointment.
It turned out that training a network with a large number of layers required computing power that was simply impossible at the time. Nowadays any gaming rig with GeForce cards outperforms the power of a data center from back then. Back then there wasn't even any hope of that, and everyone became deeply disillusioned with neural networks.
Until about ten years ago, when deep learning exploded onto the scene.
On the English Wikipedia there's a page called Timeline of machine learning, where you can clearly see the bursts of excitement and the waves of despair.
In 2012, a convolutional neural network crushed the competition at ImageNet, which caused the world to suddenly remember the deep learning methods that had been described back in the 90s. Now we finally have video cards!
The difference between deep learning and classical neural networks lay in the new training methods that could handle much larger networks. Today, though, only theorists really bother to distinguish which kind of learning counts as deep and which doesn't. As practitioners, we use popular «deep» libraries like Keras, TensorFlow, and PyTorch even when we just need to put together a tiny five-layer network. Simply because they're more convenient than anything that came before. We just call it neural networks.
Let me tell you about the two main ones at the moment.
Convolutional networks are at the peak of their popularity right now. They're used for finding objects in photos and video, face recognition, style transfer, generating and inpainting images, creating slow-mo effects, and improving photo quality. Today CNNs are used everywhere there are pictures or video. Even your iPhone has several such networks looking at your naked photos to recognize objects in them. If, of course, there's anything to recognize heh-heh.
The picture above — is the result of the Detectron library, which Facebook recently open-sourced
The problem with images has always been that it's not obvious how to extract features from them. Text can be split into sentences, and word properties can be taken from dictionaries. Images, on the other hand, had to be labeled by hand, explaining to the machine where the kitten's ears are in the photo and where its tail is. This approach was even called «feature handcrafting» and that's how everyone used to do it.
There are many problems with manual crafting.
First, if the cat in the photo has flattened its ears or turned away — that's it, the neural network won't see anything.
Second, try naming at least ten characteristic features that distinguish cats from other animals right now. I couldn't do it. Yet when a black blur runs past me at night, even out of the corner of my eye I can tell whether it's a cat or a rat. That's because a person doesn't just look at the shape of the ears and the number of legs — they evaluate the object using a huge number of different features they're not even consciously aware of. Which means they don't understand them and can't explain them to a machine.
So it turns out the machine has to learn on its own how to find these features, building them up out of some basic lines. Here's how we'll do it: first, we'll split the image into 8x8 pixel blocks and pick which line dominates in each one — horizontal [-], vertical [|], or one of the diagonals [/]. There could be two, or even three, that happens too, we're not always completely sure.
As output we get several arrays of little sticks, which are essentially the simplest features indicating the presence of object outlines in the picture. In essence, these are also images, just made of sticks. So we can again pick an 8x8 block and look at how these sticks combine with each other. And then again, and again.
This operation is called convolution, which is where the method gets its name. A convolution can be thought of as a layer of a neural network, since a neuron — is absolutely any function at all.
When we run a bunch of cat photos through our neural network, it automatically assigns larger weights to the stick combinations it saw most often. And it doesn't matter whether it's a straight line of the back or a complex geometric shape like a snout — something will definitely activate strongly.
At the output, we'll put a simple perceptron that looks at which combinations activated and says which one they're more characteristic of — a cat or a dog.
The beauty of the idea is that we end up with a neural network that finds the characteristic features of objects on its own. We no longer need to pick them out by hand. We can feed it as many images of any objects as we like, just by googling up a million pictures of them — the network will build its own feature maps out of sticks and learn to identify anything.
I even have an unfunny joke about this:
Give a neural network a fish — and it'll be able to recognize fish for the rest of its life. Give a neural network a fishing rod — and it'll be able to recognize the fishing rod too, for the rest of its life...
The second most popular architecture today. Thanks to recurrent networks we have such useful things as machine text translation (read my post about it) and computer speech synthesis. They're used to solve any task involving sequences — whether voice, text, or music.
Remember old-school voice synthesizers like Microsoft Sam from Windows XP, which funnily spelled out words letter by letter, trying somehow to glue them together? Now look at Amazon Alexa or Yandex's Alice — today they don't just pronounce words without mistakes, they even place stress and intonation within a sentence!
The neural network learns to talk
Because modern voice assistants are trained to speak not letter by letter, but in phrases. But you can't just make a neural network output whole phrases right away, because then it would need to memorize every phrase in the language and its size would be enormous. This is where it helps that text, speech, or music — are sequences. Each word or sound — is, in a sense, a self-contained unit, but one that depends on the ones before it. When that connection is lost — you get dubstep.
It's fairly easy to train a network to pronounce individual words or letters. We take a bunch of audio files labeled by word and train the network so that, given an input word, it outputs a sequence of signals resembling its pronunciation. We compare it against the original recording from a speaker and try to get as close to the ideal as possible. Even a perceptron will work for this.
The only trouble is that sequences are a problem again, since a perceptron doesn't remember what it generated before. For it, every run is like the first time. That gave rise to the idea of adding memory to each neuron. That's how recurrent networks were invented, where each neuron remembers all its previous outputs and uses them as an additional input on the next run. In other words, a neuron could tell its future self — hey dude, the next sound should be a bit higher pitched, we've got a vowel here (a very simplified example).
There was just one problem — when each neuron remembered all its past outputs, the network ended up with such a wild number of inputs that training that many connections became unrealistic.
When a neural network doesn't know how to forget — it can't be trained (people have the same problem).
At first the problem was solved head-on — every neuron's memory was simply cut off. But then people came up with the idea of using special cells as this «memory», similar to computer memory or processor registers. Each cell let you write a number into it, read it, or reset it — these were called long short-term memory cells (LSTM).
When a neuron needed to set itself a reminder for the future — it wrote it into the cell; conversely, when the whole history became unnecessary (a sentence ended, for example) — the cells were reset, leaving only the «long-term» connections, as in a classic perceptron. In other words, the network learned not only to set up current connections, but also to set reminders.
Simple, but it works!
CNN + RNN = fake Obama
Voiced texts for training started being taken from anywhere. Even BuzzFeed managed to pull video recordings of Obama's speeches and train a neural network to talk in his voice, quite convincingly. This example shows that imitating a voice — is a fairly simple task for today's machines. Video is trickier, but only for now.
Comments