Lecture 6 min.
The object generation problem (object generation problem) is a machine learning task that consists of creating new plausible objects based on a given sample. The resulting objects can be used for applied purposes (in which case they are most often images), as well as for generating objects for a training set when labeling real data is slow and expensive, or when the data needs to be anonymized. Depending on which of these purposes the generation is used for, the problem statement and the methods for solving it differ somewhat.
In object generation, the main task is usually to learn to create images that a human cannot distinguish from images obtained by other means (Fig. 1). They can be used for cheaper production of model photos, covers or landscapes. One striking example of such an application is creating photographs of fictional people for advertising, in the expectation that people will identify more with an image that does not represent anyone in particular but combines the features they themselves have.
Object generation can improve astronomical images and be used to model physical processes that are expensive to study. For example, in 2019 generative adversarial networks (GAN) were used to successfully model dark matter distributions in a particular direction in space and to make predictions of gravitational lensing.
Generation of study results is actively used in medicine. Because of the ban on using tests and examinations without the patient's consent, it is often quite hard to obtain a large amount of data, so GANs have now come into use for building large datasets. Adversarial networks can also be used to detect glaucomatous images, aiding early diagnosis, which is necessary to prevent partial or complete loss of vision.
Audio is another possible application of GANs, but it is used much less often. An example is the WaveGAN architecture, which, without labels, learns to reproduce intelligible words when trained on a speech dataset with a small vocabulary, and can also synthesize sound from other domains, such as drums, bird vocalizations and piano.
Today many music generation models use long short-term memory (LSTM). For example, as early as 2002 an LSTM network was used to build a model that generates not only a melody but also an accompaniment to it, in the form of choosing a chord from a fixed list, and, in part, achieved a pleasant-sounding result for the first time.
Google is now actively using the WaveNet model, which is based on convolutional networks. WaveNet can generate speech that resembles the voice of any person, as well as other sounds, including music (for example, piano compositions) .
Documents and texts can be generated. Text generators are widely used in website development and search engine optimization: for generating titles, descriptions and content. There are large English-language websites whose entire content is written not by journalists but by bots: articles are automatically rewritten from other sources. Unlike English, Russian has complex morphology, so the appearance of such rewriter bots on the Russian-language internet is much more difficult.
Object generation can also be used to recreate the textures of old games in higher resolution (an example of a game for which this method was used is Resident Evil). This approach also helps create characters in a cartoon style by drawing by hand only a couple of them, animate already drawn characters, and is useful for preparing frames of films or animation . In 2018 researchers from the University of Illinois and the Allen Institute for Artificial Intelligence developed a model called CRAFT (Composition, Retrieval and Fusion Network) , which takes text descriptions (or captions) from the user and generates scenes from the cartoon series "The Flintstones" (Fig. 2).
When solving a problem, it is often convenient to generate specific objects for solving it. For example, one of the machine learning tasks in medicine is generating new molecules that could potentially be drugs. Generative adversarial networks are used to solve this problem.
Another example is generating datasets with given properties for a classification task . This can be used, in particular, to generate data for automatic program checking systems or in algorithms that predict the manufacturing cost of a part from its drawing and text characteristics.
Unit tests can be generated in order to find errors faster in software development .
Object generation is developing actively today and has many applications, including for specific tasks in various fields of activity.
Generative models are usually used to achieve this goal. In this setting, the task is to recover the joint distribution p(x,y) where y is a single bit responsible for a particular attribute (that is, the class to which the created object should belong; for example, a photograph of a person), and x is the whole object (the photograph). Most often the generated object X is a set of elements X={xi} , which allows the object to be generated in parts. For an image, for example, such parts are pixels. Thus, when generating the next parts of the object we can rely on those already created, and then we face the task of maximizing the likelihood function: for a dataset X={xi} maximize
over the model parameters θ , i.e. find
. This task belongs to the class of unsupervised or semi-supervised learning problems. To solve it, one either works with explicit distributions, reducing the distribution p(x,y) to a product of distributions of a certain structure, or uses implicit models that do not recover the entire density function but only model the part of this function that is directly needed. It is worth noting that simple generative models, such as the naive Bayes classifier, do not achieve sufficient quality to generate full-fledged multimedia objects. Among generative models, adversarial networks, PixelRNN and PixelCNN, as well as DRAW (which draw an image using a combination of recurrent neural networks and an attention mechanism), have shown themselves particularly well at image generation.
Sign in and the lectures you have studied will be marked in this list.
Choose a course
Comments