Lecture
Machine translation — the process of translating texts (written, and ideally spoken as well) from one natural language into another using a special computer program. The term is also used for the field of scientific research concerned with building such systems.
The idea of using computers for translation was first voiced in 1947 in the USA, immediately after the appearance of the first computers. The first public demonstration of machine translation (the so-called Georgetown experiment) took place in 1954. Despite the primitiveness of that system (a 250-word dictionary, a grammar of 6 rules, translation of a few simple phrases), the experiment received wide resonance: research began in England, Bulgaria, the GDR, Italy, China, France, the FRG, Japan and other countries; and that same year, 1954, in the USSR as well.
By the mid-1960s two Russian-English translation systems had been made available for practical use in the USA:
However, the ALPAC commission, set up to evaluate such systems, concluded that, given the low quality of machine-translated texts, this activity was not cost-effective under US conditions. Although the commission recommended continuing and deepening the theoretical work, its conclusions overall led to growing pessimism, reduced funding, and often to a complete halt of work in this area.
Troyanskii's machine (Illustration based on descriptions.)
«a machine for selecting and printing words when translating from one language to another». The machine was extremely simple: a large table, a typewriter with a ribbon, and a film camera. On the table lay cards with words and their translations in four languages.
The operator took the first word from the text, found the matching card, photographed it, and typed its morphological information on the typewriter — «noun, plural, genitive case». Its keys had been modified for convenience, each one unambiguously encoding one of the properties. The typewriter ribbon and the camera film advanced in parallel, producing at the output a set of frames containing the words together with their morphology:
The resulting ribbon was handed to linguists who knew the specific languages, and they turned the set of photographs into a coherent literary text. It follows that, to translate texts, both the operator and the linguists needed to know only their own native language. Troyanskii's machine was the first to put into practice that very «intermediate language» (interlingua) which even Leibniz and Descartes had dreamed of creating.
Nevertheless, research continued in a number of countries, helped along by the steady progress of computing technology. An especially significant factor was the emergence of mini- and personal computers, and with them ever more sophisticated dictionary, search and similar systems oriented toward working with natural-language data. The need for translation as such also grew, owing to the growth of international ties. All this led to a new upswing in the field, beginning around the mid-1970s. The 1980s brought a time of broad practical use of translation systems, and a market for commercial developments on this subject took shape.
However, the dreams with which humanity took up the task of machine translation half a century ago largely remain dreams: high-quality translation of texts on a broad range of subjects is still unattainable. Nevertheless, it is beyond doubt that the use of machine translation systems speeds up a translator's work: according to estimates from the late 1980s, by up to five times.
Today there are many commercial machine translation projects. One of the pioneers in the field of machine translation was the company SYSTRAN[en]. In Russia, a major contribution to the development of machine translation was made by the group led by Prof. R. G. Piotrovskii (Herzen State Pedagogical University of Russia, Saint Petersburg).
In the 1960s Stanislav Lem summarized statements on the problem of machine translation and its connection with the machine's own understanding of the text (which relates, for example, to the discussion of the «Chinese room» concept formulated in 1980):
| ... we insist on endowing translating machines with the «fullness of the inner life» of a human being; yet we simply do not know to what extent it is possible to «shortchange the personality» of a machine that is meant to translate well. We do not know whether it is possible to «understand» without possessing a «personality», even in embryonic form. <…> It does not seem possible to make full, effective use of an operational language as an instrument of translation in the sphere of discursive — that is, thinking — languages. Either machines will act «with understanding», or there will never be truly effective translating machines at all . |  |
The quality of translation depends on the subject matter and style of the source text, as well as on the grammatical, syntactic and lexical relatedness of the languages between which the translation is being made. Machine translation of literary texts almost always turns out to be of unsatisfactory quality. Nevertheless, for technical documents, given specialized machine dictionaries and some tuning of the system to the peculiarities of a given type of text, it is possible to obtain a translation of acceptable quality that needs only minor editorial correction. The more formalized the style of the source document, the higher the quality of translation one can expect. The best results when using machine translation can be achieved for texts written in a technical style (various descriptions and manuals) and in an official-business style.
The use of machine translation without tuning to the subject matter (or with deliberately incorrect tuning) is the subject of numerous jokes that circulate on the Internet. Among the old and most popular examples of such jokes is the translated text of documentation for a mouse driver, known as «Gurtovshchiki Myshi» (literally, "The Mouse Drovers"), presented as «a translation of computer documentation by the Poliglossum machine translation system based on medical, commercial and legal dictionaries»[comm. 1]. Among the shorter ones is the phrase «Our cat gave birth to three kittens — two whites and one black», which the online translator «PROMT» (version 7.0, 2007) turned into «Наш кот родил трёх котят — двух белых и одного афроамериканца» (literally: "Our tomcat gave birth to three kittens — two white ones and one African American"). While «афроамериканца» ("African American") could still be turned into «чёрным» ("black") by writing «black kitten», there was no way to get the «cat» to change sex: for example, female cat was translated as «самка кот» (literally "female [animal] tomcat").
Most often such jokes are related to the fact that the program does not recognize the context of a phrase and translates terms literally, and moreover does not distinguish proper names from ordinary words. The same PROMT translator turned «Лев Толстой» (Leo Tolstoy) into «Lion Thick», «bra-ket notation» into «примечание Кети лифчика» (literally, "note of Katie's bra"), «Lie algebra» into «алгебру Лжи» (literally, "algebra of a lie"), «eccentricity vector» into «вектор оригинальности» (literally, "vector of originality"), «Shawnee Smith» into «индеец племени шони Смит» (literally, "an Indian of the Shawnee tribe named Smith"), and so on. Conversely, Google Translate often mistook the word «rice» for the surname of the US Secretary of State.
Thus, the following types of translation using computers exist
Instead of «machine» the word automatic is sometimes used, which does not affect the meaning. However the term computer-assisted translation has a quite different meaning — in this case the program simply helps a human translate texts.
Computer-assisted translation involves the following forms of interaction:
In English-language terminology a distinction is also drawn between the terms machine translation, MT (fully automatic translation) and machine-aided or machine-assisted translation (MAT) (computer-assisted); if it is necessary to denote both at once, M(A)T is written.
There are several fundamentally different approaches to building machine translation algorithms: rule-based, statistical, or based on statistics (statistical-based), and neural machine translation (NMT). The first approach is the traditional one and is used by most developers of machine translation systems (PROMT in Russia, SYSTRAN in France, Linguatec in Germany, and others). The second type includes the popular Yandex.Translate service, Google Translate, as well as a new service from ABBYY. Most systems today are hybrid — combining rules, statistics and neural networks.
Statistical machine translation is a form of machine text translation based on comparing large volumes of language pairs. Language pairs are texts containing sentences in one language along with the corresponding sentences in a second language; they can be either variant renderings of two sentences by a person who is a native speaker of both languages, or a set of sentences and their human-made translations. Statistical machine translation therefore has the property of «self-learning». The more language pairs available and the more precisely they correspond to one another, the better the result of statistical machine translation. The concept of «statistical machine translation» refers to a general approach to solving the translation problem, based on finding the most probable translation of a sentence using data obtained from a bilingual body of text. Parliamentary records, which are transcripts of debates in parliament, can be cited as an example of a bilingual body of text. Bilingual parliamentary records are published in Canada, Hong Kong and other countries; official documents of the European Economic Community are published in 11 languages; and the United Nations publishes documents in several languages. These materials have turned out to be invaluable resources for statistical machine translation.
Neural machine translation (Neural Machine Translation, NMT) is an approach to machine translation that uses a large artificial neural network. It differs from machine translation methods based on phrase statistics, which use separately developed subcomponents .
The translation services of Google, Yandex, Microsoft and PROMT already use neural translation. Google uses Google Neural Machine Translation (GNMT) instead of the previously used statistical methods. Microsoft uses a similar technology for speech translation (including in Microsoft Translator and Skype Translator). The Harvard NLP group has released OpenNMT, an open-source neural machine translation system. Yandex.Translate has a hybrid model: both the statistical model and the neural network offer their own translation variant. After that, CatBoost technology, which is based on machine learning, selects the best of the resulting outputs .
NMT models use deep learning and feature learning. They require only a small fraction of the memory compared to traditional statistical machine translation (SMT) systems. Moreover, unlike traditional translation systems, all parts of the neural translation model are trained jointly (end to end) in order to maximize translation efficiency .
A bidirectional recurrent neural network (RNN), also known as an encoder, is used by the neural network to encode the source sentence for a second recurrent network, also known as a decoder, which is used to predict words in the target language
Rule-based machine translation (Rule-Based Machine Translation) — a general term denoting machine translation systems based on linguistic information about the source and target languages . They consist of bilingual dictionaries and grammars that cover the basic semantic, morphological and syntactic regularities of each language. This approach to machine translation is also called classical. Based on this data, the source text is converted sentence by sentence, in sequence, into the translated text. These systems are contrasted with machine translation systems that are based on examples. The operating principle of such systems is the connection between the structure of the input sentence and that of the output sentence.
RBMT systems are divided into three groups:
The main advantage of transfer-based systems is a high degree of text coverage at an acceptable level of translation quality, as well as a low cost of initial development and modernization.
Components of a typical RBMT:
Features of RBMT systems:
Example-based machine translation (Example-based machine translation, EBMT) is a machine translation method that is often characterized by the use of a bilingual corpus with parallel texts as the main knowledge base at translation runtime. In essence, it is translation by analogy, which can be viewed as an application of the case-based reasoning method to machine learning.
At the heart of example-based machine translation lies the idea of translation by analogy. As applied to the process of human translation, the notion that translation is performed by analogy amounts to a rejection of the idea that people translate sentences by performing deep linguistic analysis. Instead, this notion is based on the belief that people translate by first breaking sentences down into specific phrases, then translating those phrases, and finally correctly assembling these fragments into one long sentence. Phrase-by-phrase translations are performed by analogy with previous translations. The principle of translation by analogy is encoded in example-based machine translation through the translation examples used to train such a system. Other approaches to machine translation, including statistical machine translation, also use bilingual corpora to learn the translation process.
Example-based machine translation was first proposed by Makoto Nagao in 1984 . Nagao pointed out that this type of translation is especially well suited for translation when two entirely different languages are involved, such as English and Japanese. In this case one sentence can be translated into several well-structured sentences in another language, so there is no point in doing the kind of deep linguistic analysis characteristic of rule-based machine translation.
In general, an EBMT system consists of three components: matching, recombination and alignment .
Example of a bilingual corpus
| English | Japanese |
|---|---|
| How much is that red umbrella? | Ano akai kasa wa ikuradesuka. |
| How much is that small camera? | Ano chiisai kamera wa ikura desu ka. |
Example-based machine translation systems are built from bilingual parallel corpora containing sentence pairs, such as the example shown in the table above. Sentence pairs contain sentences in one language along with their translation into another. In this example a minimal pair is shown, meaning that the sentences differ by only one element. Such sentences make it easier to memorize translations of parts of a sentence. For instance, an example-based machine translation system will memorize three translation units from the example above:
Composing these units can be used to create new translations in the future. For example, if we were trained using a text containing the sentences: President Kennedy was shot dead during the parade and The convict escaped on July 15th, we could translate the sentence The convict was shot dead during the parade by substituting the corresponding parts of the sentences.
Example-based machine translation is best suited to sub-language phenomena such as phrasal verbs. Phrasal verbs have highly context-dependent meanings. They are common in English and consist of a verb followed by an adverb and/or preposition, called a particle within the verb. Phrasal verbs form specialized, context-specific meanings that cannot be derived from the meaning of their components. When translated word for word from the source language to the target language, ambiguity almost always arises . As an example, consider the phrasal verb «put on» and its meaning in Hindi-Urdu. It can be used in any of the following ways:
Statistical machine translation (Statistical machine translation — SMT) is a form of machine translation in which the translation is generated on the basis of statistical models whose parameters are derived from the analysis of bilingual text corpora (text corpora).
Statistical machine translation is contrasted with machine translation systems based on rules, Rule-Based Machine Translation (RBMT), and on examples, Example-Based MT (EBMT).
The first ideas of statistical machine translation were published by Warren Weaver in 1949. The «second wave» — the early 1990s, IBM. The «third wave» — Google, Microsoft, Language Weaver, Yandex …
To improve quality, developers of machine translation systems introduce certain «cross-cutting» rules, thereby turning purely statistical systems into hybrid machine translation. Adding some rules, that is, creating hybrid systems, somewhat improves translation quality, especially when there is an insufficient volume of input data used to build the machine translator's index.
As a language model, statistical translation systems predominantly use various modifications of the n-gram model, which holds that the «grammaticality» of the choice of the next word when forming a text is determined only by which (n-1) words precede it .
Transformation-based machine translation is a form of machine translation (MT). At present it is one of the most widespread methods of machine translation. Unlike the simpler direct MT model, transformation-based MT splits the translation process into three stages: analyzing the source-language text to determine its grammatical structure, converting the resulting structure into a structure suitable for producing text in the target language, and generating the text. Thus, transformation-based MT systems are able to make use of knowledge of both the source language and the target language .
Transformation-based translation and interlingual machine translation are built on the same underlying idea, according to which, in order to carry out a translation, an intermediate representation must be obtained. It can be used to capture the meaning of the original sentence, so as to then construct the correct translation. In interlingual MT such an intermediate representation must be independent of both the source language and the target language, whereas in the case of transfer-based MT there is a certain degree of dependence on the specific language pair. The ways in which transformation-based MT systems work differ significantly, but in general they follow the same scheme: they apply sets of linguistic rules defined by correspondences between the structure of the source language and that of the target language. The first stage involves analyzing the input text in terms of morphology and syntax (and sometimes also semantics) to create an intermediate representation. From the resulting representation, using bilingual dictionaries and grammatical construction rules, the translation is formed. This strategy makes it possible to obtain a fairly high-quality translation with an accuracy of correspondence to the original of around 90% (although the accuracy depends to a large extent on the specific language pair and is determined by the degree of closeness of the two particular languages).
In a rule-based MT system, the source text is first analyzed in terms of morphology and syntax in order to obtain a syntactic representation. This representation may subsequently be altered toward less specificity, owing to the need to pay increased attention to the fragments most essential for translation while ignoring other kinds of information. During the transformation process, the final representation (still existing in the source language) is converted into a representation of the same level of specificity in the target language. These two representations are called intermediate representations. The process of converting the representation in the target language into finished text consists of similar stages carried out in reverse order.
Before the final result is obtained, various methods of analysis and transformation may be invoked. Alongside statistical approaches, the number of generative hybrid systems may be increased. The methods chosen and the priorities applied depend to a large extent on the design of the system itself. Nevertheless, most existing systems include at least the following stages:
One of the main features of transformation-based MT systems is the stage at which the intermediate representation of the text in the source language is translated into the intermediate representation of the text in the target language. This process can take place at one of the levels of linguistic analysis or somewhere in between them. The levels are presented below:
I give the English names because it's exactly these abbreviations that are used everywhere in texts and conversations. Like HTTP or RSA, for example.
The idea of rule-based machine translation began to appear back in the 1970s. Scientists watched how linguist-translators worked and tried to program their big, slow computers to imitate them. Their systems consisted of:
That's basically it. Optionally they were supplemented with hacks like lists of names, spell-checkers and transliterators.
PROMT and Systran — the most famous examples of RBMT systems. «Охладите траханье, углепластик, я рассматриваю её пользу» (a garbled nonsense output, roughly "Cool down the fornication, carbon fiber, I am considering her benefit") — those were golden times. AliExpress still translates like that to this day.
But even they had their own nuances and subtypes.
The simplest method of machine translation, understandable to any fifth-grader. We split the text into words, translate each one, tweak the morphology a bit so it doesn't sound too clunky, agree the cases, endings and the rest of the syntax. Specially trained linguists write rules for every single word, night after night.
What comes out is something translated. Most often complete garbage. And the linguists were ruined for nothing.
Modern systems don't use this approach at all, I'm telling you this purely for a laugh.
In them, we don't jump straight into translating by dictionary, but prepare a little first. We break the text down into subject, predicate, look for modifiers and everything else the way we were taught in school. Grown-up folks call it «extracting syntactic constructions». After that, instead of embedding translation rules for each individual word into the system, we manipulate whole constructions. In theory we can even achieve a more or less decent conversion of word order between languages.
In practice it's still hard, linguists keep dying of physical exhaustion, and the translation still comes out essentially word-for-word. On one hand it's simpler: you can set general rules for agreement by gender and case. On the other hand it's harder: there are far more word combinations than there are words themselves. You can't account for every variant by hand.
We fully convert the source sentence into some intermediate representation, unified for all the languages of the world (interlingua). Into that very interlingua that Descartes himself dreamed of. A special metalanguage whose rules are uniform and cover all the languages of the world, thereby turning translation into the technical task of shuttling back and forth. Special parsers then convert this interlingua into the needed language, and there's your singularity.
Interlingua is often confused with transfer-based systems, since there's conversion there too. However, in transfer-based systems the conversion rules are written for two specific languages, while in interlingual systems they're written between each language and the interlingua. By adding a third language to an interlingual system, we can translate between all three, whereas in a transfer-based system we cannot.
In real life it all turned out not to be so sweet. Creating a universal interlingua by hand proved extremely difficult. Some scientists devoted their whole lives to it. Nothing came of it, but thanks to them we got methods of morphological, syntactic and sometimes even semantic analysis. Just the Meaning <-> Text model alone is worth something.
But the idea of an intermediate language will come back to us later. We'll just need to wait about 30 years.
fig. Vauquois pyramid
As you can see, all RBMT systems are dumb and terrible, which is why they're rarely used nowadays. Except in specific places like translating weather reports. Among the advantages of RBMT, people note morphological accuracy (it doesn't confuse words), reproducibility (every translator gets the same result), and the ability to tune it for a specific domain (train it on specialized terms for economists or programmers).
Even if we imagine that scientists had managed to create the perfect RBMT, and linguists had built all the rules of spelling into it, fun was already waiting for them behind the door — exceptions. Irregular verbs in English, separable prefixes in German, suffixes in Russian, and just plain situations where «that's not how we say it, it should be like this». Trying to account for every nuance turns into millions of man-hours wasted for nothing.
A pile of rules still doesn't solve the main problem — homonymy. The same word can carry different meanings depending on context, and so its translation differs too. This brings to mind an example from one of Segalovich's old lectures: «Эти типы стали есть на складе» (ambiguous: e.g. "These guys started eating at the warehouse" or "These types of steel are in the warehouse"). He said he could find four different ways to read this sentence. Can you?
Our languages developed not at all on the basis of the grammars and rules that linguists love to discuss. They depend much more on who attacked and conquered whom over the last three hundred years. So how am I supposed to teach that to a machine now?
In forty years of the Cold War, scientists never managed to find a clear solution. RBMT died.
Japan was especially invested in the battle for machine translation in those years. There was no Cold War there, but they had their own reasons: very few people in the country knew English. This promised serious trouble at the party of approaching globalization, which is why the Japanese were extremely motivated to find a working method of machine translation.
English-Japanese translation based on rules alone is extremely difficult, the structure of the languages differs, and almost every word has to be reordered or new ones added. In 1984, a scientist at Kyoto University named Makoto Nagao came up with an idea. What if, instead of trying to translate everything from scratch each time, we used ready-made phrases we already had?
Suppose we need to translate the sentence «я иду в магазин» (I'm going to the store). Somewhere in our stash we already have a translation of a similar phrase, «я иду в театр» (I'm going to the theater), and a dictionary entry for the word «магазин» (store). We could try to work out the difference and translate just that one word within the existing example, without messing up the rest of the construction. And the more examples we have, the better the translation.
I do that too, when building phrases in languages I don't know!
The historical importance of the method was that scientists around the world first realized: you don't have to spend years creating rules and exceptions, you can just take a pile of existing translations and feed them to the machine. This wasn't yet a revolution, but a step toward one. Five years remained before the revolution and the invention of statistical translation.
Around 1990, IBM's research center demonstrated for the first time a machine translation system that knew nothing about rules or linguistics. Scientists showed the soulless computer a huge number of identical texts in two languages and made it figure out the patterns on its own.
The idea was simple and elegant at the same time: take one sentence in two languages, split it into words, and try to match each word with its translation. Repeat this operation about 500 million times, and the machine counts how many times the word das Haus was translated as house, building, construction, etc. Probably most often it was house, so that's what we'll use. Notice that we didn't set any rules or dictionaries. The machine found everything itself, guided by pure statistics and the logic of «people translate it this way, so I will too». That's how statistical translation was born.
The accuracy of these translators turned out to be noticeably higher than all previous ones, and development required no linguists at all. Find more texts — improve the translation.
One problem: how does the machine guess that for das Haus the pair is specifically house, and not any other word in the sentence? The word order is different, so how do we know exactly how to split things up and find the right words?
The guts of statistical translation in Google Translate. Google doesn't just show probabilities, it also computes the reverse statistics.
Answer: no way it does. At the start, the machine considers it equally likely that das Haus translates as any of the words in the given sentence. When it encounters das Haus in other sentences, the count of das Haus being translated as house starts to increase each time. This is called the "word alignment algorithm" (word alignment). A typical machine learning task, the kind solved in universities.
The machine needs millions upon millions of sentences in two languages to gather statistics for every word. And where do you get that many? Well, the European Parliament and the UN Security Council keep minutes of their sessions in the languages of all member states, so let's take those. Nowadays they're even open for download: the UN Corpora and the Europarl Corpora.
The first statistical translation systems again started by splitting into words. It seemed logical and simple. The first statistical translation model invented at IBM was called IBM Model 1. Elegant, right? Guess what they called the second one?
The classic approach — split everything into words and count statistics. No accounting for order or rearrangement whatsoever. Among its tricks, Model 1 could only translate one word into several. Der Staubsauger (vacuum cleaner) easily turned into Vacuum Cleaner, but going the other way was hit or miss.
You can find simple implementations in Python: shawa/IBM-Model-1.
The lack of knowledge about word order in languages was a problem for Model 1. In some languages it's very important. So in Model 2 they started remembering the position a word appears at in the translated sentence. They added an intermediate step — after translation, the machine tried to rearrange the words the way it thought would sound more natural.
It got better, but still terrible.
New words that weren't in the original text often appear during translation. In German, articles suddenly pop up out of nowhere; in English, the verb do gets inserted wherever it pleases. «Я не хочу хурмы» → «I do not want persimmons. To solve this problem, Model 3 added two intermediate steps:
Model 2 accounted for word order in the sentence, but knew nothing about words swapping places with each other. Often when translating you need to, say, swap a noun and an adjective. No matter how much you remember their order across the whole sentence, it won't get any better. So in Model 4 they started accounting for what's called "relative order" as well. If two words kept swapping with each other during translation, the model remembered that.
Nothing particularly new. Model 5 added more training parameters and fixed issues where two words were competing for the same spot in the sentence.
Despite all its revolutionary nature, Word-based systems still couldn't do anything about cases, gender and homonymy. Each word was translated in a single way that the system considered correct. Such systems aren't used anymore nowadays, they've been replaced by a more advanced method — phrase-based translation.
I have a favorite joke about homonymy:
— Is that your Jaguar parked by the entrance?
— Yes
— Should I finish my drink? (a pun: «Ягуар», "Jaguar", sounds almost like «я допью», "I'll finish my drink")
It took as its basis all the principles of word-based translation: statistics, reordering and lexical hacks. But for training it split the text not just into words, but into whole phrases. More precisely, N-grams, or phrasemes — overlapping sets of N consecutive words. The machine learned to translate fixed word combinations, which noticeably improved accuracy.
The trick of the method was that «phrases» weren't always the syntactic constructions we're familiar with from school. As soon as someone who knew about linguistics and sentence structure tried to intervene in the translation, quality dropped sharply. Frederick Jelinek, a pioneer of computational linguistics, once joked about this: «Every time I fire a linguist, the performance of the recognizer improves.»
Besides improving accuracy, phrase-based translation gave more freedom in finding bilingual texts for training. For Word-based translation, exact correspondence between translations was very important, which ruled out any literary or free translations. Phrase-based translation trained beautifully even on those. Many people even started parsing news sites in different languages and using those texts to improve translation.
Starting in 2006, everyone began using this approach. Google Translate, Yandex, Bing and other quality online translators worked exactly as Phrase-based systems all the way up until 2016. Every one of you can probably recall an experience where Google translated one sentence perfectly, rearranging the words in a literary way, and on another started spouting complete nonsense. That's a peculiarity of phrase-based translation.
If the good old Rule-based approach reliably gave a predictable, if terrible, result, statistical methods sometimes surprised and puzzled people. You can recall a dozen jokes about Google Translate, back when it translated «three hundred» as «300» without batting an eye. This glitch was called a statistical anomaly.
Phrase-based translation became so popular that when you hear "statistical machine translation," it most likely refers to exactly this method. Right up until 2016, in every study Phrase-based translation was praised as the state-of-art. Back then, no one even suspected that in Google's labs, truckloads of neural networks were already being brought in to once again change our understanding of machine translation.
This method is worth a brief mention too. Before neural networks came along, syntax-based translation was talked about for many years as "the future of translators," but it never managed to achieve success.
Adherents of syntax-based translation believed in combining the SMT approach with the old rule-based transfer translation. You need to learn to parse a sentence's syntax accurately enough — determine the subject, predicate, dependent members and all of that — and then build a tree. With such a tree, you can train the machine to correctly convert the structures of one language into the structures of another, doing the rest of the translation by word or phrase. Except now this isn't done by hand, but by machine learning. In theory, this would solve the word-order problem once and for all.
The example is taken from Yamada and Knight [2001] and these excellent slides.
The problem is that although humanity considers the syntactic parsing problem to have long been solved (ready-made libraries exist for many languages), in practice it works pretty poorly. I personally tried many times to use syntax trees for tasks more complex than extracting the subject and predicate, and every time I gave up in favor of other methods.
If you've had even one successful experience with them, tell us about it in the comments.
In 2014, a paper comes out with a brief description of the idea of applying deep learning neural networks to machine translation. In the wider internet, nobody noticed it at all, but in Google's labs they started digging in actively. Two years later, in November 2016, an announcement appears on Google's blog that changed the game.
The idea was similar to style transfer between photos. Remember apps like Prisma that processed photos in the style of a famous artist? There was no special magic there — a neural network was trained to recognize the artist's paintings, and then the last layers, where it makes its decision, were "torn off." The resulting guts, essentially the network's intermediate representation, turned out to be that very stylized image. That's how it sees things, and it looks pretty to us.
If we can use a neural network to transfer style onto a photo, what if we tried to overlay a different language onto our text in a similar way? Imagine the language of the text as that very «artist's style», and try to transfer it while preserving the essence of the image (that is, the essence of the text).
Imagine that I describe to you in words what my dog looks like: medium size, sharp nose, big ears, short tail and barks constantly. I hand you a set of characteristics of the dog, and with a sufficiently precise description, you could even draw it, even though you've never seen it in person.
Now what if we represent the source text as a set of the same kind of characteristic features? Essentially, encode it so that another neural network — the decoder — can then decipher it back into text, but in a different language. We specifically train the decoder to know only its own language. It has no idea where the characteristics came from, but it knows how to express them, say, in Spanish. Continuing the analogy: what does it matter to you what you draw the dog I described with — pencils, watercolors, or a finger in the mud. You draw with what you know how to use.
Once again: the first neural network only knows how to encode a sentence into a set of number-characteristics, and the second only knows how to decode them back into text. Neither has any idea about the other, each only knows its own language. Does this remind you of anything? The idea of interlingua is back. Ta-da.
But how do you find these characteristics? With a dog it's all clear, it has paws and other body parts, but what about with texts? 30 years ago scientists already tried to craft a universal language code, and it ended in complete failure.
But now we have deep learning, which is exactly what handles this! The main difference between deep learning and classical neural networks was precisely that its networks learn to find the characteristic properties of objects without understanding their nature. Given a big enough neural network and a couple thousand video cards stashed away, you can try to find such characteristics in text too!
Theoretically, the characteristics obtained by the neural networks could then be handed to linguists and they'd discover a lot of new things. Yandex talked about this at some point.
The only question is what type of neural network to use in the encoder and decoder. For images, convolutional neural networks (CNN) work great, because they work with independent blocks of pixels. But in text there are no independent blocks, every next word depends on the previous ones and even the following ones. Text, speech and music are always sequential. Recurrent neural networks (RNN) are better suited for processing them, since they remember the previous result. In our case, that's the previous words in the sentence.
RNNs are used in a lot of places today: speech recognition in Siri (we parse a sequence of sounds where each one depends on the previous one), word suggestions on the keyboard (we remember previous words and guess the next one), music generation, even chatbots.
For nerds like me: in reality, the architectures of neural translators vary quite a bit. At first researchers used plain RNNs, then switched to bidirectional ones — the translator took into account not only the words before, but also after the word in question. That was much more effective. Then they went full hardcore, using multi-layer RNNs with LSTM cells for storing translation context over long stretches.
In just two years, neural networks surpassed everything that had been invented in translation over the previous 20 years. Neural translation made 50% fewer word-order errors, 17% fewer lexical errors, and 19% fewer grammatical errors. Neural networks even learned on their own to agree gender and case across different languages, without anyone teaching them to.
The most noticeable improvements were where a direct translation had never existed before. Statistical translation methods always worked through English. If you were translating, say, from Russian to German, the machine would first convert the text into English, and only then translate it into German. Double the losses. Neural translation doesn't need that — plug in any decoder and go. For the first time it became possible to translate directly between languages that didn't share a single dictionary.
In 2016, Google turned on neural translation between nine languages, and Russian was added in 2017. Google developed its own system with the unassuming name Google Neural Machine Translation (GNMT), consisting of an 8-layer RNN on the input side and another one just like it on the output side, plus a context-matching system called the Attention Model.
During training, they didn't just split sentences into phrases and words, they even split the words themselves into parts. With this they were trying to solve one of the main problems of NMT — it's helpless when a word isn't in its vocabulary. For example, «Вастрик» (Vastrik). It's unlikely anyone trained the neural network to translate my nickname. In this case GNMT tries to break it into parts and glue together a translation from them. Clever.
Hint: the Google Translate that translates websites in your browser still uses the old Phrase-based algorithm. For some reason Google hasn't updated it, and the differences compared to the online version are very noticeable.
In the online version of Google Translate, they also added a crowdsourcing mechanism for translations. Users can now pick the version of the translation they consider most correct, and if enough people like it, Google will always translate that phrase that way from then on, marking it with a special icon. It works really well on short everyday phrases like «пойдем на обед» ("let's go to lunch") or «буду ждать внизу» ("I'll be waiting downstairs"). Google knows conversational English better than I do :(
Microsoft's Bing Translator works as a complete copy of Google Translate. Yandex, on the other hand, is different.
Yandex launched its own neural translation in 2017. They claimed hybridity as the main difference. Yandex's translator translates a sentence using two methods at once — statistical and neural — and then, using their favorite algorithm, CatBoost, finds the most suitable one.
The thing is, neural translation handles short phrases poorly. When you need to translate a phrase like «сиреневая бетономешалка» ("lilac concrete mixer"), neural networks can make up extra nonsense, while plain statistical translation finds both words dumbly, quickly and without any problems.
Yandex doesn't share any other details with us, fending us off with non-technical press releases. WELP, ALRIGHT THEN.
Apparently Google also uses SMT for translating words and short phrases. They don't mention this in their articles, but it's obvious from the difference between translations of short strings and long ones. SMT is also clearly used for displaying word statistics.
Hybrid translation technology involves using statistical methods to automatically build dictionary databases based on parallel corpora, forming several possible translations both at the lexical level and at the level of the target-language sentence's syntactic structure, applying automatic post-editing, and selecting the best (most probable) translation among the possible ones based on a language model built from a specific corpus of the target language.
Hybrid (SMT + RBMT) System differs in: (sec. 2.4.3 )
Statistical MT strives to make use of linguistic data, while "classical" rule-based systems apply statistical methods. Adding some "cross-cutting" rules, that is, creating hybrid systems, somewhat[how much?] improves translation quality, especially with an insufficient volume of input data used when building the index files that store the linguistic information of an N-gram-based machine translator.[10]
Combining RBMT and statistical machine translation:
Stages of the Hybrid SMT and RBMT technology:
Everyone is still excited by the idea of the «Babel Fish» — real-time simultaneous speech translation. Google took a step in this direction when it announced Pixel Buds, but on closer inspection it all turned out badly. Real-time simultaneous translation differs from ordinary translation, since you need to know when to start translating and when to just sit and listen. I haven't come across an approach that solves this problem yet.
upd: I got told off in the comments for not mentioning Skype's real-time translation. Correcting that now.
Here's another unplowed field, in my opinion: all training still runs up against a limited set of parallel text corpora. Even the much-hyped, deep neural networks still get trained specifically on parallel texts. We can't train a neural network without giving it the original. But a human can, starting from a certain level of language knowledge, expand their vocabulary just by reading books or articles, without even translating them into their native language.
If a human can do it, then in theory a neural network can too.
Comments