Lecture
At ROMIP-2006, Yandex representatives lifted the veil of secrecy a little on the text ranking factors they used:
Where:
When counting the number of occurrences of a word in a document, we perform preliminary lemmatization of the query words and the document words. Using the maximum TF among all lemmas of the document as a measure of document length worsens the result.
In addition to counting the number of words in the document, html formatting and the position of the word in the document can be taken into account. We account for this as a separate term. It takes into account whether the word appears in the first sentence, in the second sentence, or inside highlighting html tags.
A pair is counted when the query words occur in the text in a row (+1), with one word in between (+0.5), or in reverse order (+0.5). Plus there is a special case, where words that are one apart in the query occur in a row in the text (+0.1).
As of 2010, according to Den Raskovalov, Yandex took into account more than 420 ranking factors, and the ones given above are only a small part of them.
The ranking algorithm performs lemmatization of the words in the document and the query, so it does not matter in which form the word or its synonyms are used (different forms will be counted as the same lemma). For the query, the following string was used: «купить айофн с доставкой Бирюлево» ("buy an ayofn with delivery to Biryulyovo"; here and below, quotation marks are used to set the words apart from the surrounding text — they were not part of the actual query). Let's look at how the formula works for our example, and what the parameters are for the first 4 sites in Yandex's results.
The relevance of a document to a query is calculated using the formula:
The original article also used an additional term, Wprf — for the similarity of the document to the documents at the top of the results, but we will not take it into account for now. In our calculations, the coefficients k1, k2, k3 — are also unknown to us, so let's assume they are equal to 1.
1. Accounting for individual words:
TF — the frequency of occurrence of a word in the document ( = number_of_word_occurrences / document_length). The frequency values for the query words, for each of the positions in the top results:
DocLength — the length of the document in words;
Hdr — the sum of the word's weights for formatting. According to the article's authors:
It takes into account whether the word appears in the first sentence, in the second sentence, or inside highlighting html tags.
But the authors did not specify exactly what numbers are used (in our calculations we will take this parameter to be 0).
D — the number of documents in the collection. To get a specific value, we can search for the phrase "intellect.icu" using Yandex. The number of documents returned by this search will be taken as the number of documents in the collection. In our case, 2325 million documents were returned.
CF — the number of occurrences of a word in the document collection (the number of documents in which the word occurred at least once). The specific numbers for the words from our query turned out to be as follows:
We calculated Wsingle for each word and added their sum to the Score.
2. Accounting for word pairs:
p1, p2 — calculated the same way as for Wsingle; TF — the number of occurrences of the word pair, taking weights into account. A pair is counted when the query words occur in the text in a row (+1), with one word in between (+0.5), or in reverse order (+0.5). Plus there is a special case, where words that are one apart in the query occur in a row in the text (+0.1).
The remaining pairs did not occur in the documents.
Accounting for the occurrence of three or more query words in the document gave no improvement in our experiments
3. Accounting for all words:
Nmiss — the number of words that did not occur in the document. For the sites under consideration, this number turned out to be the same.
4. Accounting for the whole query:
TF — the number of occurrences of the whole query, divided by the document length. For the top results, the value came out to 0 for all sites.
5. Accounting for part of the query:
This term was counted when the sum of the idf of the query words in a sentence (in the formula — the sum of the logarithms) was greater than half the sum of the idf of all the query words. Here, TF is the number of sentences counted in the text, divided by the number of sentences in the document. For our top 4, this again came out to 0, since there were no such sentences there. For comparison, suppose we created a document and want to calculate its relevance for the query. Let its numeric values be as follows:
Let the remaining values be the same as for the top four. Let's gather the final numbers into a table and see what we got. Table of the resulting values for our example:
So we can see that the relevance score of the created page turned out to be even better than that of the first-place site. This can be explained by the following factors:
I. Frequently used words weakly influence relevance, or are ignored entirely. The more often a word from the query appears in the documents of the collection, the less informative it is — the smaller its contribution to relevance.
If a word occurs in only one document out of the entire collection, then its contribution to relevance will be greatest. If a word occurs in every document 1 or more times, then the contribution of such a word to relevance will equal 0. At the same time, the more documents there are in the collection, the greater the number of occurrences that is acceptable (the number of occurrences is directly proportional to the number of documents), while maintaining the same relevance. If the number of documents doubles, relevance will be preserved if the frequency of occurrence also doubles. That is why it is worth focusing attention only on words that have relatively low frequency, and using the same word in as few documents as possible. Keywords should be highlighted using formatting. Use them as close to the beginning of the document as possible.
II. The same keyword should not be used many times in the text of a document. What matters is the mere fact of its presence. The dependence of relevance on word frequency is non-linear in nature. The general principle that the more often a word occurs in a document, the greater its relevance — is true. However, starting from a certain value, an increase in frequency stops affecting relevance. The graph shows the schematic behavior of the effect of word occurrence frequency on relevance.
Horizontal axis — word frequency. Vertical — relevance
The size of a document should not be too large or too short (preferably the same as the average value of the other documents, or smaller).
The frequency of an individual word should not be high; what matters is the mere fact of its presence. Priority goes to the number of different keywords from the document's subject area — there should be as many of them as possible, but from one specific subject area, to increase the chances of the document being found.
III. Use keywords in word combinations. Keywords should, where possible, form all possible pairwise combinations with each other. Repeating the same pairs should be avoided. That is, as in point II, many different ones are better than many repeated ones. Word combinations longer than 3 words are not worth focusing attention on.
IV. It is desirable that words from potential queries be present in the document. Having various keywords related to the document's subject won't hurt. At the same time, the number of identical words or forms of the same word should be as small as possible.
V. Half the words in sentences — are topical. It is desirable that each sentence contain half of the words from the potential query. The order of the words should be coordinated with point III.
VI. Knowing what the relevance to a specific query should be, we can specify concrete values for the number of occurrences of each word. Taking as the relevance value the score for sites from the top of the search results, one can optimally select the number of occurrences of each individual word. Among the drawbacks of this approach — we become dependent on a specific query or set of queries (though this is only if we don't account for the fact that search engines expand queries)
Comments