Lecture
In computer science, phrase search allows users to retrieve content from information systems (for example, documents from file storage systems, records from databases, and web pages on the Internet) that contains a specific order and combination of words defined by the user.
Phrase search is one of many search operators that are standard in search engine technology, along with Boolean operators (AND, OR, and NOT), truncation operators and wildcards (usually denoted by an asterisk), field code operators (which search for specific words in specific fields, such as the «Author» field in a periodical database), and proximity operators (which search for specific words that appear near each other, if not immediately next to each other, as in phrase search). Search operators are used to refine search results when a simple keyword search returns too many unwanted results.
Although the exact functionality of each search engine is determined by its developers, phrase search is usually performed by enclosing the desired phrase in quotation marks. For example, a search for red applemay return records containing the word «apple», those containing the word «red», and those containing both words, regardless of where they appear in the record (that is, if the search engine applies Boolean OR logic to the keyword search function), whereas a search for "red apple"will return only records containing the phrase «red apple».
Phrase search is one of the most important techniques associated with optimizing the text content of web pages so that it can be found by someone searching for a specific string of text.
Consider a system that indexes documents based on the vector space model and a simple query, for example qwe asd. When searching, we assign weights to both the words qweand asdbased on how often they appear in the index. The idea is that if a word exists in more documents, it should have less influence on relevance. This works well because for each word we usually store statistics: the number of occurrences of the given word in the entire index and the number of occurrences of the given word in each document (possibly after normalization).
Now consider another query: "qwe asd" zxcrty. We have two parts here: a phrase and a simple word. For the word we have the statistics above, but for the phrase we do not. The question arises: how do we compare documents against a phrase search? If we find one document containing the phrase ( qwe asd), and another containing one word ( zxcrty), which one should be ranked higher?
I somehow doubt that there is a single, definitive answer to this question, but I would like to know what approaches are used in existing search engines, whether other models solve the problem, and what other information might be useful in analyzing the problem.
You can use many different approaches. Your question is not limited to the vector space model. Various language models also rely on these concepts. Here are two approaches:
One popular approach is the use of n-grams. This approach involves treating adjacent words as a single word. For example, «big house» would be stored in your controlled vocabulary as «big», «house», and «big house», if you decide to use 2-grams (bigrams). Probabilities would be calculated per n-gram.
Pros: accurately tracks the probabilities of n-gram occurrence, easy to extend from the unigram model (just follow the same ranking functions as for unigrams, such as tf-idf or Okapi BM25).
Cons: exponentially increased memory usage, many bigrams occur only once.
This approach requires storing an inverted index for each document in addition to the inverted index of the entire corpus. This would mean that you do not need to store n-grams. Of course, you would not have such important information as term frequency or document frequency of the n-gram. The idea is that you look up the first word in the n-gram and find it in the document. You compare the following words to see if the n-gram is there.
Pros: lower memory usage (no need to track n-grams in documents), greater flexibility (for example, ignoring irrelevant words).
Cons: less information is stored (no TF, DF, etc.), more space per document is required (for the inverted index), an additional lookup is required, ranking can be more difficult.
If we find one document containing the phrase (qwe asd), and another containing one word (zxcrty), which one should be ranked higher?
This depends entirely on the ranking function and how you decide to weight each thing. How common are qwe and asd? How common is zxcrty? Would you like to give extra weight to larger n-grams? All of this is important to consider.
Comments