Lecture
Stop words, or noise words — in the theory of keyword-based information retrieval, these are the words with the least meaning and importance.
In contrast, keywords carry the greatest semantic meaning and weight.
In computing, stop words are words that are filtered out before or after the natural language processing of data (text). Although «stop words» usually refers to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools, and indeed not all tools even use such a list. Some tools deliberately avoid removing these stop words to support phrase search.
Stop words can be divided into general and dependent.
General stop words include articles, prepositions, suffixes, participles, interjections, numerals, particles, and so on. General noise words are always excluded from the search query (except when searching for an exact match of the search phrase), and they are also ignored when building the inverted index. Each of the general stop words is considered to be present in almost all documents of the collection.
Dependent stop words depend on the search phrase. The idea is to treat the absence of ordinary query words differently from the absence of dependent stop words from the query in a found document.
For example, when searching for the query Pushkin Alexander Sergeyevich, it makes sense to display all documents containing:
But it hardly makes sense to display documents containing only:
That is, in this query the noise words are Alexander and Sergeyevich.
Dependent stop words differ in that in a search query they should be taken into account only when the document being searched also contains meaningful keywords.
Hans Peter Luhn, one of the pioneers in the field of information retrieval, coined the phrase and used the concept. The phrase «stop word», which does not appear in Luhn's 1959 presentation, and the related terms «stop list» and «stoplist» appear in the literature shortly thereafter.
A preceding concept was used in the creation of some concordances. For example, the first Hebrew concordance, Me'ir nativ, contained a one-page list of unindexed words with insignificant prepositions and conjunctions that resemble modern stop words.
Any group of words can be chosen as stop words for a given purpose. For some search engines, these are among the most common, short function words, such as is, on, which, and at. In this case, stop words can cause problems when searching for phrases containing them, especially in names like «Who», «That», or «Take That». Other search engines remove some of the most common words from the query, including content words such as «want», in order to improve performance.
Hans Peter Luhn, one of the pioneers in the field of information retrieval, coined the phrase and used the concept. The phrase «stop word», which does not appear in Luhn's 1959 presentation, and the related terms «stop list» and «stop list» appear in the literature shortly thereafter.
A preceding concept was used in the creation of some concordances. For example, the first Hebrew concordance, Me'ir nativ, contained a one-page list of unindexed words with insignificant prepositions and conjunctions that resemble modern stop words.
In SEO terminology, stop words are the most common words that most search engines avoid in order to save space and time when processing large amounts of data during crawling or indexing. This helps search engines save space in their databases.
этом (this), этот (this), эти (these), со (with), типа (like), это (this), их (their), каким (which), какая (what), какие (what), имеет (has), есть (is), быть (to be), те (those), такие (such), таким (such), такая (such), так (so), то (that), в (in), без (without), ' .
как (as), до (before), из (from), к (to), на (on), является (is), по (by), о (about), от (from), перед (before), при (with), через (through), или (or), он (he), я (I), ты (you), б,
'ты, ты, ты, ты, ты, с (with), у (at), и (and), нет (no), за (for), над (over), для (for), об (about), под (under), про (about), лишь (only), том (that), оно (it), они (they), тем (that), если (if)
a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will,with
To determine the least important word for the meaning of a sentence and assign a number to each word based on its importance in the sentence, various methods are used.
By "importance" is meant the following: if you removed this word from the sentence, it would have little effect on the meaning (low importance) or a large effect on the meaning (high importance).
In this case, it makes sense to use a POS tagger. It lets you tag sentences with their parts of speech (nouns, verbs, adjectives, etc.) - POS Tag NLTK (http://www.nltk.org/book/ch05.html) . Then you can write your own rules to extract only the parts of speech you are interested in. In corpus linguistics, part-of-speech tagging ( POS-tagging or PoS-tagging or POST ), also called grammatical tagging or word-category disambiguation , is the process of marking a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition and its context, i.e. its relationship to adjacent and related words in a phrase, sentence, or paragraph. A simplified form of this is usually taught to school-age children when recognizing words as nouns, verbs, adjectives, adverbs, etc.
Removing stop-words is another option
Extracting key words can be done in different ways -
Chunking
Collocation extraction
Named entity recognition
Constructing CFGs and parsing trees
Relation extraction
Filler words emerge from ordinary words in the text due to meaningless, frequent and inappropriate use.
As a rule, words with weakened semantic content become filler words. Namely: «Типа» (like), «Как бы» (as if), «Это самое» (that thing),«Видите ли… (you see...) «Значит» (so), «Собственно» (actually), «Вот» (well), «Понимаешь» (you know), uh, «Как говорится» (as they say), «Короче» (in short), «Прикинь» (imagine), «Допустим» (suppose), «Стало быть» (so), «Ну» (well)… мэ-э-э, Ну, как вам сказать…» (umm, well, how shall I put it...),
If words are used as intended, they are useful and necessary. Many examples of the «correct» use of words can be found in fiction and popular-science articles, where words carry semantic weight (convey emotions, feelings, and reflect the distinctive, characteristic features of the objects and phenomena being described). Compare:
Filler words most often play the role of parenthetical words; they can be omitted without any loss of meaning
Examples of words classified as filler words:
Sometimes, instead of filler words, the text uses a replacement with a more «eloquent» variant; however, these too are words with reduced semantic content — examples of such replacements are given below.
Прикинь — imagine that.
Ничего себе! Да ладно! — You won't believe it! Amazing! Incredible!
Таким образом — this means that...
Прикол — very cool, well thought out.
Не вопрос, без проблем — it's no trouble for me.
Короче — in a nutshell; to put it briefly, then...
На фиг! на хрен! — not needed, there's no need for this, we can do without it;
На фиг? На хрена? — why? what for?
Типа того — very similar, agreed.
В принципе — agreed, except..., I consider it correct, but...
The following words and expressions should be omitted without replacement: а-а-а, э-э-э, блин, ну, вот, как его, это самое, ешкин кот, ну это.
Comments