15: EXPERT EVALUATION USING RANKING METHOD

1. Expert evaluation ranking

If the result of the quality assessment experts

put in the form of a ranked series, the numerical definition

The final numerical quality estimates are as follows:

1) all objects of evaluation (products, properties) numbering

arbitrary;

2) experts rank objects on a scale of order;

3) ranked rows of objects compiled by experts

tami are matched;

4) determine the sum of the ranks of each of the objects ex-

pertnoy assessment;

5) on the basis of the obtained amounts of ranks build a

puppy ranked series;

6) generalized expert assessments of the quality of

objects of expertise, i.e. their weight coefficients,

calculated by the formula:

where n is the number of experts;

m is the number of estimated indicators;

Qi, j is the weight coefficient of the jth index in ranks (points), which

the second was given by the i- th expert

2. Features of the method

For example, let five experts about seven objects of expertise

Threat Q made up such ranked series in increasing

scale order:

Expert No. 1 - Q 5 < Q 3 < Q 2 < Q 1 < Q 6 < Q 4 < Q 7;

Expert No. 2 - Q 5 < Q 3 < Q 2 < Q 6 < Q 4 < Q 1 < Q 7;

Expert No. 3 - Q 3 < Q 2 < Q 5 < Q 1 < Q 6 < Q 4 < Q 7,

Expert No. 4 - Q 5 < Q 3 < Q 2 < Q 1 < Q 4 < Q 6 < Q 7;

Expert No. 5 - Q 5 < Q 3 < Q 1 < Q 2 < Q 6 < Q 4 < Q 7.

The place of an object in a ranked row is called its ranked

gom The numerical value of the rank in a series of increasing scale

row increases from 1 to m ( m is the number of estimated volumes

ektov). In this example, m = 7.

In this example, the sum of the ranks of each of the

The peer reviews are:

Q1 - 4 + 6 + 4 + 4 + 3 = 21;

Q2 - 3 + 3 + 2 + 3 + 4 = 15;

Q3 - 2 + 2 + 1 + 2 + 2 = 9;

Q4 - 6 + 5 + 6 + 5 + 6 = 28;

Q5 - 1 + 1 + 3 + 1 + 1 = 7;

Q6 - 5 + 4 + 5 + 6 + 5 = 25;

Q7 - 7 + 7 + 7 + 7 + 7 = 35.

The ranked series obtained by all the expert groups

Py, has the form:

Q5

Calculations according to the formula for the considered example give

following results:

Analyzing the estimates obtained by the expert method

not only which object is better or worse

others, but also how

3. The coefficient of concordance

The accuracy of expert assessments is determined by

expert opinions. The degree of coincidence of expert ratings,

included in the commission, characterizes the quality of the examination and

Razhaetsya coefficient of concordance:

where S is the sum of the squares of the deviations of the ranks or points of each object

that of the arithmetic mean value;

n is the number of experts;

m is the number of evaluated objects.

The sum of the squares of the deviations of the ranks ( S ) from the average

their value ( Рсp ) for all objects and experts located

given by the formula:

where Q ij is the rank score given to the i -th object by the j -th expert;

Qcp is the arithmetic mean of the ranks.

The complete record of the concordance coefficient formula has

following view:

At W = 0 - absolute inconsistency, and at W = 1.0 -

complete concurrence of opinions (ratings). Therefore, 0  W  1.

With expert assessment methods in which ranks end

are not determined, for finding the coefficient of

assignments calculated values ​​of objects should be translated

in ranks by ranking them. Otherwise, the estimate of

penalties for the consistency of expert opinions are held by others

criteria.

In the example considered here, the arithmetic average

value of ranks Q av equals:

The sum of the squares of deviations from the arithmetic mean

rank values

S = 12 + 52 + 112 + 132 + 52 + 152 + 82 = 630.

Consequently, the coefficient of concordance in this case

Increase the accuracy of expert estimates of quality indicators

You can, if you make a two-time comparison and estimate

objects, i.e. first do it in one consistently

STI, and then in the opposite