Lecture
After a model has been built, it is necessary to make sure of its quality. For this purpose, the model is checked for adequacy with respect to the process, object, or phenomenon for which it was built.
Checking the adequacy of a model means establishing how well the model describes the real processes occurring in the system and how good its predictions of the development of these processes will be. Adequacy testing is carried out on the basis of certain experimental information obtained during the operation of the system or through a specially conducted experiment in the course of which the processes of interest are observed.
Checking adequacy consists in proving that the accuracy of the results obtained from the model will be no worse than the accuracy of calculations made on the basis of experimental data.
The main method for assessing the adequacy of a model is regression analysis, which is based on the study of the linear regression equation.
Let us consider the basic concepts of regression analysis [26–28]. To check the significance of a regression equation means to establish
whether the mathematical model expressing the relationship between the variables corresponds to the experimental data, and whether the explanatory variables included in the equation (one or several) are sufficient to describe the dependent variable. To check significance, a null hypothesis about the reliability of the parameters is put forward.
A statistical hypothesis is an assumption about the properties of random variables or events that requires verification against the available data.
The null hypothesis H0 is the main hypothesis being tested,
which is usually formulated as the absence of differences, the absence of the influence of a factor, the absence of an effect, the equality of sample characteristics to zero, and so on. Another hypothesis being tested (not always strictly opposite to or the reverse of the first) is called the competing (or alternative) hypothesis.
A hypothesis that has been put forward may be correct or incorrect, which is why it needs to be tested. Since the testing is carried out using statistical methods, it is called statistical.
When testing statistical hypotheses, two types of errors (erroneous judgments) are possible:
•the null hypothesis may be rejected when it is in fact true (the so-called Type I error);
•the null hypothesis may be accepted when it is in fact false (the so-called Type II error).
The permissible probability of a Type I error may be equal to
5 or 1% (0.05 or 0.01).
The significance level is the probability of a Type I error when making a decision (the probability of erroneously rejecting the null hypothesis). Alternative hypotheses are accepted if and only if the null hypothesis is refuted. This occurs in cases where the differences in the arithmetic means of the experimental and control groups are so significant (statistically reliable) that the risk
71
of erroneously rejecting the null hypothesis and accepting the alternative one does not exceed one of the three accepted significance levels for a statistical conclusion:
Level 1 – 5% ( α = 0.05 ), where the risk of an erroneous conclusion is allowed in five cases out of a hundred theoretically possible such experiments with strictly random sampling for each experiment;
Level 2 – 1% ( α = 0.01 ), i.e., the risk of error is allowed only in one case out of a hundred;
Level 3 – 0.1% ( α = 0.001 ), i.e., the risk of error is allowed only in one case out of a thousand.
The test statistic is a certain function of the initial data, the value of which is used to test the null hypothesis. Most often, the test statistic is a numerical function.
Any rule on the basis of which the null hypothesis is rejected or accepted is called the criterion for testing this hypothesis.
A statistical criterion is a random variable used for testing statistical hypotheses.
The critical region is the set of criterion values for which the null hypothesis is rejected. The acceptance region of the null hypothesis (the region of permissible values) is the set of criterion values for which the null hypothesis is accepted. If the null hypothesis is true, the probability that the test statistic falls into the acceptance region of the null hypothesis should be equal to 1.
The procedure for testing the null hypothesis generally includes the following steps:
• the permissible probability of a Type I error is set
( α = 0.05 );
•a test statistic is chosen;
•the region of permissible values is found;
•the value of the statistic is calculated from the initial data;
•if the test statistic falls within the acceptance region of the null hypothesis, then the null hypothesis is accepted (more precisely, a conclusion is drawn that the initial data do not contradict the null hypothesis); otherwise the null hypothesis is rejected and the alternative hypothesis is accepted. This is the basic principle for testing all statistical hypotheses.
72
Analysis of the quality of an identification model based on a linear regression equation begins with computing (identifying the parameters of) this equation. The first regression equation constructed from the sample is very rarely satisfactory in terms of one characteristic or another, so the next most important assessment is checking the quality of the regression equation. In regression analysis, the following scheme is adopted for such a check, carried out in the following directions:
•checking the overall quality of the regression equation (checking the adequacy of the model);
•checking the significance of the multiple correlation coeffi-
cient;
•checking the statistical significance of the coefficients of the regression equation.
When analyzing the adequacy of the regression equation (model) to the process under study, the following variants are possible:
1.The model built is, according to Fisher's F-test, adequate overall, and all regression coefficients are significant. Such a model can be used for decision-making and for making forecasts.
2.The model is adequate according to Fisher's F-test, but some of the coefficients are not significant. The model is suitable for making certain decisions, but not for forecasts.
3.The model is adequate according to the F-test, but none of the regression coefficients are significant. The model is considered completely inadequate. It cannot be used as a basis for decisions or forecasts.
Checking the significance (quality) of a regression equation means establishing whether the mathematical model expressing the relationship between the variables corresponds to the experimental data and whether the explanatory variables included in the equation (one or several) are sufficient to describe the dependent variable.
The overall significance of the regression equation is assessed on the basis of Fisher's F-test, which is preceded by analysis of variance. In mathematical statistics, analysis of variance is regarded as an independent tool of statistical analysis.
73
Suppose the adequacy of a one-dimensional linear model is being assessed:
k
ym = ∑aiui ,
(2.3)
i=1
where ym
– the model's output variable; k
– the number of model
parameters;
ai – the model parameters being estimated; ui – the object's input
variables.
To perform the regression analysis, the following statistical characteristics are determined:
• the sum of squares due to regression:
N
QR = ∑( ymi − y)2 ,
(2.4)
i=1
1
N
where y is the mean value of the output variables, y =
∑yi ; N is the num-
N i=1
ber of observations.
The sum of squares due to regression QR
shows that
the quantity ymi participating in it determines k linear relations between the observations y1, y2 ,K, yN , since it includes k estimates of the coefficients a1, a2 ,K, ak , determined from the same observations. In addition, y determines one linear relation between them;
• the number of degrees of freedom for QR :
νR = k −1;
(2.5)
• the residual sum of squares:
N
Qres = ∑( yi − ymi )2 ,
(2.6)
i=1
where yi is the value of the object's output variable.
The residual sum of squares Qres reflects the influence of all those causes of scatter in the results y that cannot be explained by the regression;
• the number of degrees of freedom for Qres :
νres = N −k;
(2.7)
74
• the total (overall) sum of squares:
N
Q = ∑( yi − y)2 ;
(2.8)
i=1
• the number of degrees of freedom for Q :
ν = N −1.
(2.9)
The following relation holds for the sums of squares above:
Q = QR +Qres.
(2.10)
According to the basic idea of analysis of variance, the total sum
of squared deviations Q of the variable
y from the mean value
y is de-
composed into two parts – the "explained" part QR and the "unexplained"
part Qres :
N
N
N
∑(yi − y)2
= ∑(ymi − y)2
+∑(yi − ymi )2.
(2.11)
i=1
i=1
i=1
For the degrees of freedom:
ν = νR +νres = k −1+ N −k = N −1.
(2.12)
To analyze adequacy, the following variance estimates are used:
• variance due to regression:
SR2 =
QR
;
(2.13)
ν
R
• residual variance:
Sres2 =
Qres
;
(2.14)
ν
res
• total (overall) variance:
S 2 =
Q.
(2.15)
ν
The scheme of analysis of variance is presented in Table 2.1.
75
Table 2.1
Source
Sum of squares
Number of degrees
Variance
of scatter
of freedom
estimate
Regression
QR
νR = k −1
SR2 =
QR
ν
R
Residual
Qres
νres = N −k
Sres2 =
Qres
ν
res
Total
Q
ν = N −1
S 2 =
Q
ν
The value of Fisher's F-test for determining the statistical significance of the equation, or overall adequacy, is defined as a comparison of the regression variance and the residual variance per degree of freedom:
S2
1
F =
R
.
(2.16)
Sres2
N −2
To evaluate the H0 hypothesis about the overall statistical significance of the equation, the tabulated value of Fisher's Ft-criterion is chosen as Ft(αν, 1,ν2 ), where α is the specified significance level; ν1 = νR ;
ν2 = νres.
If the actual value of the F-test is greater than the tabulated
Ft (F > Ft), then the statistical significance of the equation as a whole is
recognized.
Otherwise (F < FT ) the hypothesis of the statistical significance
of the regression equation is rejected. This may mean the following: the structure of the model was chosen incorrectly, namely, the chosen model contains fewer or more coefficients than the actual one. In this case, another model structure must be chosen and its adequacy assessed.
If the hypothesis of overall statistical significance of the regression analysis is confirmed, it is then necessary to assess the significance of the multiple correlation coefficient.
76
The significance of the multiple correlation coefficient determines how well the chosen model describes the response as a function of the factor.
Suppose the regression model is represented by a 2nd-order polynomial:
y
m
= a
+ a x +a
2
x2.
(2.17)
0
1
Figures 2.1 and 2.2 show the results of the experiment.
ym
y y
x
a
ym
y
y
x
b
Fig. 2.1. Experimental plot for a one-dimensional model:
a– with significant coefficients ai (i ≠ 0);
b– with insignificant coefficients ai (i ≠ 0)
In Fig. 2.1, a, the dependence of ym as a function of x is clearly expressed. In Fig. 2.1, b, the plot of ym differs very little from the mean value of all the results y, so it can be assumed that in reality ym does not depend on x, i.e., the coefficients of the equation a1, a2 turned out to differ from zero only due to random disturbance.
77
To assess the significance of the multiple correlation coefficient, the hypothesis H0 : a1 = 0, a2 = 0 is put forward, and the coefficient of
determination R2 is introduced:
R2 =
QR
.
(2.18)
Q
The significance of the multiple correlation coefficient is checked according to the following rules:
•the coefficient of determination R2 is computed;
•Fisher's F-test is computed to determine the significance of the multiple correlation coefficient:
F =
R2
(N −k)
;
(2.19)
(1
− R2 )(k −1)
• the tabulated Fisher's F-test is computed to determine the significance of the multiple correlation coefficient. Fisher's Ft-criterion is chosen as Ft(α,ν1,ν2 ), where α is the specified significance
level; ν1 = νR ; ν2 = νres.
• a comparison is made (F > Ft).
If the actual value of the F-test is greater than the tabulated Ft (F > Ft), then the multiple correlation coefficient is significant, and its values cannot be explained by random disturbances alone.
Otherwise (F < Ft) the resulting coefficient is not signifi-
cant.
The insignificance of the multiple correlation coefficient means that the dependence of y on the input actions xi is weak or
absent altogether. This is possible for two reasons:
1) The model did not include some of the strongly influential factors. Their influence showed up in the residual sum Qres, whereas the regression reflected only some minor influences and the sum QR turned out to be quite small. Such a result often occurs
when studying complex objects in which only some of the factors (variables/coordinates) are available for measurement, while some of the most important independent variables remain outside the experimenter's field of view.
78
2) An insignificant multiple correlation coefficient can also result when all the essential factors have been included in the model, but its structure was chosen incorrectly. Such a result could occur, for example, when attempting to describe with a 1st-order polynomial an object that is substantially nonlinear with respect to the factors.
The insignificance of the multiple correlation coefficient is sufficient grounds for abandoning the chosen model.
After testing the hypotheses about the statistical significance of the linear regression equation and the significance of the multiple correlation coefficient, the task arises of assessing the adequacy of the individual (most important) coefficients of the model.
The significance of an individual model coefficient ai is checked according to the following rules:
• l additional trials are carried out at a certain operating point, and the variance of the additional trials is determined:
Sl2 =
Ql
,
(2.20)
ν
l
l
where Ql = ∑(yi − y), νl = l −1;
i=1
•the information matrix is determined from the basic observa-
tions XX T , and the coefficient c
as the i-th diagonal element of the ma-
ii
trix XX T ;
• the coefficient ti is computed:
ti =
ai
;
(2.21)
Sl cii
•the tabulated tt-criterion is chosen from the Student's distribution table as tt(α,ν), where α is the specified significance level; ν = νl ;
•a comparison is made (t > tt).
If
the actual value
of the t-test is greater than the tabulated
tt (t > tt),
then the model coefficient ai
is significant.
79
Otherwise (t > tt) the resulting coefficient ai is not signifi-
cant.
After testing the significance of the model coefficients, insignificant coefficients can be dropped, which considerably simplifies the model.
Comments