Lecture
Linear regression remains the most widely used statistical method in the data science industry and in academia for explaining the relationships between features.
A total of 1355 people registered for this skill test. It was specifically designed for you so that you could check your knowledge of linear regression techniques. If you are one of those who missed this test, here are the questions and solutions. You missed the live test, but you can read this article to find out how many of them could have answered correctly.
Here is the leaderboard of participants who took the test.
Below is the distribution of participants' scores:
You can view the results here. More than 800 people took part in the skill test, and the highest score obtained was 28.
Here are a few resources to gain deeper knowledge on the subject.
5 questions that can teach you multiple regression (with R and Python)
Going deeper into regression analysis with assumptions, plots, and solutions
7 types of regression techniques you should know!
1) True-False: linear regression is a supervised machine learning algorithm.
A) TRUE
B) FALSE
Solution: (A)
Yes, linear regression is a supervised learning algorithm because it uses true labels for training. A supervised learning algorithm must have an input variable (x) and an output variable (Y) for each example.
2) True-False: linear regression is mainly used for regression.
A) TRUE
B) FALSE
Solution: (A)
Linear regression has dependent variables that have continuous values.
3) True-False: can a linear regression algorithm be built using a neural network?
A) TRUE
B) FALSE
Solution: (A)
True. A neural network can be used as a universal approximator, so it can certainly implement a linear regression algorithm.
4) Which of the following methods do we use to find the best-fitting line for the data in linear regression?
A) Least squares method
B) Maximum likelihood
C) Logarithmic loss
D) Both A and B
Solution: (A)
In linear regression, we try to minimize the least squares errors of the model in order to determine the line of best fit.
5) Which of the following evaluation metrics can be used to evaluate a model when modeling a continuous output variable?
A) AUC-ROC
B) Deviation
C) Log-loss
D) Mean squared error
Solution: (D)
Since linear regression produces output in the form of continuous values, in such a case we use the mean squared error metric to evaluate model performance. The other options are used in cases involving a classification problem.
6) True-False: Lasso regularization can be used for variable selection in linear regression.
A) TRUE
B) FALSE
Solution: (A)
True, in the case of lasso regression we apply an absolute penalty, which makes some coefficients zero.
7) Which of the following is true regarding residuals?
A)
The lower, the better B) The higher, the better
C) A or B depending on the situation
D) None of these
Solution: (A)
Residuals refer to the values of the model's errors. Therefore, lower residuals are desirable.
8) Suppose we have N independent variables (X1, X2… Xn), and the dependent variable is Y. Now imagine that you apply linear regression by fitting the line of best fit using least squares error for this data.
You found that the correlation coefficient for one of its variables (say, X1) with Y is -0.95.
Which of the following is true for X1?
A) The relationship between X1 and Y is weak
B) The relationship between X1 and Y is strong
C) The relationship between X1 and Y is neutral
D) Correlation does not allow us to judge the relationship
Solution: (B)
The absolute value of the correlation coefficient denotes the strength of the relationship. Since the absolute correlation is very high, this means there is a strong relationship between X1 and Y.
9) Looking at the two characteristics mentioned above, which of the following options is correct for the Pearson correlation between V1 and V2?
If you are given two variables V1 and V2, and they satisfy the two characteristics below.
1. If V1 increases, then V2 also increases.
2. If V1 decreases, then the behavior of V2 is unknown.
A) Pearson correlation will be close to 1
B) Pearson correlation will be close to -1
C) Pearson correlation will be close to 0
D) None of these values
Solution: (D)
We cannot comment on the correlation coefficient using only statement 1. We need to consider both of these statements. Consider V1 as x and V2 as | x |. In this case, the correlation coefficient will not be close to 1.
10) Suppose the Pearson correlation between V1 and V2 is zero. In this case, can we conclude that V1 and V2 have no relationship between them?
A) TRUE
B) FALSE
Solution: (B)
The Pearson correlation coefficient between two variables can be zero even if they are related to each other. If the correlation coefficient is zero, this simply means that they do not move together. We can take examples such as y = | x | or y = x ^ 2.
11) Which of the following biases do we use when fitting the least squares line of linear regression? Assume that the horizontal axis is the independent variable and the vertical axis is the dependent variable.
A) Vertical offset
B) Perpendicular offset
C) Both, depending on the situation
D) None of the above
Solution: (A)
We always consider residuals as vertical offsets. We compute the direct difference between the actual value and the Y label. Perpendicular offset is useful in the case of PCA.
12) True-False: overfitting is more likely when you have a huge amount of training data?
A) TRUE
B) FALSE
Solution: (B)
With a small training dataset, it is easier to find a hypothesis that exactly matches the training data, i.e.
13) We can also compute the linear regression coefficient using an analytical method called the «Normal Equation». Which of the following is true/correct regarding the normal equation?
A) 1 and 2
B) 1 and 3
C) 2 and 3
D) 1, 2 and 3
Solution: (D)
Instead of gradient descent for finding the coefficients, the normal equation can also be used. Refer to this article to learn more about the normal equation.
14) Which of the following statements regarding the sum of residuals of A and B is true?
The graphs below show two fitted regression lines (A and B) for randomly generated data. Now I want to find the sum of residuals in both cases A and B.
Note:
A) A has a higher sum of residuals than B
B) A has a lower sum of residuals than B
C) Both have the same sum of residuals
D) None of these
Solution: (C)
The sum of residuals will always be equal to zero, so both have the same sum of residuals.
Context for questions 15-17:
Suppose you applied a complex regression model to a dataset. Now you are using ridge regression with penalty x.
15) Choose the option that best describes bias.
A) In the case of a very large x; bias is low
B) In the case of a very large x; bias is high
C
) Cannot say anything about the biasD) None of these
Solution: (B)
If the penalty is very large, this means the model is less complex, so the bias will be high.
16) What will happen if you impose a very large penalty?
A) Some of the coefficients will become absolute zero
B) Some coefficients will approach zero, but not absolute zero
C) Both A and B depending on the situation
D) None of these
Solution: (B)
In lasso, some coefficient values become equal to zero, but in the case of ridge, the coefficients become close to zero, but not equal to zero.
17) What will happen if you apply a very large penalty in the case of Lasso?
A) Some coefficients will become equal to zero
B) Some coefficients will approach zero, but not absolute zero
C) Both A and B depending on the situation
D) None of these
Solution: (A)
As already mentioned, lasso applies an absolute penalty, so some coefficients will become zero.
18) Which of the following statements regarding outliers in linear regression is true?
A) Linear regression is sensitive to outliers
B) Linear regression is not sensitive to outliers
C) Cannot say
D) None of this
Solution: (A)
The slope of the regression line will in most cases change due to outliers. So, linear regression is sensitive to outliers.
19) Suppose you built a scatter plot between the residuals and the predicted values in linear regression and found that there is a relationship between them. Which of the following conclusions would you draw about this situation?
A) Since there is a relationship, this means our model is not good
B) Since there is a relationship, this means our model is good
C) Cannot say
D) None of this
Solution: (A)
There should be no relationship between the predicted values and the residuals. If there is any relationship between them, this means the model has not fully captured the information in the data.
Context for questions 20-22:
Suppose you have a dataset D1 and you build a linear regression model with a 3rd degree polynomial, and you found that the training and testing error is «0», or in other words, it fits the data perfectly.
20) What will happen if you put a 4th degree polynomial into linear regression?
A) High chance that the 4th degree polynomial will fit the data better
B) High chance that the 4th degree polynomial will not fit the data
C) Cannot say
D) None of these
Solution: (A)
Since a degree 4 model will be more complex (overfit the data) than the degree 3 model, it will again fit the data perfectly. In this case, the training error will be zero, but the test error may not be zero.
21) What will happen if you put a 2nd degree polynomial into linear regression?
A) High chance that the degree 2 polynomial will fit the data better
B) High chance that the degree 2 polynomial will not fit the data
C) Cannot say
D) None of these
Solution: (B)
If a degree 3 polynomial fits the data perfectly, it is quite likely that a simpler model (degree 2 polynomial) may not fit the data.
22) In terms of bias and variance. Which of the following is true when you fit a degree 2 polynomial?
A) bias will be high, variance will be high
B) bias will be low, variance will be high
C) bias will be high, variance will be low
D) bias will be low, variance will be low
Solution: (C)
Since the degree 2 polynomial will be less complex compared to the degree 3 polynomial, the bias will be high, and the variance will be low.
Context for question 23:
Which of the following is true regarding the graphs below (A, B, C from left to right) between the cost function and the number of iterations?
23) Suppose l1, l2, and l3 are three learning rates for A, B, C respectively. Which of the following is true regarding l1, l2, and l3?
A) l2 <l1 <l3
B) l1> l2> l3
C) l1 = l2 = l3
D) None of these
Solution: (A)
In the case of a high learning rate, the step will be large, the objective function will initially decrease quickly, but it will not find the global minimum, and the objective function will start to increase after a few iterations.
In the case of a low learning rate, the step will be small. Thus, the objective function will decrease slowly
Context for questions 24-25:
We were given a dataset with n records, in which the input attribute is x and the output attribute is y. Suppose we use the linear regression method to model this data. To validate our linear regressor, we randomly split the data into a training set and a test set.
24) Now we gradually increase the size of the training sample. As the size of the training sample increases, what do you expect will happen to the average training error?
A) Increase
B) Decrease
C) Remain constant
D) Cannot say
Solution: (D)
The training error may increase or decrease depending on the values used to fit the model. If the values used for training gradually contain more outliers, the error may simply increase.
25) What do you expect will happen to bias and variance as the size of the training data increases?
A) Bias increases and variance increases
B) Bias decreases and variance increases
C) Bias decreases and variance decreases
D) bias increases and variance decreases
E) Cannot say false
Solution: (D)
As the size of the training data increases, the bias will increase and the variance will decrease.
Context for question 26:
Consider the following data, in which one input (X) and one output (Y) are given.
26) What will be the mean squared training error for this data if you run a linear regression model of the form (Y = A0 + A1X)?
A) Less than 0
B) Greater than zero
C) Equal to 0
D) None of these values
Solution: (C)
We can fit the line to the following data perfectly, so the average error will be equal to zero.
Context for questions 27-28:
Suppose you are given the following scenario of training and validation error for linear regression.
| Scenario | Learning rate | Number of iterations | Training error | Validation error |
| 1 | 0.1 | 1000 | 100 | 110 |
| 2 | 0.2 | 600 | 90 | 105 |
| 3 | 0.3 | 400 | 110 | 110 |
| 4 | 0.4 | 300 | 120 | 130 |
| 5 | 0.4 | 250 | 130 | 150 |
27) Which of the following scenarios would give you the correct hyperparameter?
A) 1
B) 2
C) 3
D) 4
Solution: (B)
Option B would be the best choice because it leads to lower training as well as validation error.
28) Suppose you got the tuned hyperparameters from the previous question. Now imagine that you want to add a variable to the variable space, such that this added feature is important. Which of the following would you notice in this case?
A) Training error will decrease and validation error will increase
B) Training error will increase and validation error will increase
C) Training error will increase and validation error will decrease
D) Training error will decrease and validation error will decrease
E) None of the above
Solution: (D)
If the added feature is important, both training and validation error will decrease.
Context for questions 29-30:
Suppose you find yourself in a situation where you discover that your linear regression model does not fit the data well.
29) In such a situation, which of the following options would you consider?
A) 1 and 2
B) 2 and 3
C) 1 and 3
D) 1, 2 and 3
Solution: (A)
In the case of underfitting, you need to create more variables in the variable space, or you can add several polynomial degree variables to make the model more complex, so that it can better fit the data.
30) Now the situation is the same as in the previous question (underfitting). Which of the following regularization algorithms would you prefer?
A) L1
B) L2
C) Either
D) None of these
Solution: (D)
I would not use any regularization methods, because regularization is used in the case of overfitting.
I have tried my best to make the solutions as comprehensive as possible, but if you have any questions/doubts, leave your comments below. I would love to hear your feedback on the skills test
Comments