Suppose we estimate the regression coefficients in a linear regression model by

Question

Suppose we estimate the regression coefficients in a linear regression model by minimizing 2 for a particular value of s. For parts (a) through (e), indicate which of i. through v. is correct. Justify your answer. (a) As we increase s from 0, the training RSS will: i. Increase initially, and then eventually start decreasing in an inverted U shape. ii. Decrease initially, and then eventually start increasing in a U shape. iii. Steadily increase. iv. Steadily decrease. v. Remain constant. (b) Repeat (a) for test RSS. (c) Repeat (a) for variance (d) Repeat (a) for (squared) bias. (e) Repeat (a) for the irreducible error

Explanation / Answer

Explanation : As we increase s from 0, the j variables are less and less restricted, since they can take values from an wider range. Hence, the model becomes more flexible and fits the training data better which causes a steady decrease in training RSS.

(b)The answer is (ii). As we increase s from 0, the test RSS will decrease initially, and then eventually start increasing in an U shape.

Explanation : As explained above, as we increase s from 0, the model fits the training data better and better. At first, this causes the test RSS to decrease, because the model becomes better at explaining the data. But, after a certain point, further increase in the value of s makes model more flexible and it starts overfitting the training data and loses generality. Hence, the test RSS starts increasing after this point.

(c)The answer is (iii). As we increase s from 0, the variance will steadily increase.

Explanation : At s = 0, the model has no variance. But as s increases from 0, more and more j variables are incorporated into the model, the model starts fitting the training data better and better and finally starts overfitting, which increases the variance.

(d)The answer is (iv). As we increase s from 0, the squared bias will steadily decrease.

Explanation : As explained above, the increase in the value of s causes the model to be more flexible and hence fit the training data better and better. Hence the squared bias will steadily decrease.

(e)The answer is (v). As we increase s from 0, the irreducible error will remain constant.

Explanation : By definition, the irreducible error is independent of the model. Hence, no matter how the model changes due to the increase in value of s, the irreducible error will not change.

Related Questions

Navigate

• Browse (All)
• Browse S
• Subjects

---

---

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.