1. (20 points) Recall the objective function for linear regression can be expres

Question

1. (20 points) Recall the objective function for linear regression can be expressed as as in (3.3) of LFD. Minimizing this function with respect to w leads to the optimal w as (XTX) Xy. This solution holds only when XTX is nonsingular. To overcome this prob- lem, the following objective function is commonly minimized instead where > 0 is a user-specified parameter. Please do the following: (a) (1O points) Derive the optimal w that minimize E() (b) (10 points) Explain how this new objective function can overcome the singularity problem of XTX

Explanation / Answer

E2(w) = (Xw-y)'(Xw-y) + lambda.(w'w)

= w'X'Xw -2w'X'y + y'y + lambda.(w'w)

Differentiating wrt w to min E2(w), we get:

2X'Xw - 2X'y + lambda.w = 0

Reaggranding the terms,

(2X'X + lambda.I)w=2X'y

w=inv(2X'X + lambda.I) * 2X'y

Earlier the problem was that X'X could be singular. As lambda>0 and user defined we can use lambda such that 2X'X + lambda.I is not singular and hence invertible. Hence a solution to w can be found as inv(X'X + lambda.I) exists.

Related Questions

Navigate

• Browse (All)
• Browse 1
• Subjects

---

---

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.