L28: Linear regression | error functions, overfitting & generalization

Summary

Key Points

• Regression aims to learn a function that maps d-dimensional features to real-valued labels for predicting future input instances. 1:31
• The goodness of a function is measured by its error with respect to the given dataset, typically quantified as a single numerical value. 2:38
• The squared error, calculated as the sum of (predicted value - actual value)^2 for all data points, is a common and justified method for measuring error. 3:11
• This memorization results in overfitting, meaning the model performs poorly on unseen test data despite perfect training performance. 8:09
• Minimizing training error to zero by allowing any function can lead to memorization, where the model perfectly matches training data but fails to generalize. 8:16
• To prevent overfitting, the search space for the function (hypothesis space) must be restricted, assuming a certain structure for the true underlying mapping. 10:32
• The simplest structural restriction is to consider only linear functions, such as h(x) = w^T x, which are parameterized by a weight vector 'w'. 12:19
• By imposing a linear structure, the model aims to fit the underlying relationship between input and output, rather than fitting noise, which helps prevent overfitting. 13:31
• Achieving zero error in linear regression implies that the data perfectly aligns with the hypothesized linear structure, which is a desirable fit within the constrained space, not overfitting. 13:43
• Linear regression is defined as minimizing the squared error specifically within this restricted space of linear functions. 15:02