*To see this with sample code, check out my Kaggle notebook.

Like simple and multiple regression, Polynomial regression is a “supervised” “regression” algorithm.

Supervised meaning we use labeled data to train the model.

Regression meaning we predict a numerical value, instead of a “class”.

However, polynomial regression differs in that it allows us fit curved lines/planes to our data.

This is accomplished by “exponentiating” our variable by taking it to powers greater than 1.

This makes it a fantastic modeling tool for “exponential” datasets.

Polynomial regression still uses “least squares“, but it is an “overdetermined system” meaning that there are more equations than unknowns.

Therefore, we must rely on matrix multiplication to solve for the the best fit.

In practice, this is handled by our statistical software, we just need to select the “degree” of the polynomial.

Our selection should seek to fit the current data well, and generalize to new data.

This challenge is known as the “bias variance tradeoff”.

As we increase the degree, the bias decreases, but the variance increases. We want to stop where these two factors are minimized.

Overall, Polynomial Regression is an essential tool in your Machine Learning arsenal. However, it is very sensitive to outliers, and we must take care when selecting the degree to avoid overfitting.