Polynomial Linear Regression: Comprehensive Notes

Polynomial regression is an extension of linear regression that is used when the relationship between the independent and dependent variables is non-linear. Despite its name, it is still called linear regression because the model remains linear in the coefficients (b). For instance, even though the input x is transformed into polynomial features (x², x³, etc.), the regression equation remains a linear combination of these features:

y=b0+b1x+b2x2+b3x3+⋯+bnxn y = b_0 + b_1x + b_2x^2 + b_3x^3 + \dots + b_nx^nThis flexibility allows polynomial regression to model more complex relationships than simple linear regression.

In this project, we aim to predict salaries based on position levels using polynomial regression. The dataset used is Position_Salaries.csv. Below is a step-by-step explanation along with the code for implementation:

1. Importing Libraries

First, we import the necessary libraries for numerical computations, data handling, and visualization:

import numpy as np
import pandas as pd
import matplotlib.pyplot as mp

2. Loading the Dataset

The dataset is loaded using pandas, and the independent (x) and dependent (y) variables are extracted. The independent variable x is reshaped into a 2D array for compatibility with scikit-learn models:

# Importing the dataset
dataset = pd.read_csv('Position_Salaries.csv')
x = dataset.iloc[:, 1].values.reshape(-1, 1)  # Independent variable
y = dataset.iloc[:, -1].values  # Dependent variable

3. Training the Linear Regression Model

A baseline linear regression model is trained on the entire dataset:

from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(x, y)

4. Training the Polynomial Regression Model

To handle non-linear relationships, we transform x into polynomial features using the PolynomialFeatures class from scikit-learn. For this example, we use a polynomial of degree 4:

from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree=4)
x_poly = poly_reg.fit_transform(x)

line_reg_2 = LinearRegression()
line_reg_2.fit(x_poly, y)

5. Visualization of Results

Linear Regression Results

We visualize the results of the linear regression model. The scatter plot shows the original data points in red, while the blue line represents predictions from the linear regression model:

mp.scatter(x, y, color='red')
mp.plot(x, lin_reg.predict(x), color='blue')
mp.title('Truth or Bluff (Linear Regression)')
mp.xlabel('Position Level')
mp.ylabel('Salary')
mp.show()

Polynomial Regression Results

For polynomial regression, we use a finer grid of x values to create a smoother curve:

x_grid = np.arange(min(x), max(x), 0.1).reshape(-1, 1)
mp.scatter(x, y, color='red')
mp.plot(x_grid, line_reg_2.predict(poly_reg.fit_transform(x_grid)), color='blue')
mp.title('Truth or Bluff (Polynomial Regression)')
mp.xlabel('Position Level')
mp.ylabel('Salary')
mp.show()

6. Making Predictions

Linear Regression Prediction

We can make predictions using the linear regression model. For example:

prediction = lin_reg.predict([[6.5]])
print(prediction)  # Output may not be accurate for non-linear data

Polynomial Regression Prediction

Using the polynomial regression model, we can predict more accurate results for non-linear data:

poly_prediction = line_reg_2.predict(poly_reg.fit_transform([[6.5]]))
print(poly_prediction)

Key Concepts

Why Polynomial Regression?

Linear regression assumes a straight-line relationship, which fails for complex, non-linear datasets.
Polynomial regression introduces higher-order terms (x², x³, etc.), enabling the model to capture non-linear trends effectively.

Structure of Polynomial Data

Polynomial transformation creates additional features representing powers of the original variable.
For degree=2 and x=[1, 2, 3], the transformed data becomes: x_poly=[111212221332]\text{x\_poly} = \begin{bmatrix} 1 & 1 & 1^2 \\ 1 & 2 & 2^2 \\ 1 & 3 & 3^2 \end{bmatrix}

Final Notes

The choice of polynomial degree is critical. Too low a degree results in underfitting, while too high a degree may lead to overfitting.
Always compare results from both linear and polynomial regression models to evaluate their performance on the given dataset.