Types of Regression Functions

Simple Linear Regression:

One independent variable.
Predicts a continuous dependent variable.
Example: Predicting salary based on years of experience.

Multiple Linear Regression:

Multiple independent variables.
Predicts a continuous dependent variable.
Example: Predicting house prices using features like area, number of rooms, and location.

Polynomial Regression:

A linear model that fits a polynomial relationship between independent and dependent variables.
Useful when the relationship between variables is non-linear.

Support Vector Regression (SVR):

Based on Support Vector Machines.
Works well with small datasets and non-linear data by using kernel functions.

Decision Tree Regression:

Splits the data into subsets based on feature values to predict a continuous variable.
Useful for non-linear and complex relationships.

Random Forest Regression:

An ensemble of decision trees.
Averages the predictions from multiple trees to improve accuracy and reduce overfitting.

Ordinary Least Squares (OLS)

Concept: Minimizes the sum of the squared residuals (differences between actual and predicted values).
Residual: Residual=yactual−ypredicted\text{Residual} = y_{\text{actual}} - y_{\text{predicted}}.
Key Property: In linear regression using OLS, the sum of residuals is zero, ensuring an unbiased estimate of the mean.

Regression vs. Classification

Regression: Predicts continuous real values (e.g., salary, price).
Classification: Predicts categorical outcomes (e.g., spam vs. not spam).

Implementing Regression Using scikit-learn

# Import the necessary library
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt

# Create a Linear Regression instance
regressor = LinearRegression()

# Train the model
regressor.fit(x_train, y_train)

# Predict results
y_pred = regressor.predict(x_test)

# Visualize results
plt.scatter(x_train, y_train, color='red')  # Training data points
plt.plot(x_train, regressor.predict(x_train), color='blue')  # Regression line
plt.title('Years of Experience vs Salary (Training Set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

# Visualize test set results
plt.scatter(x_test, y_test, color='red')  # Test data points
plt.plot(x_test, regressor.predict(x_test), color='blue')  # Regression line
plt.title('Years of Experience vs Salary (Test Set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

Key Points

fit() Method:

Trains the model on the x_train and y_train datasets.

Prediction:

regressor.predict() computes predicted values for the test set.

Visualization:

matplotlib is used for scatter plots and regression lines.
Helps understand the correlation and fit.