Day7: Linear Regression

## 📑 Table of Contents

1. 🌟 Welcome to Day 7
2. 🤔 What is Linear Regression?
3. 📜 Mathematical Foundations of Linear Regression
   • The Model (Simple & Multiple)
   • The Cost Function (MSE)
   • Normal Equations
   • Gradient Descent
   • Key Assumptions
4. 🏗️ Practical Techniques and Code Examples
   • Fitting a Linear Model with Scikit-Learn
   • Interpreting Coefficients
   • Visualizing the Regression Line
   • Evaluating the Model (MSE, R²)
   • Multiple Linear Regression
   • Polynomial Regression
   • Regularization (Ridge, Lasso)
   • Residual Analysis
5. 🔍 Integrating Linear Regression with Preprocessing, Feature Engineering, and Model Selection
6. 💻 Practical Examples and Use Cases
7. 📚 Resources
8. 💡 Tips and Tricks
9. 🧮 Worked Mathematical Examples (Complete Solutions)

---

1. 🌟 Welcome to Day 7

Welcome to Day 7 of your 90-day machine learning journey! Today, we will thoroughly explore Linear Regression, a fundamental technique in predictive modeling. By the end, you’ll know the math behind it, how to implement it, interpret parameters, evaluate model performance, handle multiple features, add polynomial terms, and even apply regularization methods.

---

2. 🤔 What is Linear Regression?

Linear regression fits a linear equation relating independent variables (features) to a continuous dependent variable (target).

• Simple Linear Regression (one feature x):

  

• Multiple Linear Regression (features x₁,…,xₚ):

  

The task: find parameters β₀,…,βₚ that best fit the data by minimizing errors.

---

3. 📜 Mathematical Foundations of Linear Regression

📝 The Model

In vector form:

• y: n×1 target vector
• X: n×(p+1) feature matrix (first column = 1s for intercept)
• β: (p+1)×1 parameter vector
• ε: n×1 error vector

📝 The Cost Function (MSE)

We measure fit using Mean Squared Error (MSE):

Matrix form:

📝 Normal Equations

Closed-form solution for β:

📝 Gradient Descent

Iterative approach:

For simple linear regression:

📝 Key Assumptions

• Linearity
• Constant variance of errors (homoscedasticity)
• Independence of errors
• No perfect multicollinearity
• Errors normally distributed (for inference)

---

4. 🏗️ Practical Techniques and Code Examples

📝 Fitting a Linear Model (Scikit-Learn)

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1],[2],[3],[4],[5]], dtype=float)
y = np.array([2,4,5,4,5], dtype=float)

model = LinearRegression()
model.fit(X, y)

print("Intercept:", model.intercept_)
print("Coefficient:", model.coef_)

📝 Interpreting Coefficients

If β₁ = 2, increasing x by 1 increases y by about 2 units on average.

📝 Visualizing the Regression Line

import matplotlib.pyplot as plt
y_pred = model.predict(X)
plt.scatter(X, y, color='blue', label='Data')
plt.plot(X, y_pred, color='red', label='Line')
plt.xlabel("X")
plt.ylabel("y")
plt.title("Linear Regression")
plt.legend()
plt.show()

📝 Evaluating the Model (MSE, R²)

from sklearn.metrics import mean_squared_error, r2_score
mse = mean_squared_error(y, y_pred)
r2 = r2_score(y, y_pred)
print("MSE:", mse)
print("R^2:", r2)

📝 Multiple Linear Regression

X_multi = np.array([[1,2],[2,3],[3,5],[4,6],[5,7]], dtype=float)
y_multi = np.array([5,7,10,11,14], dtype=float)
model_multi = LinearRegression()
model_multi.fit(X_multi, y_multi)
print("Intercept:", model_multi.intercept_)
print("Coefficients:", model_multi.coef_)

📝 Polynomial Regression

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
model_poly = LinearRegression()
model_poly.fit(X_poly, y)
print("Poly Coeff:", model_poly.coef_, "Intercept:", model_poly.intercept_)

📝 Regularization (Ridge, Lasso)

from sklearn.linear_model import Ridge, Lasso
ridge = Ridge(alpha=1.0)
ridge.fit(X_multi, y_multi)
print("Ridge Coeff:", ridge.coef_)

lasso = Lasso(alpha=0.1)
lasso.fit(X_multi, y_multi)
print("Lasso Coeff:", lasso.coef_)

📝 Residual Analysis

residuals = y - y_pred
plt.scatter(y_pred, residuals, color='green')
plt.axhline(y=0, color='red', linestyle='--')
plt.xlabel("Predicted")
plt.ylabel("Residuals")
plt.title("Residual Analysis")
plt.show()

---

5. 🔍 Integrating Linear Regression with Preprocessing, Feature Engineering, and Model Selection

Ensure data is clean, scaled if needed, and consider feature engineering. Compare linear regression with other models via cross-validation.

---

6. 💻 Practical Examples and Use Cases

• Real Estate: Predict house prices
• Economics: Relate GDP to consumption
• Medicine: Relate lifestyle factors to health metrics
• Marketing: Ads spend vs. sales

---

7. 📚 Resources

• Scikit-Learn LinearRegression
• ISL (Introduction to Statistical Learning)
• Kaggle datasets
• "The Elements of Statistical Learning"

---

8. 💡 Tips and Tricks

• Check residuals for patterns
• Try polynomial/log transforms if needed
• Use Ridge/Lasso for stability with correlated predictors
• Apply domain knowledge for feature selection
• Cross-validate for robust performance

---

9. 🧮 Worked Mathematical Examples (Complete Solutions)

Example 1: Closed-Form (Normal Equations) Simple Linear Regression

Data: (x,y) = (1,2),(2,4),(3,5),(4,4),(5,5)

Means: x̄=3, ȳ=4

Slope:

Compute deviations:
x_i - x̄: (-2,-1,0,1,2)
y_i - ȳ: (-2,0,1,0,1)

Sum numerator: (-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)=4+0+0+0+2=6
Sum denominator: 4+1+0+1+4=10

β₁ = 6/10 = 0.6

Intercept:

Model:

Example 2: One Iteration of Gradient Descent

Start: β₀=0, β₁=0, α=0.01
y actual: 2,4,5,4,5 (predictions all 0)

Update:
β₀ = 0 - 0.01(-8)=0.08
β₁ = 0 - 0.01(-26.4)=0.264

Closer to final (~2.2 and 0.6).

Example 3: Multiple Regression (Matrix Form)

For multiple regression:

Compute XᵀX, invert, multiply by Xᵀy.

Example 4: Interpreting Multiple Coefficients

If β₁=3, β₂=-1:

• Increasing x₁ by 1 raises y by 3 (holding x₂ constant)
• Increasing x₂ by 1 lowers y by 1 (holding x₁ constant)

---

Next steps: building on this foundation, exploring advanced techniques, and pushing your analytical skills to new heights! 🚀