Linear Regression

Linear regression assumes that each feature contributes a steady amount to the prediction. If the temperature rises by one degree, the model adds the same amount of expected iced coffee sales each time.

With one feature, the model is the familiar equation for a line.

The symbol $\hat{y}$ means predicted target. The weight $w$ controls how strongly the input changes the prediction. A positive weight makes the line rise. A negative weight makes it fall. The bias $b$ is the prediction when the input contributes nothing.

Real datasets often have many features. A house price model might use square footage, number of bedrooms, age, and distance from a station. We collect those feature values in a vector $x$ and give each feature its own weight.

The expression $w^T x$ means multiply each feature by its weight, then add the results. The bias shifts the final answer up or down. This is still a line in one dimension, a plane in two dimensions, and a hyperplane when there are more features.

Now the model can make predictions, but we still have not said which line is best. To choose between lines, we need a way to measure how wrong a line is.

A loss function turns prediction quality into a number. Small loss means the model is close to the data. Large loss means it is missing the pattern.

For one training example, the raw error is the actual value minus the predicted value.

If we simply averaged raw errors, positive and negative mistakes could cancel. A prediction that is ten too high and another prediction that is ten too low would look perfect together, even though both are wrong. Squaring fixes that.

Squaring makes every error non-negative and gives larger mistakes a larger penalty. An error of $2$ becomes $4$, while an error of $10$ becomes $100$. The model is therefore pushed to avoid big misses.

To judge the whole dataset, we average the squared errors across all $n$ examples.

This gives training a concrete goal. Find the weights and bias that make MSE as small as possible. For a simple line, there is a direct formula. For models used later in the course, we need a general way to improve parameters step by step.

That need leads naturally to gradient descent. Before we go there, one more question exposes the limit of linear regression. What happens if the output is no longer a continuous number?

This minimal implementation uses the closed form solution for one feature. The method is useful for seeing the mechanics, while gradient descent will give us a more general training tool.

import numpy as np

class LinearRegression:
    def __init__(self):
        self.w = 0.0
        self.b = 0.0

    def fit(self, X, y):
        n = len(X)
        numerator = n * np.sum(X * y) - np.sum(X) * np.sum(y)
        denominator = n * np.sum(X ** 2) - np.sum(X) ** 2
        self.w = numerator / denominator
        self.b = (np.sum(y) - self.w * np.sum(X)) / n

    def predict(self, X):
        return self.w * X + self.b

    def mse(self, X, y):
        predictions = self.predict(X)
        return np.mean((y - predictions) ** 2)

X = np.array([1, 2, 3, 4, 5])
y = np.array([2.1, 4.0, 5.8, 8.2, 9.7])

model = LinearRegression()
model.fit(X, y)

print(model.w, model.b)
print(model.predict(6))
print(model.mse(X, y))