Perceptron

So far, we have learned models that fit lines, build trees, transform features, compress representations, and discover groups. Deep learning begins with a smaller question.

What is the simplest computational unit that can learn from data?

The answer is the perceptron. A perceptron is one artificial neuron. By itself, it is limited. Combined with many other neurons, it becomes the building block of modern neural networks.

Historical Motivation

In 1958, Frank Rosenblatt introduced the perceptron. Researchers wanted machines inspired by the brain: units that receive signals, combine them, and fire when the combined signal is strong enough.

The perceptron was a mathematical version of that idea. It was primitive, but it started the neural-network story.

Imagine predicting whether Jessie orders a burger using hours since last meal, burger price, and number of friends ordering burgers.

The perceptron gives every feature a weight. Positive weights push toward Burger. Negative weights push toward No Burger. Weights near zero mean the feature barely matters.

This should feel familiar. A perceptron is logistic regression stripped down to its most basic neural idea: compute one weighted sum, then make one decision.

The Artificial Neuron

Suppose the input vector is $x$ and the weights are $w$. The perceptron first computes:

Expanded for three features:

This quantity is sometimes called the activation or logit. The equation is the same weighted sum used by linear regression and logistic regression.

The Step Function

The perceptron turns the weighted sum into a hard decision with a step function:

If the total evidence is positive, predict Burger. Otherwise, predict No Burger. Unlike a sigmoid, the step function does not produce probabilities.

Worked Example

Suppose $x_1=8$ hours since meal, $x_2=15$ dollars, and $x_3=4$ friends.

Since $z>0$, the perceptron predicts Burger. The calculation is interpretable as weighted evidence: hunger and friends push up, price pushes down.

The Decision Boundary

Because the score is linear, a single perceptron learns a linear decision boundary. In two dimensions, the boundary is a line. In three dimensions, it is a plane. In higher dimensions, it is a hyperplane.

The Learning Rule

If the perceptron makes a mistake, move the weights so the correct answer becomes more likely:

Here $y$ is the true label, $\hat y$ is the prediction, and $\eta$ is the learning rate.

If Jessie did not order a burger but the model predicted Burger, then $y-\hat y=-1$. With $\eta=0.1$:

The features that incorrectly pushed the model toward Burger are weakened. Unlike gradient descent, this update happens only when the perceptron is wrong.

Why It Works

If the prediction is correct, $y-\hat y=0$, so nothing changes. If the prediction is wrong, the weights move in a direction that makes the correct label more likely for that example.

If the data is linearly separable, the Perceptron Convergence Theorem says this learning rule will find a separating hyperplane in a finite number of updates. If the data is not linearly separable, it may never converge.

Perceptron Versus Logistic Regression

Why One Perceptron Is Limited

A single perceptron cannot solve XOR:

No single straight line separates the positive and negative XOR examples. The solution is not a better single perceptron. The solution is to connect perceptrons in layers.

This implementation trains a perceptron from scratch. Labels are encoded as 0 and 1. The update happens only when the prediction is wrong.

import numpy as np

class Perceptron:
    def __init__(self, learning_rate=0.1, epochs=20):
        self.learning_rate = learning_rate
        self.epochs = epochs
        self.w = None
        self.b = 0.0

    def step(self, z):
        return 1 if z >= 0 else 0

    def fit(self, X, y):
        self.w = np.zeros(X.shape[1])
        self.b = 0.0

        for _ in range(self.epochs):
            for x_i, y_i in zip(X, y):
                z = np.dot(self.w, x_i) + self.b
                y_hat = self.step(z)
                error = y_i - y_hat

                if error != 0:
                    self.w += self.learning_rate * error * x_i
                    self.b += self.learning_rate * error

        return self

    def predict(self, X):
        scores = X @ self.w + self.b
        return np.where(scores >= 0, 1, 0)