Perceptron

Intuition

A perceptron is a decision made by voting. Every input gets a vote, and every vote carries a weight. Add the weighted votes together, add a bias that tilts the outcome one way or the other, and check the total against a threshold. Over the line, output 1. Under it, output 0.

That is the entire mechanism. It is the smallest thing that can still be called a neural network, and every large model is built from this shape repeated.

Formal definition

The perceptron computes the dot product of the weight vector with the input vector, adds a bias, and applies a step function to the result. There is no probability and no confidence — the output is a hard 1 or 0.

How it works

The order in the animation matters. Nothing is decided until every weighted input has arrived at the summation node — the perceptron cannot "partly" fire. The step at the end is what makes it a classifier rather than a regression: it throws away the magnitude of the sum and keeps only which side of zero it landed on.

Training is equally blunt. The perceptron updates its weights only when it gets an example wrong [1, §1.1.13.1]. A correct prediction produces no change at all. Where a model fitted by least squares nudges its coefficients on every observation, this one ignores everything it already handles and moves only on its mistakes.

Where it's used

Rarely on its own any more — but it is load-bearing in three places:

• As a building block. Stack perceptron-like units into layers and you have a feedforward neural network, which is where the modern field starts [2, §6.1].
• Large-scale linear classification. scikit-learn keeps it as a fast option for large-scale learning: it needs no learning rate and applies no regularisation, which makes it slightly quicker to train than SGD with hinge loss and produces sparser models [1, §1.1.13.1].
• Teaching. It is the shortest path to understanding weights, bias, activation and the training loop, because none of them are hidden.

Why & when

Reach for it when you want the smallest possible model that draws a straight boundary, on a large dataset, with no tuning. There is no learning rate to choose [1, §1.1.13.1], which removes the most common source of fiddling.

Reach for something else almost any other time. The perceptron gives you no probability, no margin, and no guarantee of a good boundary — only a boundary, if one exists. Logistic regression is more likely to give well-calibrated probabilities for the same shape of problem [5, §1.16.1]; a linear SVM gives the widest separating margin rather than the first one found [3, §1.4.7].

Worked example

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import Perceptron
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

# Scaling matters: the perceptron derives from SGD, which is sensitive to
# feature scaling.
model = make_pipeline(StandardScaler(), Perceptron(random_state=0))
model.fit(X_train, y_train)

print(model.score(X_test, y_test))

0.9473684210526315

Under the hood it "derives from SGD" and shares scikit-learn's stochastic gradient machinery [1, §1.1.13.1]. It scores 0.947 on held-out data here, with no learning rate chosen.

Variants & extensions

That last row is the important one. A single perceptron can only carve space with one straight cut, so any problem whose classes cannot be separated by a single line is out of reach. Stacking layers with non-linear activations is the move that removes the restriction — the XOR example is the standard demonstration [2, §6.1].

Pitfalls

Scale your features. The perceptron derives from SGD, which "is sensitive to feature scaling": a feature measured in thousands will dominate one measured in fractions purely because of its units [4, §1.5.7].

It only converges if the data is linearly separable. It updates only on mistakes [1, §1.1.13.1], and if no straight boundary exists there is always a mistake to make, so the weights never settle. scikit-learn stops it after max_iter passes, or once the loss stops improving by tol, and hands back wherever it happened to be. It doesn't fail loudly; it just never converges.

The output is not a confidence. A 1 from a perceptron means "on this side of the line", not "probably this class" [5, §1.16]. Reading it as certainty is a mistake the step function makes very easy.