How does backpropagation work? Follow one gradient backward

What you're seeing

The left pane is the whole network: two input nodes, a hidden layer of tanh units (three by default), one sigmoid output node, and a LOSS box at the far right. Each edge is a weight. A solid edge is a positive weight, a dashed edge is a negative one, and thickness is the size of the weight. During the backward pass each edge glows in proportion to its gradient and shows the number. The small dashed nodes labeled 1 are the bias inputs, which are weights too.

The right pane is the decision surface. The network's output is sampled on a 72 by 72 grid across the square from -1 to 1, and the color is that output: near-black where the network says class 0, bright red where it says class 1. The 80 training points sit on top, a filled dot for class 1 and a hollow ring for class 0. The point with a glowing ring is the example currently moving through the step cycle.

The phase indicator above the network reads FORWARD, LOSS, BACKWARD, UPDATE. Each press of Step advances one phase for one example. In the forward phase a pulse travels left to right and each node shows its activation. In the loss phase the LOSS box shows the error for this example. In the backward phase the pulse travels right to left and every edge shows its gradient. In the update phase each weight moves a small step against its gradient, the edge thickness changes, and the surface redraws. The stats in the top right track STEP (weight updates so far), LOSS (mean over all 80 points), ACC (points classified correctly), and GRAD (the size of the last gradient).

A few things to try:

Tap an edge. The chain-rule inspector opens and writes that weight's gradient as a product of local factors with the live number under each one. For an input-to-hidden weight there are four: the error at the output, the weight the signal passed through, the slope of tanh at that unit, and the input value. Multiply them and you get the gradient the update uses.
Look for SAT. When a hidden unit's activation is above 0.98 or below -0.98, its tanh slope is near zero and the demo marks it SAT. Tap a weight feeding that unit. The slope factor is tiny, so the product is tiny, and the edge barely glows. The weight learns almost nothing from this example.
Switch to XOR with 1 hidden unit. Press Play and watch the accuracy stall near 66% no matter how long it trains. Slide hidden units to 3, press Play again, and it reaches 100%.
Push the learning rate up. Above 1 the loss curve along the bottom starts to jump around and the surface flickers. Each step overshoots the valley it was aiming for.
Tap a data point. It becomes the step example. Pick a point the surface gets wrong, step through its backward pass, and watch the gradients grow. A confident mistake produces a big error term, and every factor downstream of it scales with that error.

How does backpropagation work?

A network learns by being wrong. One training step has four beats, and the demo's Step button walks through them in order.

Forward pass. The turns an input into a prediction. Each hidden unit takes a weighted sum of the inputs plus its bias and squashes it with tanh. The output unit takes a weighted sum of the hidden activations and squashes it with a sigmoid, which gives a number between 0 and 1: the network's probability that this point is class 1.
Loss. The scores the miss as a single number. The demo uses binary cross-entropy, which is small when the prediction is confident and right, and large when it is confident and wrong.
Backward pass. asks, for every weight, how much the loss would change if that weight moved a little. That number is the weight's gradient. It is computed from the output backward, one layer at a time, reusing the error already computed for the layer ahead. This is the bookkeeping that gives each weight its share of the blame.
Update. moves each weight a small step against its gradient: new weight equals old weight minus the learning rate times the gradient. The learning rate is a you pick, and the slider in the demo sets it.

Repeat those four beats over many examples and the network learns. That cycle is the . The demo's Play button runs it with mini-batches of 8 points: the gradients of 8 examples are averaged before each update, which is what real training does with a in the thousands.

What is the chain rule in neural networks?

The chain rule is the calculus fact that makes backpropagation cheap. If the loss depends on the output, the output depends on a hidden unit, and the hidden unit depends on a weight, then the slope of the loss with respect to that weight is the product of the slopes along the path. Nothing in the chain needs to know about the rest of the network. Each factor is local.

For the demo's network, with ŷ the prediction, y the true label (0 or 1), h_j the activation of hidden unit j, v_j its weight to the output, and x_i an input, the two kinds of gradient are:

dL/dv_j  = (ŷ - y) * h_j
dL/dw_ij = (ŷ - y) * v_j * (1 - h_j^2) * x_i

Read the second line left to right and it is the path from the loss back to the weight. (ŷ - y) is the error at the output, which is what the sigmoid and cross-entropy pair collapse to. v_j is the weight the signal passed through on its way forward. (1 - h_j^2) is the slope of tanh at that unit. x_i is the input value. The inspector shows exactly these factors with their live numbers, and the demo asserts at load that the product it displays equals the gradient the update uses, and that every analytic gradient matches a finite-difference check to within 0.00001.

The product form explains the saturation caption. If h_j is 0.99, the slope factor is about 0.02, and every weight feeding that unit gets 2% of the gradient it would otherwise get. Stack many layers and many such factors and the gradient reaching the early layers can vanish. That is why modern networks lean on activations like ReLU, whose slope is 1 over half its range, and on tricks like residual connections that give the gradient a path around the products.

Backpropagation vs gradient descent

People use the two names interchangeably, and they are different jobs. Backpropagation computes the gradient: which direction each weight should move and how strongly. Gradient descent uses that gradient to take the step. In the demo, the backward phase is backpropagation and the update phase is gradient descent. Neither works alone. Gradient descent without backpropagation has no gradient to follow, and backpropagation without an update changes nothing.

The gradient descent demo shows the second job on its own: a ball rolling down a loss landscape whose gradient is simply known. This demo shows where that gradient comes from when the landscape is a neural network with 13 weights and the only thing you can measure is how wrong one example was.

Why do neural networks need hidden layers?

Because one unit can only draw one straight boundary. A single tanh unit feeding a sigmoid output is a monotone function of one weighted sum of the inputs, so its decision surface is a half-plane. That is enough for the LINE dataset and useless for XOR, where opposite corners share a class and no single line separates them. With three hidden units the network has three lines to combine, and the output unit can carve the square into the two diagonal regions XOR needs.

The demo checks this at load with three runs of 1,500 mini-batch steps at the default learning rate of 0.30, each from the same seeded starting weights:

LINE with 1 hidden unit: 100% accuracy.
XOR with 1 hidden unit: 66%, and it stays there. The loss curve flattens because no setting of a one-line boundary does better.
XOR with 3 hidden units: 100%.

Nothing about the algorithm changed between those runs. The same forward pass, the same loss, the same backward pass, the same update. Only the number of weights changed. Hidden layers are what give backpropagation something worth blaming.

From 25 weights to billions

This network has at most 25 weights, counting the biases, when the hidden slider is at 6. A language model has billions (the 7B in a model name is that count), and it is trained with the same algorithm. Every one of those weights gets a gradient from the same chain-rule bookkeeping, computed layer by layer from the output backward, and every one takes a step against it. The attention layers in a transformer are more elaborate functions than tanh, but they are differentiable, so the chain runs through them the same way. Fine-tuning methods like LoRA work by choosing which weights get a gradient at all.

Backpropagation runs in training only. Once a model is deployed, answering a prompt is a forward pass and nothing else. No gradients are computed and no weights move, which is why a model does not learn from your conversation unless someone trains it again.

The method became the standard way to train multi-layer networks after Rumelhart, Hinton, and Williams published "Learning representations by back-propagating errors" in Nature in 1986. The underlying math had been worked out earlier by several people; the 1986 paper is the one that showed hidden units learning useful internal representations, and the name stuck.

What the demo simplifies

Nothing in the math. The forward pass, the cross-entropy loss, every gradient, and the update are computed by hand in the page, with no library. Every number on screen falls out of that computation.
Scale. Two inputs, one hidden layer, 80 points, mini-batches of 8. Real training uses millions of examples and batches in the thousands, and the loss shown in the HUD is the mean over all 80 points after every update, which would be far too expensive at scale.
Plain gradient descent. The update is the textbook rule with a fixed learning rate. Real optimizers like Adam keep running averages of past gradients to pick a step size per weight. The gradient they consume comes from the same backward pass.
Seeded randomness. The starting weights, the data, and the batch order come from fixed seeds so two loads play out identically. Changing the dataset or the hidden-unit count resets the weights to the same seeded draw.

Frequently asked questions

What is backpropagation in simple terms?

It is how a neural network figures out which weights to blame for a wrong answer. Starting from the error at the output, it works backward through the layers and gives each weight a number, its gradient, that says which way to move it and how much. A separate step, gradient descent, then moves the weights.

What is a forward pass?

The forward pass is data flowing from input to output. Each layer takes the previous layer's values, multiplies by its weights, adds its biases, and applies an activation function. The last layer's output is the prediction. It runs in both training and inference; the backward pass runs in training only.

What is the difference between backpropagation and gradient descent?

Backpropagation computes the gradient of the loss with respect to every weight. Gradient descent uses that gradient to update the weights. In the demo, the BACKWARD phase is backpropagation and the UPDATE phase is gradient descent.

What is the chain rule in neural networks?

The rule that the slope of a chain of functions is the product of the slopes of each link. A weight deep in a network affects the loss through every layer after it, so its gradient is the product of local slopes along that path. Backpropagation is the chain rule applied systematically, reusing each layer's result for the layer before it.

Does backpropagation run when a model answers a prompt?

No. Answering a prompt is a forward pass only. Backpropagation runs during training, when the model's weights are being changed. A deployed model's weights are frozen.

Who invented backpropagation?

The 1986 Nature paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams made it the standard method for training multi-layer networks. The underlying technique, reverse-mode differentiation, had been derived independently by others before that.

Key takeaways

Backpropagation does not change any weights. It only computes gradients. Gradient descent is the step that moves the weights, and the two run back to back in every training step.
Each weight's gradient is a product of local numbers, so one near-zero factor kills the whole product. A saturated tanh unit has a slope near zero, and almost no gradient reaches the weights behind it. That is the vanishing gradient in miniature.
One hidden unit can only draw one straight boundary. In the demo, XOR with 1 hidden unit stalls at 66% while 3 hidden units reach 100% with the same algorithm, the same data, and the same learning rate.
The algorithm is the same at every scale. This network has at most 25 weights. A language model has billions, and it is trained with the same four beats: forward, loss, backward, update.
Backpropagation runs during training only. When a deployed model answers a prompt, only the forward pass runs.