Rosetta Stone · 4 of 30
Derivative
Operator: it does something
In plain English
How fast one thing changes as another changes. asks: if I nudge by a tiny amount, by how much does move, per unit of nudge? It is the slope of the curve at a point. Velocity is the derivative of position with respect to time. In ML, the derivative of the loss with respect to a weight says which way, and how strongly, to adjust that weight.
As code
// finite-difference estimate of dy/dx at x
const f = (x) => x * x;
const h = 1e-6;
const slopeAt = (x) => (f(x + h) - f(x)) / h;
slopeAt(3); // ≈ 6, because d(x²)/dx = 2xWhere the analogy bends: Real ML frameworks do not do this. They use automatic differentiation, which computes exact derivatives via the chain rule.
See it
What it means, field by field
Velocity dx/dt, acceleration d²x/dt², rates of decay; the language of differential equations.
Population growth rate dN/dt; reaction rates; the slope of a dose-response curve.
Sensitivity of the loss to a parameter; the building block of gradients and backpropagation.
Limit of (f(x+h) − f(x))/h as h → 0.
In a research paper
The population grows at a rate proportional to its size, slowing to zero as it approaches the carrying capacity K.
Common misreadings
Cancelling the d’s as if dy/dx were an ordinary fraction. It sometimes behaves like one (chain rule), but it is a limit.
Reading d as a variable named d. In dx, the d means "a tiny change in".
Assuming a derivative exists everywhere. Corners and jumps have no single slope (ReLU at 0 is a famous example).