Neural Networks & Transformers Explainer

Gradient Descent & Optimization

Visualizing the Loss Landscape & Neuron Activations

In a neural network, learning is the process of minimizing a cost function. We visualize this cost as a loss landscape in high-dimensional parameter spaces. By taking steps proportional to the negative gradient (w ← w - η ∇C), the weights roll down the curve. Watch how the learning rate η affects this trajectory: normal values converge smoothly, while a learning rate above 0.08 oscillates wildly and climbs up the landscape (divergence!). Simultaneously, observe the output layer activations converging onto the target digit "3".

Current Epoch

Simulation steps run

Mean Squared Error (C)

62.9200

↓ -0.0000

Active Parameters

13,002

MNIST 784→16→16→10

Ready to run

The Loss Landscape (2D Projection)

The red line plots the loss function C(w) = 13(w - 2)^2 + 0.1. The yellow dot is the network's weight parameter. Notice how high learning rates overshoot the minimum.

Activation Distribution (Digit Output Neurons)

As the network minimizes loss, the output activation of target neuron "3" should rise to 1.0 (indicating confidence), while wrong classes converge to 0.

Attention Mechanisms & Semantic Embeddings

Demystifying Self-Attention & High-Dimensional Superposition

In transformer architectures, words are mapped to a high-dimensional semantic space (word embeddings). Through Self-Attention, tokens communicate. Each token generates a Query, Key, and Value. By computing dot products (softmax(Q K^T / √d_k)), tokens route information. Observe how the word "bank" shifts its embedding vector: originally ambiguous, attention weights pool context from "river", pulling the representation into a hydrology-related semantic space.

Embedding Dimensions (d_model)

12,288

GPT-3 scale vector space

Attention Heads (h)

Parallel routing pipelines

Softmax Output Range

[0.0, 1.0]

Attention weights normalization

Self-Attention Weights: α = softmax(Q K^T / √d_k)

Attention weights representing how the query token (from the dropdown) looks at the keys in the sequence: "The bank of the river". Note the strong link between "bank" and "river".

Vector Superposition & Contextual Update

Compare the original static dictionary embedding (teal) with the attention-updated contextual representation (yellow) along key semantic dimensions.