The Universal Weight Subspace Hypothesis

Prakhar Kaushik

CONTEXT JAMMING / EXPERIMENT 01 / JULY 2026

Filtering Training Data to the Universal Weight Subspace

Neural networks keep arriving at the same narrow geometry. The experiment asks a blunt question: why make the optimizer rediscover the road every time?

Enter the geometry Read Kaushik et al. ↗

EVIDENCE BOUNDARY

One controlled mechanism check. One real digits MLP. No ViT result. No 7B result. The page keeps those facts separate.

CONCEPTUAL RECONSTRUCTIONThe corridor is the hypothesis made operational—not a measured loss landscape.

SECTION 01 / 13THESIS / OPERATIONALIZATION

The hidden geometry of training

A billion parameters. A much smaller road.

The parameter count describes the room. It does not tell us where training walks. Kaushik and colleagues stacked more than 1,100 trained models—ViTs, Mistral LoRAs, LLaMA-3-8B models, ResNets—and found sharp, layer-wise spectral decay inside shared architectures.

The original result is about weights. This project turns it into a training intervention: measure whether each example's gradient points into the recovered basis, then filter, reweight, or constrain accordingly.

The room is huge. The traffic keeps choosing the same eight exits.— The thesis, compressed

SECTION 02 / 13PAPER / ALGORITHM 1 + APPENDIX B.1

The universal weight subspace hypothesis

Stack the models. Center the layers. Watch the variance collapse.

PAPER FINDING

For each same-shaped layer, flatten the parameters from many same-architecture models into rows. Subtract the feature-wise mean. Run PCA—the paper's practical order-1 HOSVD case—and retain the smallest rank that crosses a variance threshold.

ORDER-1 HOSVD / PCAKAUSHIK ET AL. · ALG. 1

X_c=X-\mu,\qquad X_c=U\Sigma V^\top,\qquad U_k=V_{1:k}^\top

The page's implementation operates layer by layer and can analyze absolute weights or updates from a common reference.

REAL SMALL-SCALE MEASUREMENTFC2.WEIGHT · TEN SOURCE MODELS

How much geometry survives?

Choose a rank. Rust bars are retained directions; pale bars are the residual. The eighth direction crosses the pre-registered 90% variance threshold.

Retained directions k8

94.9%cumulative centered variance

PC123.1%

PC216.8%

PC312.4%

PC410.9%

PC510.1%

PC69.4%

PC76.4%

PC85.9%

PC95.1%

Component ratios are read from the downloadable serialized subspace. This is one middle layer of a digits MLP—not an LLM layer.

SECTION 03 / 13PROJECT HYPOTHESIS / NOT A PAPER RESULT

Why pre-filtering matters

The optimizer pays for directions the model may later abandon.

Score before spending

Measure how much of each example's gradient energy falls inside the recovered basis.

Protect coverage

Rank inside labels or task strata. Raw global filtering can erase a minority class.

Constrain what remains

Project updates toward the shared basis—or learn only coefficients over frozen directions.

UNPROVEN AT SCALE“Waste” is the hypothesis under test. A discarded direction may contain the very novelty a new task needs.

SECTION 04 / 13IMPLEMENTED KERNEL / ILLUSTRATIVE DEMO

The gradient alignment scorer

Turn every training example into a geometric question.

PER-EXAMPLE SCOREIMPLEMENTED

s_i=\frac{\lVert U_k^\top g_i\rVert_2^2}{\lVert g_i\rVert_2^2}

g is the selected per-example layer gradient. The score lies between zero and one and is invariant to gradient scale.

A high score says direction, not usefulness. Easy examples, mislabeled examples, and duplicated examples can still score highly. Geometry becomes one feature in the sampling decision—not the whole decision.

ILLUSTRATIVE BATCHCALIBRATED TO THE REAL SCORE RANGE

Keep top fraction60%Preserve class coverage

Y00.0005DROP

Y10.0178KEEP

Y20.0161KEEP

Y30.0115KEEP

Y40.0012KEEP

Y00.0005KEEP

Y10.0158KEEP

Y20.0008DROP

Y30.0023KEEP

Y40.0011KEEP

Y00.0017KEEP

Y10.0066DROP

Y20.0130KEEP

Y30.0185KEEP

Y40.0035KEEP

Y00.0106KEEP

Y10.0151KEEP

Y20.0115KEEP

Y30.0010DROP

Y40.0007DROP

15 / 20 examples retainedmean retained alignment 0.0094label coverage protected

The displayed examples are deterministic teaching inputs, not hidden rows from the digits dataset. Their scale is bounded by the observed real-data minimum and maximum. The class-coverage switch encodes a failure found during the smoke run.

SECTION 05 / 13CONTROLLED ABLATION

Four training regimes

One baseline. Three ways to intervene.

Baseline

Uniform data. Full gradients. The control arm.

no intervention

Filtered

Keep the highest-alignment examples inside every class.

60% kept per class

Importance

Sample all examples, but visit aligned examples more often.

group-normalized sampler

Hard constraint

Project most of each update back into the recovered basis.

85% residual removal

SECTION 06 / 13CONTROLLED SYNTHETIC / FIVE SEEDS

Controlled mechanism check

When the geometry is planted, the scorer finds it.

8 / 8rank recoveredknown planted basis

100%selected scoreinformative examples

3.8e−15rejected scoreorthogonal noise

87.5%filtered alignmentbaseline: 33.0%

Selected mean1.0000

Rejected mean3.83e-15

The scorer sees the planted boundary almost perfectly. That proves the instrument can detect geometry deliberately placed in the data.

The control proves that the instrument detects signal deliberately placed in a known subspace. It does not prove that real models contain the same clean boundary.

SECTION 07 / 13REAL SMALL-SCALE / THREE SEEDS

Real-data results on digits

The result was positive in one window—and mixed everywhere else.

92.37%constrained IID @ 5,040baseline: 91.70%

85.93%constrained OOD @ 5,040baseline: 85.26%

+0.74 ppfiltered final OODIID cost: −1.19 pp

64.8%constrained alignmentbaseline: 9.0%

REAL SMALL-SCALE RESULTSMEAN CURVES · THREE DETERMINISTIC SEEDS

Curves come directly from the hash-verified experiment record. Toggle the split and methods; no smoothing or interpolation is applied.

Loading verified result record…

REAL SMALL-SCALE RESULTS

The constraint changed direction. It did not clean the spectrum.

Toggle the target-layer update spectrum. The constrained run places far more energy inside the recovered universal basis—but its matrix effective rank rises from 7.77 to 11.54. Those are different measurements.

Baseline · normalized to the leading singular value

SECTION 08 / 13EPISTEMIC LEDGER

What improved—and what did not

The geometry moved. The generalization claim did not clear the bar.

POSITIVE

Matched-budget checkpoint

The hard constraint improved both IID and noisy OOD accuracy at 5,040 examples seen.

MIXED

Final filtering tradeoff

Filtered data gained 0.74 OOD points and lost 1.19 IID points. Robustness and fit moved apart.

NOT MET

Cleaner spectra

Effective rank rose under the constraint. More basis alignment did not mean sharper matrix decay.

OPEN

Scale and universality

Ten MLP source models do not establish a ViT, LoRA, or frontier-pretraining result.

SECTION 09 / 13REAL SMALL-SCALE / THIRTY REFITS PER RANK

Leave-one-source-out stability

Seven directions stayed close. The weakest one nearly disappeared.

12.95°rank-4 mean angleacross directions

8.90°rank-8 mean anglelooks reassuring alone

71.17°rank-8 mean maximumweakest direction moves

89.81°rank-8 worst maximumnearly orthogonal

MEASURED NEGATIVE FINDING

The average hid the edge.

For each of three independently generated ten-checkpoint collections, the experiment removed one source, refit centered PCA, and compared principal angles with the full basis. That produced thirty comparisons at each measurable rank.

At rank eight, most directions can remain close enough to pull the mean down while the weakest included direction rotates toward orthogonality. A scorer that uses all eight directions inherits that sensitivity.

Rank nine is NOT IDENTIFIABLE after holdout. Nine centered held-in checkpoints support at most eight independent PCA directions. The record does not pad the basis or change the protocol to manufacture an answer.

SECTION 10 / 13PYTORCH PACKAGE / VERIFIED KERNELS

Implementation

Extract the basis once. Then choose where to intervene.

Same-architecture checkpoints

→

Layer-wise centered PCA

→

Per-example gradient scores

→

Filter / reweight / constrain

→

Measure accuracy + spectra

MINIMAL EXTRACTION PATH

subspace = extract_layer_subspaces(
    checkpoints,
    parameter_names=["fc2.weight"],
    variance_threshold=0.90,
    max_rank=8,
    reference_state=initial_state,
)

scores = score_examples(model, x, y, subspace)
chosen = select_top_fraction_per_group(scores, y, 0.60)

loss.backward()
project_gradients_(model, subspace, strength=0.85)

HASH-VERIFIED ARTIFACT

Download the recovered digits subspace

PyTorch payload containing the basis, layer mean, shape, variance ratios, source-model count, and threshold metadata.

Download digits_subspace.pt Inspect the mini-package ↗SHA-256 · f507d7bc5ac700088d…

α10.70α2-0.30α30.45

PAPER METHOD + IMPLEMENTED HOOK

Freeze the directions. Learn the coefficients.

\Delta W_t \approx U_k\alpha_t

The expensive object is the shared basis. A new task changes only the small coefficient vector. The local package implements this parametrization; the 7B run did not execute on this machine.

SECTION 11 / 13CONTEXT JAMMING EXTENSION

TUCG + CAIRN

A narrow basis is not yet a cognitive geometry—or a memory cell.

TUCG

Sweep the Goldilocks zone

Test k = 16, 24, and 32 across layers. Numerical coincidence is not evidence; held-out behavior and subspace stability decide.

Open TUCG →

CAIRN

Record provenance per update

Log example IDs, score, basis hash, coefficient delta, residual energy, and contradiction outcomes. Then ask whether cleaner attachments actually follow.

Open CAIRN →

SECTION 12 / 13FAILURE MODES / UNKNOWN

Limitations and open questions

The missing experiments are the point.

01
Discovery cost
How few source models can recover a stable basis? Ten models cap centered PCA rank at nine.
02
Novelty suppression
Filtering toward yesterday's geometry may delete the example that creates tomorrow's capability.
03
Layer transport
A score in one middle layer may not predict a useful whole-model update.
04
Selection bias
Alignment ranking can collapse class or task coverage unless the sampler protects it explicitly.
05
Hardware and carbon
No calibrated power sensor was available. Wall time was recorded; energy was left null.
06
Medium scale
The 7B entry point is implemented but unexecuted. This 16 GiB host failed the 32 GiB safety floor.

SECTION 13 / 13ROADMAP / NO RESULT YET

Next experiments

Make the next claim harder to earn.

Diverse-source stability

The ten-source leave-one-out result is unstable. Add architectures, checkpoints, and tasks before trusting the basis.

ViT / LoRA

Run CIFAR-10/100 or Food-101 with at least twenty adapters and a genuinely held-out task.

Score ablations

Compare alignment fraction, aligned gradient magnitude, influence normalization, and coverage constraints.

Medium protocol

On a ≥32 GiB accelerator host, match tokens across baseline, filtered, sampled, and coefficient-only arms.

Read the paper PDF ↗Inspect raw digits JSON Inspect stability JSON Read the agent summary Inspect controlled JSON

DATA RECORD · 3cc738c29955c0ac… · GATE verify_20260715T225801Z_2c4bd74f