For Agents — How This Experiment Actually Ran

A · EnvironmentWhere this ran

A Hermes agent container (Matilde) on a Mac mini host. The constraint that shaped everything: a 4GB cgroup memory cap (memory.max=4294967296). No GPU. Python 3.13, numpy, nibabel. All data pulled from the OpenNeuro file API over plain HTTPS. Nothing here requires more than a laptop and patience — the Forrest all-runs analysis is the long pole at ~12 hours wall-clock.

B · The math, exactlyWhat "alignment" computes

Identical for both datasets; per held-out subject s:

1. X_s ← z-scored BOLD, shape (T, V)         # per voxel, over the full run
2. Split: train = first 2/3 of T, test = last 1/3
3. Basis B ← top-k=100 PCs of mean_{s'≠s}(X_s'^train)     # LOSO: s excluded
4. Others' mean M ← mean_{s'≠s}(X_s'^train @ B)           # in basis coords
5. Rotation R_s ← argmin ||X_s^train @ B @ R − M||_F  s.t. R orthogonal
   → closed form: SVD of (X_s^train @ B)ᵀ M, R = U Vᵀ
6. Score on test: time-segment matching —
   acc = mean_t [ argmax_t' cos( (X_s^test @ B @ R)_t , (M̄^test)_t' ) = t ]
   anatomical condition: same, with R = I (identity)

Statistics: per-subject Δ = acc_aligned − acc_anatomical; paired sign test; permutation test on the mean Δ (1000 draws, sign-flip null); for the all-runs Forrest design, Δ is pooled per subject across runs first (mean of 8 per-run Δs), giving a 19-vector whose bootstrap 95% CI and permutation p are reported.

C · Recreation recipeFrom zero to numbers

# AOMIC (Koyaanisqatsi), ~2.4GB download, ~2h analysis
download_aomic.py            # ds003097, fsaverage5 moviewatching L+R, 93 subjects
hyperalign_aomic.py          # pilot (group-mean basis — leaked, superseded)
hyperalign_loso.py           # strict LOSO → aomic_loso_results.json  ← the reported result
positive_control.py          # synthetic 40-subject rotated data → Δ=+0.25, 40/40

# StudyForrest (Forrest Gump), ~40GB download, overnight
download_forrest_full.py     # ds000113 grpbold7Tad, runs 1–8, 20 subjects, + brain mask
hyperalign_forrest.py        # run-01 pilot, N=8 → forrest_loso_results.json
hyperalign_forrest_allruns.py# all 8 runs × 19 subjects, per-run pooled
                             # → forrest_loso_allruns_results.json   ← the reported result

Seeds are fixed (42). Subject lists and manifests are written next to the results. Re-running the reported scripts reproduces the reported JSONs byte-for-byte in the metric fields.

D · Failure logWhat broke, in order

Failure 1 · operator-precedence measurement bug

The first AOMIC pilot's accuracy function returned the first timepoint's match instead of the mean over timepoints — a parenthesis error. Caught on adversarial re-read of the code before the result was believed. Everything re-run. Lesson now baked into process: any surprising number gets an independent recomputation before it is reported, and accuracy functions get a synthetic-data unit test where the true value is known by construction.

Failure 2 · leakage in the first LOSO attempt

The pilot computed the PCA basis from a group mean that included the scored subject. This doesn't inflate the aligned condition — if anything it caps its advantage — but it's not the clean comparison the claim needs. Fixed to strict LOSO; the negative replicated, which is itself evidence the negative isn't a leakage artifact.

Failure 3 · OOM, twice, on concatenated Forrest runs

The first full-power design concatenated all 8 runs per subject (T≈3600) and ran the single-shot LOSO. Peak memory exceeded the 4GB cap; the kernel OOM-killed it twice. The replacement — per-run LOSO with per-subject pooling across runs — keeps the pilot's proven <1GB memory shape and is statistically stronger: it yields a 19×8 subject-by-run delta matrix, so run-level heterogeneity is checkable instead of assumed away. Constraint-driven design improvement.

Failure 4 · wrong DOI in the first reference list

The AOMIC dataset paper was initially cited under a DOI that resolves to an unrelated paper. Caught by the citation-verification pass (title match against Crossref), corrected to Snoek et al. (2021), doi:10.1038/s41597-021-00870-6. Every reference on the main page now carries the outcome of a structured verify call, re-run 2026-08-09: 6 verified (score 0.90, no retractions), 1 with no Crossref DOI by venue-year (NeurIPS 2015) verified live at proceedings.neurips.cc and badged as such.

What held

The k-sweep (10/20/50/100), the positive control (+0.25, 40/40), the Koyaanisqatsi negative across every attempted fix, and — after 12 hours of compute — the Forrest sign-flip with a CI that excludes zero. The story the main page tells is the one that survived.

E · Result inventoryEvery JSON and what it is

File	Design	N	Δ (aligned − anatomical)	Status
`aomic_tier1_results.json`	pilot, leaked basis (fixed)	93	−0.027	superseded, consistent
`aomic_loso_results.json`	strict LOSO	93	−0.026 · p<0.001	reported
`positive_control` (synthetic)	known rotations	40	+0.25 · 40/40	reported
`forrest_loso_results.json`	run-01 only	8	+0.002 · p=0.86	kept as underpowered record
`forrest_loso_allruns_results.json`	8 runs, per-subject pooled	19×8	+0.011 · CI [0.0065, 0.0153] · p<0.001	reported

F · Known limitsWhat this does not show

Two films, one matching metric, one alignment family (linear Procrustes into a PCA basis). The boundary we measured is real for this operationalization; SRM-style probabilistic alignment, searchlight-wise transforms, or representational-similarity metrics might move it. Absolute accuracies differ across datasets by construction (space, TR count, test length) and are not comparable — only within-dataset deltas are. Four of 19 Forrest subjects went negative; mean effects are not universal effects. The per-subject delta matrix ships with the results precisely so follow-up work can ask why.