Train / Test Splits¶

LAION-fMRI comes with predefined train/test splits so that encoding-model and representation-similarity analyses can report comparable generalization scores. The splits implement all three methods of the re:vision initiative:

Method 1 - Independent within-distribution. An 80/20 split that covers the image distribution as fully as possible while penalising train/test pairs that are too close in feature space. Use the tau split.
Method 2 - Out-of-distribution clusters. Five-fold CLIP-feature cluster holdout. Use the cluster_k5_0 … cluster_k5_4 splits and average results across folds.
Method 3 - Out-of-distribution images. Train on the pool’s regular images, test on the 371 held-out OOD stimuli. Use the ood split. Optionally restrict the test set to specific OOD categories with the ood_types= keyword on get_train_test_ids() / get_split_masks().
Random baselines (random_0 … random_4) are also bundled as one shuffled five-fold CV partition - comparing them against tau quantifies how much of a model’s score depends on train/test similarity.

Pools¶

The existing splits are bundled for the shared and five subject pools. An additional pooled pool provides across-subject splits. Pick the pool whose stimulus subset matches your analysis; use list_pools() to inspect the available pools.

"shared" - the 1,121 cross-subject shared images (non-OOD subset of the shared block). Use this when the original study used only stimuli that every subject saw.
"sub-01", "sub-03", "sub-05", "sub-06", "sub-07" - the per-subject pools (1,121 shared + 4,712 subject-unique images = 5,833 images each). Use a subject’s pool when the original analysis was per-subject and used that subject’s full stimulus set.

Across-subject splits¶

Use pool="pooled" when combining data across subjects. These splits cover all 24,681 regular images, counting shared images once. Use the same split for every subject so that shared images are assigned consistently. Available names are tau and cluster_k5_0 through cluster_k5_4.

from laion_fmri.splits import get_split_masks

train_mask, test_mask = get_split_masks(trials, "tau", pool="pooled")

The pooled tau split has 19,745 training and 4,936 test images, with identified duplicate images assigned to training. To reduce train/test overlap from highly similar images, the pooled cluster splits group similar images before five-cluster CLIPA k-means, keeping each group in one fold. Each image is tested once across the five cluster folds; train/test counts within individual subjects can differ.

During construction, similarity constraints increased the first percentile of nearest-training distance from 0.138 to 0.273 in CLIPA while retaining 93.2% of the baseline mean train/test MMD². See the diagnostic figure for the pooled clustering before and after adding these constraints, and generation details for the feature models and thresholds.

Split names¶

Twelve names exist in the shared and per-subject pools - see list_splits().

name	re:vision method	notes
`tau`	Method 1	The MMD-matched 80/20 nearest-neighbour-distance split. Train and test match at the population level (low MMD), but each test image is kept maximally far from its nearest training neighbour in feature space. Single fixed split per pool.
`cluster_k5_0` … `cluster_k5_4`	Method 2	Each split holds out one of five CLIP-feature k-means clusters as the test fold. Test sizes vary with cluster population; train + test always sum to the pool.
`ood`	Method 3	Train = the pool’s regular images, test = the 371 held-out OOD shared images. The test set is identical across pools; train varies. See Method 3 - OOD images below for the 9 OOD categories and how to filter to a subset of them.
`random_0` … `random_4`	baseline	One seeded shuffled five-fold CV partition. The validation folds are disjoint and together cover the full regular pool. Identified duplicate images are kept in the same fold to prevent overlap between training and test data. Use these splits as a baseline for any generalization metric.

See the changelog for the correction to the per-subject random splits and the previous files.

Split sizes:

tau is fixed at 80/20 of the pool: 897 / 224 for shared, 4666 / 1167 per subject, and 19,745 / 4,936 for pooled.
random_* are five-fold CV folds. For shared, random_0 is 896 / 225 and random_1 … random_4 are 897 / 224. For subject pools, random_0 … random_2 are 4666 / 1167 and random_3 … random_4 are 4667 / 1166.
cluster_k5_* test sizes vary with cluster population; train + test always equals the pool size.
ood test = 371 for every pool. Train = the pool itself (1,121 for shared, 5,833 for the subject pools).

Loading splits¶

The simplest path is get_train_test_ids(), which returns the train and test image-id lists. Match those against the label column of the get_trial_info() output, then apply the mask to your betas:

import numpy as np
import pandas as pd
from laion_fmri.subject import load_subject
from laion_fmri.splits import get_train_test_ids

sub = load_subject("sub-01")

# Load betas and trial info as usual (see laion_fmri_package/load).
sessions = sub.get_sessions()
betas_per_ses  = sub.get_betas(session=sessions, roi="laion")
trials_per_ses = sub.get_trial_info(session=sessions)

# Concatenate across sessions (standard idiom).
betas  = np.concatenate(list(betas_per_ses.values()), axis=0)
trials = pd.concat(list(trials_per_ses.values()), ignore_index=True)

# Method 1: within-distribution generalization on the shared pool.
train_ids, test_ids = get_train_test_ids("tau", pool="shared")
train_mask = trials["label"].isin(train_ids).to_numpy()
test_mask  = trials["label"].isin(test_ids).to_numpy()

X_train, X_test = betas[train_mask], betas[test_mask]

Matching is by string equality: train_ids / test_ids are the same strings as the label column entries (e.g. "shared_12rep_LAION_cluster_1003_i0.jpg"), so no normalisation is needed.

For convenience, get_split_masks() collapses those last three lines:

from laion_fmri.splits import get_split_masks

train_mask, test_mask = get_split_masks(trials, "tau", pool="shared")
X_train, X_test = betas[train_mask], betas[test_mask]

Method 2 - five-fold cluster average¶

from laion_fmri.splits import get_split_masks

scores = []
for k in range(5):
    train_mask, test_mask = get_split_masks(
        trials, f"cluster_k5_{k}", pool="shared",
    )
    fit_encoding(features[train_mask], betas[train_mask])
    scores.append(score(features[test_mask], betas[test_mask]))

m2 = float(np.mean(scores))

Method 3 - OOD images¶

The ood split holds out 371 OOD stimuli as a test set, with the train side being the pool’s regular images. Every subject saw all 371 OOD images, so test_ids is the same set regardless of pool; what changes is what counts as “regular” training data.

from laion_fmri.splits import get_split_masks

# Method 3: train on shared, test on OOD-shared
train_mask, test_mask = get_split_masks(trials, "ood", pool="shared")
fit_encoding(features[train_mask], betas[train_mask])
m3 = score(features[test_mask], betas[test_mask])

OOD categories¶

The 371 OOD images span 9 categories. Discoverable at runtime via list_ood_types():

type	n	what’s in it
`shape`	82	abstract shape, colour and digit stimuli
`relations`	72	unusual spatial relations between objects
`unusual`	64	natural scenes with unusual content
`cropped`	60	isolated object cutouts with transparent backgrounds
`selfmade`	30	custom-shot stimuli
`gaudy`	24	highly saturated / patterned scenes
`illusion-classic`	21	textbook visual illusions
`gabor`	10	Gabor patches
`illusion-natural`	8	natural-image illusions

Pass ood_types= to get_train_test_ids() or get_split_masks() to restrict the test set to specific categories - useful when only some OOD types make sense for the original study (e.g. an object-recognition finding probably shouldn’t be evaluated on Gabor patches):

# Just shape / unusual / cropped (positively biased toward objects)
train_mask, test_mask = get_split_masks(
    trials, "ood", pool="shared",
    ood_types=["shape", "unusual", "cropped"],
)

# Or one type at a time, to compare per-category generalization:
per_type_scores = {}
for t in list_ood_types():
    _, m = get_split_masks(trials, "ood", pool="shared", ood_types=[t])
    per_type_scores[t] = score(features[m], betas[m])

Inspection¶

For the full Split object - including splitter / params / variant metadata - use load_split():

>>> from laion_fmri.splits import list_pools, list_splits, load_split
>>> list_pools()
['shared', 'sub-01', 'sub-03', 'sub-05', 'sub-06', 'sub-07', 'pooled']
>>> list_splits()
['cluster_k5_0', ..., 'random_0', ..., 'ood', 'tau']
>>> sp = load_split("tau", pool="shared")
>>> sp.n_train, sp.n_test, sp.split_family
(897, 224, 'tau')
>>> sp.params
{'method': 'min_nn_filter + stochastic_mmd_swap', ...}