01
Item response theory
Rasch, 2PL, and 3PL models with maximum-likelihood and Bayesian estimation. Recover latent ability, item difficulty, discrimination, and guessing from response matrices.
torch_measure
A PyTorch library for measurement science. Includes IRT models (Rasch, 2PL, 3PL), computerized adaptive testing, psychometric metrics, and GPU-accelerated estimation.
Capabilities
A complete toolkit for psychometric evaluation of AI systems, from raw response matrices to calibrated ability estimates.
02
Computerized adaptive testing
Item selection, exposure control, and stopping rules for adaptive evaluation. Reach a stable ability estimate in a fraction of the items a fixed test would need.
03
Psychometric metrics
Reliability, information functions, standard errors, and fit statistics: the diagnostics that tell you what a benchmark score actually measures.
04
GPU-accelerated estimation
Built on native PyTorch tensors and autograd, so fitting scales to millions of item responses on the same hardware you train and evaluate models with.
Quickstart
Fit a model in a few lines.
Install from PyPI, load a benchmark response matrix, and fit an IRT model with the same tensors you already use for training and evaluation.
Install & fit a 2PL model
pip install torch-measureimport torch
from torch_measure.irt import TwoPL
from torch_measure.metrics import reliability
# responses: (n_examinees, n_items) binary correctness matrix
responses = torch.load("benchmark_responses.pt")
# Fit a 2PL IRT model on the GPU
model = TwoPL(n_items=responses.shape[1]).to("cuda")
model.fit(responses.to("cuda"), max_epochs=500)
ability = model.ability # latent ability per examinee
difficulty = model.difficulty # per-item difficulty
alpha = reliability(model, responses)
print(f"marginal reliability: {alpha:.3f}")Run an adaptive test
from torch_measure.cat import AdaptiveTest
# Select the next item, update the ability estimate, and stop
# once the standard error drops below the target.
test = AdaptiveTest(item_bank, selector="max_info", stop_se=0.30)
theta = test.run(examinee)
print(f"ability {theta.estimate:.2f} ± {theta.se:.2f} "
f"in {theta.n_items} items")Design principles
Built for careful measurement.
Three commitments shape every part of the library.
Composable by default
Metrics, datasets, and uncertainty estimates work together without requiring custom pipelines.
Measurement-aware outputs
Every output makes its assumptions visible. Results always include uncertainty and comparability information.
Built for community use
Researchers across institutions can adopt, extend, and contribute back to shared infrastructure.
Open-source infrastructure for the next generation of AI evaluation research.