How a model leaderboard puts mixed benchmarks on one scale
AIHOT documents how its model leaderboard turns heterogeneous benchmarks into one capability scale. Each pass-rate benchmark is fitted as a logistic ruler with a difficulty and a steepness estimated from the models that took it; a raw score is inverted into x=(logit(pass rate)+e)/A. Pass rates above 95% or below 5% are truncated to a floor or ceiling, same-family benchmarks count once, and any calibration more than two noise units from the rest has its pull on the total capped. Domain scores shrink toward the model's overall level, then combine at 38% coding, 19% research, 19% office work, 14% knowledge, 10% human blind preference, displayed as 50+15z and anchored to 123 frozen reference models. It also states the ranking thresholds, suspension rules and missing-data handling.