Glean 拾遗
Recent picks

2picks · chronological

11-15

How a model leaderboard puts mixed benchmarks on one scale

AIHOT documents how its model leaderboard turns heterogeneous benchmarks into one capability scale. Each pass-rate benchmark is fitted as a logistic ruler with a difficulty and a steepness estimated from the models that took it; a raw score is inverted into x=(logit(pass rate)+e)/A. Pass rates above 95% or below 5% are truncated to a floor or ceiling, same-family benchmarks count once, and any calibration more than two noise units from the rest has its pull on the total capped. Domain scores shrink toward the model's overall level, then combine at 38% coding, 19% research, 19% office work, 14% knowledge, 10% human blind preference, displayed as 50+15z and anchored to 123 frozen reference models. It also states the ranking thresholds, suspension rules and missing-data handling.

aihot.news · 5 min · Benchmark · LLM Evaluation · Measurement
09-29

Claude Sonnet 5.5 launches with 70.6% on Terminal-Bench 4.0

Anthropic announced Claude Sonnet 5.5, the second model in the Claude 5.5 family, positioned as a faster and cheaper complement to Opus 5.5. It generates output 30%+ faster and costs up to 30% less per task than Sonnet 5. On Terminal-Bench 4.0 it scores 70.6% versus Sonnet 5's 10.3%; gains also show on FrontierCode, CursorBench, GDPval-AA and OSWorld 2.1, with some results near Opus 5.5. Token pricing is unchanged at $2 input / $10 output per million, with $0.20 cache reads, but fewer tokens are needed per task. It is the first Sonnet released with cybersecurity safeguards that fall back to Sonnet 5 for high-risk requests, plus anti-distillation classifiers and preserved thinking. Relevant to engineers choosing models and budgeting agentic coding.

www.anthropic.com · 10 min · Agentic Coding · Ai Safety · Anthropic