Glean 拾遗
Recent picks

2picks · chronological

11-15

How a model leaderboard puts mixed benchmarks on one scale

AIHOT documents how its model leaderboard turns heterogeneous benchmarks into one capability scale. Each pass-rate benchmark is fitted as a logistic ruler with a difficulty and a steepness estimated from the models that took it; a raw score is inverted into x=(logit(pass rate)+e)/A. Pass rates above 95% or below 5% are truncated to a floor or ceiling, same-family benchmarks count once, and any calibration more than two noise units from the rest has its pull on the total capped. Domain scores shrink toward the model's overall level, then combine at 38% coding, 19% research, 19% office work, 14% knowledge, 10% human blind preference, displayed as 50+15z and anchored to 123 frozen reference models. It also states the ranking thresholds, suspension rules and missing-data handling.

aihot.news · 5 min · Benchmark · LLM Evaluation · Measurement
08-09

From Vibes to Data-Driven Development: The AI Engineer Mindset

This article argues that building with LLMs requires a fundamental mindset shift from deterministic to probabilistic systems. The author introduces two concepts: the 'Vibes-Only Trough' (where demos appear to work but lack real-world coverage) and the 'Data-Driven Slope' (where systematic evaluation and user data guide improvements). Success criteria must be specific and measurable, such as 92% agreement with human labels for sentiment analysis, 95% of classifications within 500ms, and 90% match on high-confidence predictions. The key practice is collecting data from every user interaction - starting with simple thumbs up/down feedback and evolving into richer pipelines. The article honestly states that first versions usually fail due to insufficient data, and that performance volatility is inherent to probabilistic systems. The trick is tracking changes systematically and building a culture of iterative improvement based on real metrics. Aimed at engineers moving LLM apps to production and teams needing an evaluation-first culture.

www.aihero.dev · 6 min · AI Engineering · LLM · LLM Evaluation