标签 · Testing — Glean

Recent picks

5picks · chronological

09-20

How Anthropic scaled test impact analysis as agentic coding broke CI

Anthropic's CI absorbed a 25x increase in jobs over six months: Claude now writes 80% of the code, per-engineer quarterly output is 8x the 2021-2025 rate, and the test suite grew 10x while headcount barely moved. The bottleneck moved from writing code to PR review to CI, landing on the test impact analysis service that picks which tests run on each change. Because v0 needed a single writer to keep per-test history ordered, it ran as one process and could not be sharded. Three patches followed: doubling cores bought 70 days, per-package sharding 29 days, and daily restarts under a day. The rewrite moved history into an in-memory data store: any listener worker appends results to a journal and exits stateless, a small consumer rolls the journal into per-test history every few seconds, and the selector queries it. One engineer finished in three weeks.

claude.com · 10 min · Agentic Coding · CI/CD · Engineering Productivity

08-26

Mutmut: A Python Mutation Testing System

The author, building Python libraries, realized that even 100% code coverage can miss boundary conditions and error handling. Mutation testing works by making tiny changes to source code (e.g., turning < into <=) and then running the test suite; if the tests still pass, the mutation is considered unkilled, revealing weaknesses. After evaluating Mutpy and Cosmic Ray, the author decided to write Mutmut from scratch. A key design choice was using the baron library (later replaced by parso) to enable lossless AST round-tripping, so mutations can be written back to disk without disturbing formatting. The author attempted to intercept module loading via import hooks to mutate in memory and parallelize, but abandoned this after discovering that the Python import system forces reimplementing every loader. The disk-based serial approach was kept for its simplicity and flexibility across test runners. Applied to tri.declarative and tri.struct, Mutmut found untested edge cases and dead code even with 100% coverage, genuinely improving the test suites. This is a first-hand, honest account for engineers interested in Python testing tooling.

kodare.net · 7 min · Developer Tools · Mutation Testing · Python

07-11

Agentic test processes: from chip design to AI workflows

Drawing from his experience at chip company Centaur, the author compares test processes that scale well with LLM agents: no code review by default, heavy reliance on fuzzing, and a dedicated test team. He argues that while LLMs are poor at writing tests directly, directed fuzzing with LLMs can find real bugs in minutes. The article highlights the high variance of LLM outputs—benchmark rankings often flip with minor task changes—and cautions against over-reliance on aggregated metrics. Through examples like building a superhuman board game AI, he advocates systematic data-driven iteration over prompt tricks. Targeted at engineers interested in AI-assisted development, testing, and agent workflows.

danluu.com · 91 min · AI Engineering · Benchmarks · Developer Tools

07-11

Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island

Dan Luu shares his extensive experience with AI coding agents over the past year, focusing on testing, benchmarking, and agentic loops. He compares fuzzing vs. LLM-driven bug finding, finding fuzzing faster with lower false-positives; evaluates 'caveman mode' with 50 runs showing inconsistent savings; highlights high variance in LLM benchmarks, making public evals nearly useless for individual users. He also discusses automated PR generation from support tickets, multi-persona false-positive reduction, and challenges in data analysis and autonomous loops. For engineers interested in real-world effectiveness of AI coding tools.

danluu.com · 91 min · Agent Engineering · Fuzzing · LLM Benchmarking

06-20

Skillify: turn every agent failure into a permanent structural fix

Garry Tan presents 'Skillify': a methodology that turns every AI agent failure into a permanent structural fix instead of relying on prompt tweaks or apologies. Using two real failures—an agent bypassing a local script for calendar search and doing mental timezone math—he walks through a 10-step verification checklist: SKILL.md contract, deterministic script, unit tests, integration tests, LLM evals, resolver trigger, resolver eval, reachability audit, smoke test, and brain filing rules. This workflow is built into GBrain, an open-source knowledge engine that ensures agent judgment improves permanently and verifiably. Targeted at developers frustrated by recurring agent mistakes.

x.com · 22 min · Agent Architecture · Agents · Ai-Memory