Eval report · Better AI Playlists
Spotify's AI playlist ignores requests like "only use songs I've already liked." This project compiles that request into a typed constraint, enforces it with code instead of a prompt, and measures the result against a naive one-shot LLM on a 30-prompt golden set — same spec, same validator, same yardstick.
Method
The architecture splits every request into hard constraints (source, counts, durations, year ranges, artist caps — enforced by a deterministic validator, zero LLM) and soft intent (vibe, flow — the agent's quality bar). The eval mirrors that split: constraints are measured, aesthetics are judged, and the judge itself gets audited.
Results
Pass rate per constraint type across the golden set, after the repair-loop and budget fixes. The baseline doesn't fail because it picks badly — it fails because without library access and a validator, most of these constraints are unenforceable.
The repair loop
Only 53% of runs validated clean on the first attempt. The rest shipped correct because the validator returned specific, actionable violations ("Future appears 3 times as primary artist, max is 2; remove 1") and the agent repaired them — up to its attempt budget. That gap between 53% and 99% is the measured value of deterministic validation.
It's also what makes the cheap model safe: Haiku sometimes needs three repair rounds where Opus needs one, but the validator converts model weakness into latency instead of wrong playlists. Model choice became a cost knob, not a correctness risk.
Case study
"A 4-hour liked-songs marathon, max 3 per artist" — a ~65-track assembly — failed repeatedly, and each failure exposed a different flaw. This sequence is the project's best argument that evals drive design:
30 calls fits a 20-track playlist, not a 65-track one. Fix: budgets scale with the assembly size the spec implies.
Haiku mis-summed durations across
combos and burned repair attempts. Fix: a free playlist_stats
calculator tool — agents get calculators, not mental math.
Given a report_infeasible tool, the
agent declared a feasible task impossible after five failed repairs.
The eval caught it — lazy infeasibility claims score as failures. Fix: the tool
demands counted evidence from searches; "assembly is difficult" doesn't qualify.
±5 minutes on a 240-minute request is not what "4-hour marathon" means. Fix: duration tolerance defaults proportional; the compiler pins it only when the user is precise.
Resolution: after all four fixes, Haiku still exhausted its budget — but Opus 5 solved the same prompt on its first finalize: 13 tool calls, 69 tracks, 239.2 of 240 minutes, zero violations, $0.63. The spec reveals assembly size before any spend, so this shipped as free model routing: ≤40 tracks → Haiku, larger → Opus.
Judge audit
A pairwise LLM judge compared agent vs. baseline playlists on pure vibe fit and scored it close (12–10). Before trusting that, the judge was calibrated against ground truth: 10 blinded pairs labeled by the library's owner.
The owner preferred the agent's playlist 7 times, called 3 ties, and never once picked the baseline. The judge agreed with the owner only 4/10 — and in 5 of the 6 disagreements it favored the baseline's canonical, well-known tracks. Conclusion: the judge carries a popularity bias and is reported as context only; the measured constraint numbers stay the headline.
| Pair | Owner pick | Judge pick | Verdict |
|---|---|---|---|
| g15 | Agent | Agent | agree |
| g07 | Agent | Agent | agree |
| g18 | Agent | Agent | agree |
| d24 | Agent | Agent | agree |
| d25 | Tie | Baseline | differ |
| g06 | Agent | Baseline | differ |
| g16 | Tie | Agent | differ |
| g13 | Agent | Baseline | differ |
| a21 | Agent | Baseline | differ |
| g12 | Tie | Baseline | differ |
Caveat, recorded honestly: blinding was imperfect — the owner sometimes recognized their own library, so owner preference partly measures familiarity. For a personal-playlist product, that is arguably the point.
Cost
| Item | Cost | Notes |
|---|---|---|
| Typical agent run | ~$0.05 | Haiku 4.5, prompt caching on every loop turn |
| Large-assembly run (Opus route) | ~$0.63 | 60+ track playlists |
| Full 30-prompt sweep + baseline + judge | $1.43 | the headline numbers above |
| Entire eval program to date | ~$7 | smoke runs, sweep, failure-chasing, model experiment |
First Opus-only prototype ran ~$0.40/run; caching plus validator-backed model downgrade cut it 8×, measured before adoption.
Limitations