Describe a bike ride. Get a GPX that actually matches.

Route Gen AI turns a plain-English ride request into ranked, Garmin-importable GPX routes on a map, and then lets you edit them by talking to them — “avoid Whitney Way”, “make it ten miles longer”, “add a stop at Olbrich”. It exists because route planners make you draw, and drawing a 40-mile loop that avoids one particular road is tedious in a way a sentence is not.

The technically interesting part is not the parsing. A language model reads “under 2000 feet of climbing” correctly on the first try. The hard part is that a route is a geometric object with physical constraints, and language models cannot be trusted to produce one. So the architecture puts the model in exactly one place, at the front, translating intent — and everything after it is ordinary deterministic code that generates candidates and throws away the ones that do not measure up.

That split is also what makes this evaluable. Because the acceptance criteria are computed, the eval can use the same kind of computation to check them — independently, from the file on disk.

These are unedited runs from the campaign below. Each map is drawn from the GPX file the app wrote; the measurements next to it were recomputed from that file by the eval harness, not read back from the app's own report of itself. The GPX links are the real files.

Featured because they span the product's range, not because they scored well: one floor case, one hard constraint, one waypoint case, and the most constrained request in the set — which is held out. Every case, including the failures, is in the full table at the bottom.

62 requests were written as a versioned set (routegen-eval-v1) with expectations frozen on 2026-09-19. They are split five ways on purpose: straightforward asks, hard combinations, ambiguous asks, requests the system should decline, and requests naming features the product does not have.

Each case is graded on three separate questions, because conflating them hides where a failure actually is:

• Did it understand? The parsed spec is compared field-by-field against the frozen expectation — distance in range, correct shape, correct edit mode, waypoints present.
• Did it produce something valid? The GPX is re-parsed with a real XML parser (the app itself uses regexes) and checked for range-valid coordinates, finite elevations, continuity, and at least two points.
• Does it satisfy the sentence? Distance measured by haversine over the geometry, loop closure, start proximity, closest approach to each requested waypoint, meters still ridden on a road the user excluded, climb against a stated ceiling.

Checks are marked hard (a promise the product makes, and a failure turns the case red) or soft (a preference it tries to honor, and a failure makes the case amber). Nothing is graded by a model. In particular, the model that produced the route is never asked whether the route is good.

Development and held-out sets are reported separately, and live end-to-end runs are reported separately from the offline failure-injection cases — a passing mock says nothing about a real router.

Everything after the parse is deterministic, so the only place run-to-run variation can enter is the one model call. To find out how much, 9 requests were parsed 5 times each with the cache disabled — a mix of crisply specified asks as a control and deliberately vague ones.

The reproducible number is the one about constraints — request type, distance, shape, climb cap, how many waypoints and avoid-places. Those are the fields that decide whether a candidate is accepted. 5 of 9 were identical on all of them across all five runs, and the split is not subtle: every crisply specified request was stable, and every unstable one was a request that genuinely underspecifies the ride or depends on session state.

Below that level, wording moves constantly and mostly harmlessly — the same start is written “Monona Terrace Madison WI” one run and “Monona Terrace, Madison, WI” the next, which geocodes identically. Only 2 of 9 drifted far enough to change the graded outcome. Free-text notes are excluded throughout: the model rewords those every run and they do not affect what gets generated.

The campaign found four distinct defects. This one is worth walking through because the failure was total — the rider got no route at all for a request the product fully supports — and because the held-out set contained the same bug pointing the other way.

Three of the four were the same shape, which is the part worth noticing. None of them was a wrong computation. In each case the program had already worked out the truth — that the router was down, that the lookup failed, that the stretch needs two laps — and then handed the user a sentence that did not contain it. A correct internal state and a misleading message is indistinguishable from a bug to the person using the thing, and no unit test asserting on return values would have caught any of them. They were found by writing down what the user should see and then checking the user-visible string against it.

An eval can be wrong in the same ways the thing it measures can. Three checks in this harness produced false failures on real routes and were corrected before any product change — recorded here because a report that only lists the subject's mistakes is not a report worth trusting. Each correction is pinned by a test asserting both that the legitimate geometry passes and that the broken geometry it exists to catch still fails; every case carries its scoring history in the saved results.

Both columns are the same 62 cases, scored by the same rules. The development set is where the failures were diagnosed; the held-out set never informed a fix, so it is the honest read.

• No route in this campaign was ridden. Every claim here is about geometry and stated constraints. Whether a route is safe, pleasant, or has a shoulder is not something OSM tags and this harness can establish. The project's own history is field-testing — the devlog records 35 phases of riding routes and fixing what the ride exposed — but this report is not that evidence.
• Elevation is not ground truth. SRTM-derived elevations carried in BRouter's GeoJSON output. The app's climb figure is routes/elevation.track_ascent, a model calibrated against RideWithGPS and one barometric .fit file; the devlog records that instruments themselves disagree by 8-16%, so +/-10% is the attainable accuracy. The eval reports the app's number AND an unsmoothed independent sum over the same elevations. Neither is ground truth -- only a barometric ride is.
• One region. BRouter 1.7.10, self-hosted, profile fastbike-quiet, OSM extract tiles W90_N40 / W95_N40 (southern Wisconsin). Nothing here says the product works outside southern Wisconsin; other regions are a tile download, but that is a claim this campaign does not test.
• Avoid-clearance is scored softly. A road at the start may be genuinely unavoidable, so meters-on-road is reported rather than gating the verdict. The number is the evidence; the judgement is the reader's.
• Vague requests are not reproducible. Measured, not assumed: 5 of 9 repeated requests were identical on every constraint-driving field, and the ones that were not are the ones that underspecify the ride. The product cannot ask a clarifying question, so it guesses, and it does not say that it guessed.
• One request type is still misread. An edit that states no length — “route me through Vilas Park and then the Arboretum” — is classified correctly three times in five. When it is not, the app generates a brand-new route and the rider's current one is gone, with nothing to undo back to. The durable fix is recording a new route's parent so undo can recover it; that is designed and not built, because the only evidence for it here is held out.
• Single evaluator. The request set is written by the author. It is deliberately adversarial in places, but it is not a sample of real user traffic, and a real user base would produce phrasings this set does not contain.
• The harness was wrong three times before it was right. Every correction is above and each is pinned by a test, but a first-run number from a new eval deserves less trust than this page's numbers now do, and that applies to anyone else's eval too.

All 62 cases, with the measured outcome and the first hard check that failed. The featured demos above are four of these rows.

The harness lives in the repository next to the app it tests. It drives routes.service.handle_request — the same entry point the web app and the CLI use — so a pass here is a pass for the shipped code path, not for a test double.

Parses are cached by request text, so re-running after a routing-side change costs nothing and scores the identical interpretation. Scoring is a pure function of the saved records, so python -m evals.rescore re-grades every run under the same rules when a tolerance changes.

• Full per-case results, sanitized (JSON) — the file this page is built from.
• The versioned request set with the frozen expectation for every case.
• The scorers — every tolerance and the reason for it.
• The development log — 35 phases of build, ride, fix.