An agent that reads 24 council websites every night — and is not allowed to make anything up.

The problem

PawMap tells people where they can walk a dog off-leash. The map data — where the parks are — comes from council open-data feeds and is straightforward. The rules are not. Whether a park is off-leash between 7:30pm and 9am, whether dogs are banned within 20 metres of a playground, whether the beach is seasonal: that lives in prose, on 24 different council websites, in 24 different layouts, behind no API at all.

A human can read one council page and extract the rules in ten minutes. Doing it for 3,617 areas, and then doing it again every time a council edits a page, is not a job anyone will keep doing.

This is the shape that keeps showing up in client work. Some authoritative source publishes what you need in a format built for human eyes. You need it as rows. And the moment an LLM is doing the reading unsupervised, the question stops being can it extract this — it obviously can — and becomes how do you know it did not invent any of it. Everything below is the answer to that question.

Spec first: decide what the agent may not do

Before any code, we wrote down the contract. Not a description of the happy path — a list of the things the agent is structurally prevented from doing. Three decisions came out of that, and they shaped every line afterwards:

• Every fact carries evidence. The model does not return a rule; it returns a rule plus the exact sentence on the page that supports it. No quote, no write.
• The agent fills gaps; it never overwrites. If a field already has a value, the agent cannot touch it. It can only turn empty into populated.
• Some fields are off-limits entirely. Trust and status flags stay the domain of structured feeds and human moderators. The agent has no path to them at all.

Writing this first is what let the rest be simple. The extraction logic is a pure module with no database access, so the unit suite covers all of it; the orchestration that touches the DB is separate and boring.

* restrictions gains new deduplicated entries; existing ones are kept.
* dog_policy_status / off_leash_status / trust are never touched —
  status changes remain the domain of structured feeds and moderators.

Every extracted fact must carry a verbatim evidence quote from the page;
facts whose evidence cannot be found in the page text are dropped. The DB
orchestration lives in app.db (enrich-rules command); this module is
pure logic so the unit suite can cover it fully.

"You extract dog-area rules from Australian council web pages. "
"Only report rules the page explicitly states for the specific areas you are given. "
"Never guess, generalise from one area to another, or use outside knowledge. "
"Every entry must include a short verbatim quote from the page as evidence; "
"entries whose evidence is not copied exactly from the page are discarded. "
"Skip areas the page says nothing about."

Guardrails: assume the model will get it wrong

A prompt that says “never guess” is a request. The guardrail is the code that checks. After the model answers, and before anything reaches the database, every fact is re-validated against the page text that was bundled with the request:

• The evidence quote must appear verbatim in the page text — whitespace-normalised, case-insensitive, but otherwise exact. A paraphrase fails. A plausible-sounding sentence that is not on the page fails.
• The area must be one we actually asked about. A hallucinated identifier is dropped rather than inserted.
• Writes are computed as fill-gaps-only, with length caps and deduplication, and the whole plan returns nothing if there is nothing genuinely new.
• Every applied fact is written to an audit log with its evidence quote. That log turns out to matter enormously in section 04.

The result is that the worst case for a bad model response is no data, never wrong data. That is the trade we make on every extraction agent we build: silence is recoverable, confident fabrication is not.

def validate_extraction(
    extraction: RulesExtraction, *, page_text: str, known_area_ids: set[str]
) -> list[AreaRuleFacts]:
    """Keep only facts for known areas whose evidence appears verbatim on the page."""
    haystack = _normalize(page_text)
    valid: list[AreaRuleFacts] = []
    for facts in extraction.areas:
        if facts.areaId not in known_area_ids:
            continue
        if not facts.evidence.strip() or _normalize(facts.evidence) not in haystack:
            continue
        if facts.hoursText is None and not facts.restrictions:
            continue
        valid.append(facts)
    return valid

"""Subscription-mode step 2: validate model output and write the DB.

Reads each <slug>.bundle.json + <slug>.extraction.txt pair from
in_dir and applies the same guardrails as API mode: evidence must
appear verbatim in the bundled page text, hours_text is filled only when
null, restrictions only gain entries, statuses/trust untouched.
"""

Evals: measure the thing the feature actually needs

The easy metric would have been extraction accuracy on a sample. It would also have been useless, because it answers a question nobody was asking. What the product needed to know was: can a user standing in a park see the rules for that park?

So the eval harness measures two different numbers, deliberately kept apart:

Separating them is the whole point. A single blended “coverage” number would have hidden the fact that some of the gap is a data problem the agent can fix and some of it is a geometry problem no amount of model quality will touch. Teams that measure one number end up tuning prompts against a ceiling they cannot see.

The harness is read-only and runs on demand against production. It is about 150 lines. It is also the single highest-leverage thing in this entire build, for the reason the next section is about.

"""Measure how much of the dog-area dataset carries usable rules.

Two numbers matter, and they answer different questions:

  Depth   — what fraction of council areas carry hours / restrictions. This is
            what the council-area screens and the in-area banner show, with no
            proximity constraint. Enrichment moves this number.

  Reach   — what fraction of *places* can show council rules on their detail
            screen. Bounded by geometry, not data: a place only qualifies when
            a mapped zone sits within NEARBY_RULES_RADIUS_M of it. Enrichment
            lifts this only up to that geometric ceiling; more polygon coverage
            is the only thing that raises the ceiling itself.

Read-only. Depth queries the database directly; reach samples the live API.
"""

The feedback loop: the number that would not move

The agent runs nightly on a scheduled job, and reports to Slack when it finishes or fails. For weeks it reported success. Pages fetched, facts extracted, writes applied, no errors.

And depth sat flat at roughly 2%.

Nobody filed a bug. No user complained. Nothing threw an exception. The only reason anyone knew something was wrong is that we had a number that was supposed to go up, and it was not going up.

What we found first: a ratchet that only turned down

The weekly re-ingest that refreshes council data was overwriting the agent's work. Its upsert set the hours field to whatever the council feed contained on every conflict — and council feeds almost never publish hours, so “whatever the feed contained” was NULL. Every re-ingest quietly deleted enriched hours.

Worse, the content-hash optimisation then made it permanent. The agent skips pages whose content has not changed. The page had not changed. So it never re-derived what the re-ingest had just erased. Our own note at the time called it a ratchet that only turned down.

57 areas had ever been enriched with hours. 33 had been wiped — 58% of everything the pipeline had ever produced. Two guardrails, each correct in isolation, combining into silent data loss.

The recovery is the part worth stealing. Because every applied fact had been written to an audit log with its evidence quote, the lost data could be restored without re-running the model or re-fetching a single page. The guardrail from section 02 paid for the whole build right here.

hours_text = excluded.hours_text
      ↓
hours_text = coalesce(excluded.hours_text, council_dog_areas.hours_text)

"""Restore audited enrichment facts the database no longer reflects.

Ingestion owns the council_dog_areas row, so a re-ingest used to overwrite
enriched hours with the feed's (usually absent) value. Every fact ever
applied is retained in rules_enrichment_log with its evidence quote, so the
loss is recoverable without re-running the model or refetching a page.

Fill-gaps-only, matching the live apply path: an area is touched only when
its hours_text is currently null, and the most recent audited value wins.
Restrictions are left alone — the feeds do republish those, so they come
back on their own.
"""

What we found next: three more causes, none of them the model

With the data loss stopped, coverage still under-performed. A second investigation found three more mechanisms, and the commit message opens with the line that sums up most agent debugging we have ever done: “Three separate mechanisms, none of them the model.”

• A blocked page looked like an unchanged page. The fetcher fell back to a cached copy on HTTP errors. That stale text hashed to the stored snapshot, so the run reported “skipped, no change” — in the summary and in the nightly Slack message. Three councils had been silently 403-ing for weeks.
• The page text was mostly furniture. The extractor took every text node outside script and style tags, so nav, footers and alert banners came too — one council yielded 27,349 characters against 3,706 when the cache was last good. A rotating “Current alerts (3)” banner was also changing the content hash, re-billing an LLM call every night for identical dog content.
• A page could lock itself out. One code path recorded a page hash even when zero areas had been considered, excluding that page from every future run.

After the fix: 7 broken sources now visible where only 4 were known, 15 sources with work queued, 329 areas considered, 130 areas gained restrictions.

And then the finding that ended the project

Hours coverage still did not move. It sits at 105 of 3,617 areas — 2.9%.

That number is low, and it is the most useful thing the eval ever told us. Because when we went and looked at why, 11 of the 15 reachable council pages contain zero time references at all. One points at sub-pages. Two simply have no hours published anywhere. The agent was not failing to extract hours. The hours were not there.

“Hours coverage does NOT move, and that is the real finding … The pipeline has already taken what these 24 sources can give — raising hours needs different sources, not more code.”— commit c64bba0

Restrictions, meanwhile, sit at 3,088 of 3,617 areas — 85%. That is the coverage carrying the shipped feature, and it is the number the product needed.

The loop closed by telling us to stop building. Without the eval we would have spent another month tuning prompts against a ceiling made of missing source data. That is what a feedback loop is for — not proving the agent works, but finding out precisely where it cannot.

Why this is about your workflow, not dog parks

Swap council websites for supplier invoices, insurance policies, regulatory notices, or supplier price lists, and nothing about the method changes:

• Spec first. Write down what the agent is forbidden from doing before writing what it does.
• Guardrails in code, not in the prompt. Every claim traceable to source text, validated after the model answers. Worst case is a gap, never a fabrication.
• Evals that measure the business outcome — can the user see the answer — not model-flavoured proxies.
• A loop that runs unattended and is trusted enough that a flat number counts as an alarm.
• An audit log with evidence, because the day something erases your agent's output, that log is the difference between a restore and a rebuild.

If a vendor cannot show you their equivalent of section 04 — the time their own instrumentation caught them — you are looking at a demo, not a system.