Self-healing regulatory pipeline

200 legal sources scraped, repaired by AI when they break, and mined for insight.

Role: Senior engineer — architecture, fetchers, LLM stages
Stack: Python Airflow LLM agents PostgreSQL MongoDB Elasticsearch AWS

Problem

Regulation lives on roughly two hundred official and legal portals. None of them have an API, all of them change their markup without warning, and when a scraper breaks it fails silently — the customer just sees a jurisdiction that quietly stopped updating. On top of that, deciding what each regulation means for a customer was a large body of hand-maintained rules: expensive to change and drifting away from how analysts actually reason.

Approach

One fetcher per source, scheduled and validated: a run only promotes its data once the row counts look sane, so a broken parse degrades to "no update" instead of "everything is empty". On top of that sits an AI heal loop — when a fetcher stops returning plausible data, an LLM agent diagnoses the change and proposes the fix, so a markup change is a review rather than an incident. Classification and insight extraction moved from hand-written rules to LLM stages backed by a labelled evaluation set.

Outcome

Report processing now takes the legal team 70% less time than before. The whole scraping layer covering all 200 sources, self-healing loop included, was built in about a week with AI assistance — work that would previously have been months of hand-written parsers. Sources that break heal themselves instead of paging someone, and a classification change is a prompt edit plus an evaluation run rather than weeks of rule-writing.

Trust nothing upstream

Every scraper breaks eventually. The difference between a side project and something two hundred sources deep is what happens when it does. A validation gate between staging and live means a broken parse shows up as stale data, never as corrupted data — always the cheaper failure. The heal loop is the layer above that: it turns “someone has to go and fix the parser” into “someone has to review a proposed fix”.

Where AI actually paid for itself

Writing two hundred bespoke parsers is exactly the kind of high-volume, low-novelty work that AI is genuinely good at — and exactly the kind of work that is miserable and error-prone by hand. Getting that layer done in a week is the clearest example I have of AI changing what is worth attempting, not just how fast you type.

What I can and cannot show

Under NDA: shape and outcome only, no customer data, prompts or internal diagrams.