News intelligence pipeline

20,000 news items a day, ingested, clustered and classified by LLMs.

Role: Senior engineer — design, implementation and operation
Stack: Python Airflow 3 Kubernetes LLMs PostgreSQL MongoDB Elasticsearch AWS

Problem

Enterprise customers need to know which news actually touches them. The raw feed is around twenty thousand items a day, heavily duplicated — the same story rewritten by fifteen outlets — and almost none of it is relevant to any given company. Volume was the product's biggest liability: more input meant more noise, not more signal.

Approach

A staged pipeline on Airflow 3 running over Kubernetes. Ingestion, deduplication and clustering happen before anything expensive runs, so fifteen versions of the same story collapse into one before an LLM ever sees them. Classification and enrichment stages are backed by LLMs and, crucially, by an evaluation set drawn from real analyst decisions — every prompt or model candidate is scored against it with cost sitting in the column next to accuracy. Each stage is its own task, so a failure is retryable in isolation instead of restarting hours of work.

Outcome

Twenty thousand items a day flow through unattended, and the pipeline's annual LLM bill came down by 85% — from a six-figure sum to a five-figure one. Clustering cuts the volume reaching the expensive stages by a large factor, and more than one stage ended up on a smaller, cheaper model once the evaluation showed it matched the larger one on the task that mattered.

Cheap work before expensive work

The single most useful decision was ordering the stages by cost. Deduplication and clustering are cheap and deterministic; LLM classification is neither. Doing the cheap collapsing first means the expensive stage runs on a fraction of the input, and the quality of the whole pipeline is bounded by how well you cluster, not by how much you spend.

Evaluation is what makes it shippable

Choosing a model is the least interesting decision in an LLM pipeline. What makes a stage safe to deploy is being able to answer “is this better than what we had?” before a customer sees it. Building the evaluation harness first turned model changes into one-line experiments instead of arguments.

What I can and cannot show

This work is under NDA, so this page describes shape and outcome rather than internals: no customer data, no proprietary prompts, no internal architecture diagrams. Happy to walk through the engineering reasoning in an interview.