Thomas
Duffy

Polly had data. It did not have analytics. Everything lived in the operational systems that produced it — MongoDB behind the product, PostgreSQL behind another service, files arriving over SFTP — and there was no way to ask a question that spanned any two of them.

I was one of two founding engineers on the platform that changed that, and I built it from the ground up: the architecture, the infrastructure it runs on, the ingestion into it, the transformation layers through it, and the delivery out of it. Bronze for raw landing, Silver for governed and conformed data in Unity Catalog, Gold for the customer-facing aggregated models and predictive results that everything downstream reads from.

Two things came out of it. Polly can now run analytics on its own data — the internal capability that didn't exist before. And clients receive their data through multiple channels: dashboards they log into, reports delivered on a schedule, and direct data delivery into their own systems.

I own the repository, its architecture, and its release process. Every system further down this page — the delivery channels, the agent, the classifier, the deploy tooling, the knowledge graph — runs on this foundation.

Stack

Databricks, AWS, Spark, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, dbt, Redshift, Terraform

Sources

MongoDB, PostgreSQL, Amazon S3, external SFTP feeds, web-scraped public data

Owned

Architecture, infrastructure, release process, repository

The legacy dbx deployment path made every release a question of what state the workspace was actually in. Jobs drifted between staging and production, and nobody could tell you with confidence which.

There was no off-the-shelf answer that fit, so I built one. Infrastructure as code first: every job and pipeline became a typed Pydantic definition, so the repository is the source of truth for what should exist rather than a rough description of what probably does. Deploys turned into idempotent name-based upserts rather than creates, rollout follows dependency order, and drift between environments is detected automatically instead of discovered during an incident.

Around that I built the team's CI/CD process — automated test gates before anything promotes, a defined release cadence rather than deploys whenever someone felt ready, and a governed path from staging to production that applies to everyone including me. Deployment stopped being an act of institutional memory.

The real outcome was social rather than technical: the barrier to shipping a pipeline change dropped enough that other engineers stopped routing changes through me. I own this release process, along with the architecture of the repository it governs.

Stack

Python, Pydantic, Databricks SDK, Terraform, GitHub, CI/CD, Delta Live Tables

Scale

~80 jobs and pipelines across staging and production

Ingestion is the part of a platform that never stops being wrong. Sources change, volumes grow, and the approach that was right at the start stops being right about eighteen months in. I've rebuilt Polly's ingestion layer three times.

The first generation was hand-built Python pipelines — fast to write, total control, and increasingly a maintenance tax as the number of sources climbed. The second moved to AWS Glue, trading some of that control for managed infrastructure and less operational surface.

The third is Estuary Flow reading the PostgreSQL write-ahead log directly. Change data capture rather than scheduled extraction, which cut replication latency and deleted an entire category of custom ingestion code along with the maintenance it demanded.

The useful thing isn't the final answer. It's having built all three and knowing concretely what each costs — where hand-rolled beats managed, and where paying for managed infrastructure buys back more than it costs.

Stack

Python, AWS Glue, PySpark, Estuary Flow, PostgreSQL, Delta Lake

Sources

MongoDB, PostgreSQL, Amazon S3, external SFTP feeds, web-scraped public data

A dashboard assumes the client wants to log into your product. Plenty don't. They want files landing in their own systems on a schedule, in formats their existing processes already understand, so the data shows up where their analysts and their downstream tooling already live.

I built the delivery system for that end to end. Reports generate as CSV and PDF off the Gold models and go out to client-owned SFTP endpoints on a schedule — no manual export step, no analyst assembling a file on a Friday.

The other direction needed solving too, so I built and operate SFTP hosting on our side. Clients who would rather pull than be pushed to can connect directly and take what they need on their own cadence.

The third channel skips files entirely. Delta Sharing gives clients live read access to the Gold tables themselves — no export, no copy, no transfer window. They query the data where it already lives, which is both the lowest-latency option available and the cheapest, since nothing is duplicated or moved to serve it.

It's the same infrastructure problem as ingestion pointed the other way: scheduling, credentials, formats, failure handling, and a delivery guarantee. Getting data out reliably turns out to be roughly as hard as getting it in, and it's the half that clients actually notice when it breaks.

Stack

Python, SFTP, CSV and PDF generation, Delta Sharing, Delta Lake, Redshift

Channels

Scheduled push to client endpoints; hosted pull from ours; live Delta Sharing access

Dashboards answer the questions you anticipated. The analytics agent exists for the ones you didn't — letting clients interrogate their own data conversationally rather than filing a request and waiting on the analytics team.

Draft — needs your detail: what the agent can do, who uses it, whether it shipped, and how design responsibility split with the wider team.

Owned

Design

Every lender names their loan pipeline stages differently, and analytics across clients needs one vocabulary. The system that solves this was co-built with the data science team: a three-tier pipeline that tries a deterministic lookup first, then a cross-client SQL consensus, and only pays for LLM inference on genuinely novel cases.

My piece was making the model tier trustworthy. A language model that is usually right is not the same as one you can run unattended over client data, and the gap between those two is entirely engineering. Each classification samples five times and resolves by majority. Every prediction carries a confidence score and flags itself for human review below threshold. Results land in an append-only Unity Catalog ledger, so nothing is silently overwritten and every decision stays traceable to the run that produced it.

The accuracy problem turned out to be linguistic rather than statistical. General-purpose models read mortgage acronyms as ordinary English and confidently guess wrong. Enriching each prompt with the lender's own preceding and following statuses, pulled with SQL window functions, gave the model the sequence context to disambiguate them.

Stack

Databricks Foundation Models, Unity Catalog, PySpark, SQL

Owned

The reliability and auditability layer, and the prompt-context design

Shared

Overall system co-built with the data science team

At DataPrime the modelling was rarely the hard part. Getting a model to a client meant standing up the environment it ran in, and doing it inside SOC 2 boundaries where the data was healthcare records and personally identifiable information.

I provisioned that infrastructure with Terraform so environments were reproducible rather than hand-assembled, and owned them through deployment. Where client access restrictions blocked the straightforward path, I designed a compliant handling system that met the constraint instead of routing around it.

The models themselves projected freight shipping costs and logistics — forecasting work whose value depended entirely on it being delivered somewhere clients could actually use it.

Stack

Terraform, Python, cloud infrastructure

Context

SOC 2, healthcare and PII data

A platform this size accumulates knowledge that lives in the heads of whoever built it. "What breaks if I change this column?" was a question only a few people could answer, and answering it took an afternoon.

I built a machine-navigable knowledge graph over the pipelines — lineage, blast radius, field-level mapping across ingestion, Delta Live Tables, and dbt — then wrapped it in agentic tooling: five custom Claude Code skills and three specialized agents covering lineage lookup, guided deployment, workspace drift reconciliation, safe deprecation, and dependency modernization.

Tribal knowledge became self-service. New contributors could answer impact questions on their first week without interrupting anyone.

More recently the emphasis has moved from documenting the platform to making it workable by agents. The skills and tooling I've built encode how this repository actually behaves — its conventions, its deploy path, its dependency structure — so a coding agent operating in it works from real context instead of inferring from filenames. Whatever's in the editor, Claude or ChatGPT or whatever comes next, arrives at the same authoritative answer a senior engineer on the team would give.

This has become the part of the work I'm most interested in. Every workflow that gets automated is one fewer thing that depends on somebody remembering it, and the velocity gain compounds across the whole team rather than stopping with the person who built the tool.

Stack

Python, Claude Code, dbt, Delta Live Tables