Thomas
Duffy

There was data but no analytics. It lived scattered across the operational systems that produced it: MongoDB behind the product, PostgreSQL behind another service, files landing over SFTP. There was no way to ask a question that touched more than one of those at a time.

I was one of two founding engineers on the platform that changed that, and I built it from the ground up: the architecture, the infrastructure it runs on, the ingestion into it, the transformation layers through it, and the delivery out of it. Bronze for raw landing, Silver for governed and conformed data in Unity Catalog, Gold for the customer-facing aggregated models and predictive results that everything downstream reads from.

Two things came out of it: an internal analytics capability that didn't exist before, and multiple channels for clients to actually get their data — dashboards they log into, scheduled reports, or direct delivery into their own systems.

I own the repository, its architecture, and its release process. Every system further down this page — the delivery channels, the agent, the classifier, the deploy tooling, the knowledge graph — runs on this foundation.

Stack

Databricks, AWS, Spark, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, dbt, Redshift, Terraform

Sources

MongoDB, PostgreSQL, Amazon S3, external SFTP feeds, web-scraped public data

Owned

Architecture, infrastructure, release process, repository

The legacy dbx deployment path made every release a question of what state the workspace was actually in. Jobs drifted between staging and production, and nobody could tell you with confidence which.

Nothing off the shelf fit, so I built the deploy path myself. Infrastructure as code first: every job and pipeline became a typed Pydantic definition, so the repository is the source of truth for what should exist rather than a rough description of what probably does. Deploys turned into idempotent name-based upserts rather than creates, rollout follows dependency order, and drift between environments is detected automatically instead of discovered during an incident.

Around that I built the team's CI/CD process — automated test gates before anything promotes, a defined release cadence rather than deploys whenever someone felt ready, and a governed path from staging to production that applies to everyone including me. Deployment stopped being an act of institutional memory.

The real outcome was social rather than technical: the barrier to shipping a pipeline change dropped enough that other engineers stopped routing changes through me. I own this release process, along with the architecture of the repository it governs.

Stack

Python, Pydantic, Databricks SDK, Terraform, GitHub, CI/CD, Delta Live Tables

Scale

~80 jobs and pipelines across staging and production

I've rebuilt the ingestion layer three times. Sources change, volumes grow, and whatever approach was right at the start usually stops being right about a year and a half later.

The first generation was hand-built Python pipelines — fast to write, total control, and increasingly a maintenance tax as the number of sources climbed. The second moved to AWS Glue, trading some of that control for managed infrastructure and less operational surface.

The third is Estuary Flow reading the PostgreSQL write-ahead log directly. Change data capture rather than scheduled extraction, which cut replication latency and deleted an entire category of custom ingestion code along with the maintenance it demanded.

The useful thing isn't the final answer. It's having built all three and knowing concretely what each costs — where hand-rolled beats managed, and where paying for managed infrastructure buys back more than it costs.

Stack

Python, AWS Glue, PySpark, Estuary Flow, PostgreSQL, Delta Lake

Sources

MongoDB, PostgreSQL, Amazon S3, external SFTP feeds, web-scraped public data

Getting the data to move was the straightforward part. Deciding what it should actually become once it landed was the real job, and that answer almost always started as something a project manager or a client said out loud in a meeting, not anything written down as a spec.

Raw to Silver is mapping and cleanup: enum and field mappings, flattening unstructured sources like MongoDB documents and scraped web data into something conformed and queryable. Silver to Gold is where it turns into an actual answer: model predictions get attached, and the conformed data gets shaped into what a client receives — Delta Shares, filterable data models, CSV extracts, dashboards.

That design work started with people more than code: sitting with project managers, clients, and leadership to work out the business logic behind how a given lender actually operated, then turning that into SQL models built on a real understanding of the data. It's closer to business analysis than engineering some days, and at a company this size it landed on me anyway, along with every test, alert, and quality check behind it. There's no separate data-quality team here.

It started as five models for one client. It's roughly 80 now, across a growing client base and a widening set of use cases. The pipelines that build them are dynamic rather than fixed-batch — refreshed live on a cadence, with some running as streaming builds outright, so the lag between a source system updating and a client seeing it stays close to nothing.

Stack

dbt, Delta Live Tables, Kubernetes, dbt Cloud, SQL, Alembic

Scale

~80 models in production, up from 5 at the start

Plenty of clients don't want to log into a dashboard at all. They want files landing in their own systems on a schedule, in formats their existing processes already understand, so the data shows up where their analysts and their downstream tooling already live.

I built the delivery system for that end to end. Reports generate as CSV and PDF off the Gold models and go out to client-owned SFTP endpoints on a schedule — no manual export step, no analyst assembling a file on a Friday.

The other direction needed solving too, so I built and operate SFTP hosting on our side. Clients who would rather pull than be pushed to can connect directly and take what they need on their own cadence.

The third channel skips files entirely. Delta Sharing gives clients live read access to the Gold tables themselves — no export, no copy, no transfer window. They query the data where it already lives, which is both the lowest-latency option available and the cheapest, since nothing is duplicated or moved to serve it.

It's the same infrastructure problem as ingestion pointed the other way: scheduling, credentials, formats, failure handling, and a delivery guarantee. Getting data out reliably turns out to be roughly as hard as getting it in, and it's the half that clients actually notice when it breaks.

Stack

Python, SFTP, CSV and PDF generation, Delta Sharing, Delta Lake, Redshift

Channels

Scheduled push to client endpoints; hosted pull from ours; live Delta Sharing access

Every data role I've had turned into a BI role too, whether or not that was the actual title. If nobody can see the data, nobody uses it, so I ended up building the reporting layer by default at every one of them.

I've built dashboards inside GUI tools — Databricks' native dashboards, Metabase, Tableau, Power BI — and built custom dashboards from scratch with Apache ECharts when none of those fit what a specific report needed.

The audience was never just internal. Some of it is analytics for the team itself and operational metrics like billing. Some of it goes straight to clients, as the actual interface they use to see their own data.

It started as something only I built. As teams grew, other people started building and owning dashboards of their own — but I'm still the one accountable for the whole layer.

Stack

Databricks Dashboards, Metabase, Tableau, Power BI, Apache ECharts

Audience

Internal analytics and operations, client-facing insight delivery

A dashboard only answers the questions you thought to build it for. I'm building an analytics agent to handle everything else, so clients can ask questions about their own data directly instead of filing a request and waiting on the analytics team.

Status

Currently building

Owned

Design

Every lender names their loan pipeline stages differently, and analytics across clients needs one vocabulary. The system that solves this was co-built with the data science team: a three-tier pipeline that tries a deterministic lookup first, then a cross-client SQL consensus, and only pays for LLM inference on genuinely novel cases.

My piece was making that model tier trustworthy enough to run unattended over client data. Being usually right isn't good enough for that, so I built the reliability layer around it. Each classification samples five times and resolves by majority. Every prediction carries a confidence score and flags itself for human review below threshold. Results land in an append-only Unity Catalog ledger, so nothing is silently overwritten and every decision stays traceable to the run that produced it.

General-purpose models read mortgage acronyms as ordinary English and confidently guess wrong, which turned out to be the real accuracy problem, not the model's underlying statistics. Enriching each prompt with the lender's own preceding and following statuses, pulled with SQL window functions, gave the model the sequence context to disambiguate them.

Stack

Databricks Foundation Models, Unity Catalog, PySpark, SQL

Owned

The reliability and auditability layer, and the prompt-context design

Shared

Overall system co-built with the data science team

Getting a model to a client at DataPrime meant standing up the infrastructure it would run on first, inside SOC 2 boundaries, with the data itself being healthcare records and personally identifiable information.

I provisioned that infrastructure with Terraform so environments were reproducible rather than hand-assembled, and owned them through deployment. Where client access restrictions blocked the straightforward path, I designed a compliant handling system that met the constraint instead of routing around it.

The models themselves projected freight shipping costs and logistics — forecasting work whose value depended entirely on it being delivered somewhere clients could actually use it.

Stack

Terraform, Python, cloud infrastructure

Context

SOC 2, healthcare and PII data

A platform this size accumulates knowledge that lives in the heads of whoever built it. "What breaks if I change this column?" was a question only a few people could answer, and answering it took an afternoon.

I built a machine-navigable knowledge graph over the pipelines — lineage, blast radius, field-level mapping across ingestion, Delta Live Tables, and dbt — then wrapped it in agentic tooling: five custom Claude Code skills and three specialized agents covering lineage lookup, guided deployment, workspace drift reconciliation, safe deprecation, and dependency modernization.

New contributors could answer impact questions in their first week without interrupting anyone, instead of waiting on whoever happened to remember the answer.

More recently the emphasis has moved from documenting the platform to making it workable by agents. The skills and tooling I've built encode how this repository actually behaves — its conventions, its deploy path, its dependency structure — so a coding agent operating in it works from real context instead of inferring from filenames. Whatever's in the editor, Claude or ChatGPT or whatever comes next, arrives at the same authoritative answer a senior engineer on the team would give.

This has become the part of the work I'm most interested in. Every workflow that gets automated is one fewer thing that depends on somebody remembering it, and the velocity gain compounds across the whole team rather than stopping with the person who built the tool.

Stack

Python, Claude Code, dbt, Delta Live Tables