Data Engineering / Full-Stack2026

DataHarbor

A dependency-free Python scraping engine paired with a Next.js operator dashboard — fetch, clean, validate, dedupe, export, with live data-quality analytics.

Role: Architect & sole engineer

Python 3.11TypeScriptNext.js 16React 19PrismaSQLiteRechartsZod

The application

Screens, captured from the running app

Booted against its seeded database and walked route by route — these are the real pages, not mockups.

DataHarbor dashboard showing job throughput, record counts and data-quality metrics. — The dashboard leads with data quality, not job count — how much of what came back is usable.

DataHarbor job list with status, target, record count and duration per scrape run. — Every run is a row: status, target, records returned, duration.

DataHarbor job detail view with per-stage metrics and the run's extracted records. — Job detail breaks the run down by pipeline stage, so a drop in yield points at the stage that caused it.

DataHarbor extracted records table with per-field validation status. — Validation state is per field, not per record — a row with one bad phone number is still worth keeping.

DataHarbor structured log view filtered by severity. — Structured logs from the Python runner, surfaced without leaving the dashboard.

The idea

Most scraping projects die the same way. The extraction works, the data is dirty, and nobody can tell you how dirty without opening the CSV. The scraper and the thing that judges the scraper's output are the same tangled script, so quality is invisible until it's a problem downstream.

DataHarbor separates the two: a scraping engine with no opinions about presentation, and an operator dashboard whose whole job is to show you what came back and how much of it is usable.

The engine

Pure Python standard library — no requests, no BeautifulSoup, no Scrapy. That constraint was deliberate. It forced every stage to be small enough to own:

fetcher — retry with backoff, polite rate limiting
parser — built on html.parser rather than a DOM library
cleaner — whitespace, encoding, entity normalization
validator — per-field rules producing a validation status, not a boolean
deduplicator — content-keyed, not URL-keyed
exporter — CSV and JSON

A runner orchestrates them and emits a JobResult carrying both the records and the metrics. Nothing in the pipeline prints; everything returns.

The bridge

A Node seeder spawns the Python pipeline and pipes its JSON output into SQLite through Prisma. Two languages, one process boundary, JSON as the contract — no shared database schema between them, no ORM in Python, no Python in the dashboard.

The Prisma model covers ScrapeJob, ExtractedRecord, ScrapeLog and ExportFile, with status and validation enums and cascade relations, so deleting a job takes its records and logs with it.

The dashboard

Nine screens over Next.js 16 App Router and React 19: job configuration, live run status, the record table with validation state per field, quality analytics in Recharts, log inspection, and export management. Zod validates at the boundary so bad input never reaches Prisma.

Why it runs against a fictional directory

The whole thing is exercised against a generated business directory rather than a live site. That makes every run reproducible — the same input produces the same metrics, which is the only way to tell whether a change to the cleaner actually improved anything. It also means the project can be shown to anyone without scraping a real third party to demonstrate it.

What it demonstrates

Modular, single-responsibility pipeline design; cross-language integration without a shared runtime; relational modelling of a job/record/log domain; and the idea that a data pipeline's quality reporting is a feature, not an afterthought.