Commercial product

Tori Market Intelligence

A commercial Tori monitoring service that collects listings from customer-selected categories, keeps them searchable and records price and availability changes. Each subscription is priced individually by the number of tracked categories.

Role: System Architecture & Data Engineering
Period: 2023—present
Categories: FrontendBackendInfrastructure
Stack: TypeScriptReactViteFastifyPrismaPostgreSQLMinIOCheerioZodDocker

Overview

The product gives customers a repeatable view of the Tori categories they choose to monitor instead of relying on manually saved listing links. Collected listings remain searchable, and later refreshes show what changed, what disappeared and how each price developed.

Each subscription defines the monitored categories. Their category and search URLs become bounded discovery partitions, and a durable scheduler revisits those partitions and every discovered listing according to current activity and source behavior.

Challenge

A live marketplace page is not a stable dataset. Listings are edited, removed and republished, while the source HTML and extraction paths can also change. Keeping only the latest parsed fields would make earlier states impossible to reproduce or reprocess.

Monitoring a changing set of categories cannot be treated as one unbounded request loop. Discovery must be partitioned, resumable and paced, and temporary throttling or server failures must delay work without losing its place or duplicating state.

Listing images repeat across pages and versions and can consume more storage than the normalized records themselves. Raw HTML is also untrusted input, so it must be retained for replay without ever being rendered directly in the operations interface.

Solution

I modelled each subscribed category and its search targets as discovery partitions that collectively cover the customer's selected scope. A PostgreSQL-backed scheduler claims due targets and listings with short leases, allowing interrupted work to become eligible again without clearing queue state.

The worker applies adaptive polling, randomized pacing, retry and backoff for retriable source responses, and conditional requests with ETag and If-Modified-Since during listing refresh. Every meaningful response can produce an immutable raw HTML snapshot by content hash alongside a normalized current listing, listing versions and a separate price timeline.

Images are downloaded into MinIO as S3-compatible, SHA-256-addressed media objects and linked back to listings in display order, so identical bytes are stored once. A Fastify API exposes the normalized catalogue and operational controls to a React/Vite dashboard for crawl management, filtering and change inspection.

Responsibilities

Define the product workflow for subscription-scoped discovery, refresh and history.
Design target partitions that keep selected-category ingestion bounded, resumable and observable.
Define individual subscription pricing around the number of tracked categories.
Implement the Fastify API, parser and PostgreSQL-backed scheduler with durable crawl state.
Persist reproducible raw snapshots, normalized listing records, immutable versions and price history.
Build SHA-256 media deduplication and S3-compatible image storage in MinIO.
Create the React/Vite operations dashboard and Docker-based local and production runtime.

Architecture

Interface

React operations dashboard

Services

PostgreSQL scheduler
Fetch + parse worker
Fastify API

Data

Raw HTML snapshots
Normalized listings
Versions + price history
SHA-256 media objects

Automation

Tori marketplace catalogue
Discovery partitions

Infrastructure

Docker runtime

Catalogue pages into discovery partitions
Tori marketplace catalogue -> Discovery partitions
Partitions enter the durable schedule
Discovery partitions -> PostgreSQL scheduler
Due work is claimed with short leases
PostgreSQL scheduler -> Fetch + parse worker
Reproducible HTML snapshots
Fetch + parse worker -> Raw HTML snapshots
Parsed normalized listing state
Fetch + parse worker -> Normalized listings
Stored responses support parser replay
Raw HTML snapshots -> Normalized listings
Semantic versions and price observations
Normalized listings -> Versions + price history
Downloaded bytes are hashed and stored
Fetch + parse worker -> SHA-256 media objects
Searchable catalogue read model
Normalized listings -> Fastify API
Version and price timelines
Versions + price history -> Fastify API
Stored listing image references
SHA-256 media objects -> Fastify API
Typed operations and catalogue requests
Fastify API -> React operations dashboard
Crawl controls and pipeline state
PostgreSQL scheduler -> React operations dashboard
Worker production runtime
Fetch + parse worker -> Docker runtime
API production runtime
Fastify API -> Docker runtime
Dashboard production runtime
React operations dashboard -> Docker runtime

Technical decisions

Partition each monitored category

Problem: Subscriptions can cover different sets of categories, and each selected scope still has to remain resumable and controlled as listings change.
Decision: Represent the category and search pages in each subscription as managed discovery partitions that feed one normalized listing corpus.
Reason: Each partition can be scheduled, retried and resumed independently while the selected listings remain unified for search and analytics.
Trade-off: Coverage depends on maintaining the partition plan as the marketplace taxonomy and search behavior evolve.

Price subscriptions by monitored coverage

Problem: Every additional category expands discovery, refresh, history and media work, so one flat price would ignore a major difference in operating scope.
Decision: Calculate each subscription individually from the number of Tori categories the customer chooses to track.
Reason: The price follows the clearest driver of collection scope and remains easy to explain before monitoring starts.
Trade-off: Categories differ in size and activity, so category count is a practical pricing basis rather than an exact measure of workload.

Store raw evidence beside normalized data

Problem: Parser changes or extraction mistakes cannot be investigated if only the latest normalized fields survive.
Decision: Persist immutable HTML snapshots by content hash while keeping a separate normalized Listing model for product queries.
Reason: A source response can be replayed and reparsed without fetching the live page again, while the API remains fast and predictable.
Trade-off: Raw snapshots increase storage usage and must be treated as untrusted content rather than rendered directly.

Use PostgreSQL as the durable scheduler

Problem: Target and listing refresh work must survive worker restarts and avoid permanent ownership when a process stops mid-job.
Decision: Persist next-run state in PostgreSQL and claim due work through short database leases.
Reason: Interrupted jobs become eligible again automatically and operational state remains inspectable alongside the listing data.
Trade-off: Lease duration, polling and indexes require deliberate tuning and favor operational simplicity over maximum queue throughput.

Refresh according to source behavior

Problem: Polling every listing at the same fixed interval wastes requests and increases the risk of throttling without improving useful freshness.
Decision: Adapt target and listing intervals to recent changes, respect Retry-After, back off on retriable responses and use conditional request headers.
Reason: Active listings can be revisited sooner while stable or failing work slows down without leaving the durable schedule.
Trade-off: The scheduler carries more state and freshness becomes an explicit policy rather than a single global interval.

Address media by content

Problem: The same image bytes can appear through multiple listing references, versions or source URLs.
Decision: Hash downloaded media with SHA-256, store one object in MinIO and map listings to the shared object and display order.
Reason: Storage identity no longer depends on temporary source URLs and duplicate bytes do not need separate objects.
Trade-off: Every new media response must be downloaded and hashed before deduplication can be confirmed.

Media

Gallery

Outcomes

The service is sold by subscription, with each quote based on the number of Tori categories a customer chooses to track.
Selected-category collection is coordinated through resumable discovery partitions while discovered products feed one searchable catalogue.
Current listing state, immutable versions, price observations and raw source evidence remain distinct, making changes inspectable without sacrificing reproducibility.
The operations dashboard provides one place to control crawling, inspect scheduler state and open a listing's normalized details and history.

Verified highlights

The running system separates the React/Vite dashboard, Fastify API, worker, PostgreSQL and MinIO services.
Raw HTML snapshots, normalized listings, listing versions and price history have separate persistence responsibilities.
The scheduler uses PostgreSQL-backed due times and short leases for recoverable target and listing work.
Refresh behavior includes adaptive intervals, retry/backoff, Retry-After handling and ETag or If-Modified-Since requests.
Listing media is stored as SHA-256-addressed objects in S3-compatible MinIO storage.

Learnings

For marketplace monitoring, category and search pages work best as discovery partitions inside each selected scope; they should not become separate data silos.
A scraper becomes an analytics system only when raw evidence, current records and historical changes have explicit, different ownership.