Competitor Price and Markdown Intelligence Platform

Problem

A mid-market apparel group sells in 25 countries online and through 900 stores. Merchandising sets prices and markdowns in spreadsheets fed by two external vendors. Their data arrives weekly, covers 30% of the assortment and is mapped to the group’s products by hand. Gross margin fell two points because markdowns started too late in some countries and too early in others. The CFO wants daily competitor prices for the full assortment. Legal insists on collection within each country’s law and the competitors’ terms, and on a human who can see why a recommendation was made.

Approach

Three constraints matter, and none is solved by “write a model”. Collection must stay within legal and politeness limits. Matching must reach a measured precision above 95% across 120,000 products. The decision must stay human. The design follows one order. No collection without a registry entry. Raw snapshots first. Parsing as a separate stage. Tiered matching, measured daily. Each recommendation carries its evidence in the same row.

Architecture

• Site registry: robots rules, request rate, country legal status and selector version in one place. A worker does not start without an approved row.
• Collection: Cloud Run Jobs with Playwright and HTTP workers that scale to zero overnight, and fixed egress addresses per country. Proxy rotation is out of scope.
• Raw store: every page is kept as (site, URL, capture time) in locked Cloud Storage for five years.
• Matching: deterministic rules, then embedding similarity with pgvector, then an LLM with structured output only in the grey zone. Low confidence goes to a human queue.
• Warehouse and advice: BigQuery with Dataform layers, a daily price index, and a read-only API with no write path into the pricing system.

Decisions and trade-offs

• Raw first, parse separately. Five-year reproducibility needs an untouched source. A price from three years ago is proven by opening the locked object and parsing it again with that day’s selector version. A lost collection day is never filled from history; it shows as missing. The cost is storage.
• LLM only in the matching grey zone. The kata sizes this at about 20% of items. Asking an LLM for all 120,000 products against 60 competitors would break the budget and make precision impossible to measure. The daily production run uses no LLM. Precision is measured daily against a golden sample. Below 95% in a country, the threshold rises the same day: I give up recall, never precision.
• Advice, not action. A recommendation is a table row that carries its evidence: competitor, date, observed price, raw snapshot and fired rule. The only output is an export file that a person uploads. No service account can write prices, so the rule lives in IAM, not in a policy text. Each applied recommendation is measured four weeks later to calibrate confidence.
• The registry comes before collection, and blocks are respected. A 403, a 429 or a captcha stops work for that site. Its registry status changes and a ticket goes to legal. No proxy rotation, no captcha solving, no IP switching. The cost is coverage: the site shows as missing in the index and the coverage metric drops.
• A fixed budget with hard caps. Collection runs on jobs that scale to zero overnight. The kata’s arithmetic comes to about 9,000 USD a month against a 15,000 budget. The LLM has a daily call cap, BigQuery a byte quota per project. Overruns stop the work and raise an alert; the bill does not grow. Browser-heavy sites are the main threat to that number.
• Keep the two old suppliers for twelve months. Their data covers about 30% of the scope. Comparing it with our index is an independent check of precision, and it is the only history for the elasticity model’s early years. The cost is paying for both feeds for a year.

What I would revisit

• Legal approval per site. Which sites and countries did legal really approve? If a site’s terms ban automated access, it stays out of the registry and coverage drops.
• Browser share. I assumed 20% of pages need a browser. At 40%, collection cost doubles and becomes the largest item.
• Human review capacity. I assumed 1,500 decisions a day. If the real number is lower, thresholds go up and precision is protected over coverage.
• Reviews and social listings. Their legal risk is higher than price pages. If they are not truly needed, I would drop them from scope.

Outcome notes (design assumptions, not measurements)

This is a design exercise, not a running scraper or pricing system. The output is a four-page diagram set, a one-page whiteboard version and a 16-page Turkish PDF with a line-by-line budget. Precision and share figures are targets and assumptions from the brief.

Sources

• Architecture PDF (Turkish, 16 pages).
• draw.io diagrams (English labels): four-page set and a one-page 30-minute version.