GTIN matching

Why dedup matters

Without dedup, price comparison is a lie. Three retailers can list "Sony WH-1000XM5 headphones" with different titles, different MPNs, and different bundled accessories — but they’re the same physical product. A naive comparison picks the cheapest title, not the cheapest product.

Gridscoot deduplicates at ingestion. Every offer is matched to a canonical product before it ever surfaces in a search result. The canonical product is the unit of price comparison; offers are the per-retailer instances of it.

GTIN-first matching

The first match attempt uses GTIN (Global Trade Item Number — also known as EAN, UPC, or barcode). Every retailer feed we ingest is required to publish GTIN where the manufacturer publishes one.

The matching engine:

• Normalises the GTIN (strips whitespace + dashes, pads 12-digit UPC-A to 13)
• Verifies the Modulo-10 checksum (rejects malformed entries with a structured warn)
• Resolves the canonical product by exact GTIN match
• If two retailers publish the same GTIN, they’re collapsed onto one canonical row

// GS1 Modulo-10 checksum verification
export function isValidGtinChecksum(gtin: string): boolean {
  if (!/^\d{13}$/.test(gtin)) return false;
  let sum = 0;
  for (let i = 0; i < 12; i++) {
    const digit = parseInt(gtin[i]!, 10);
    sum += i % 2 === 0 ? digit : digit * 3;
  }
  const expected = (10 - (sum % 10)) % 10;
  return parseInt(gtin[12]!, 10) === expected;
}

Brand aliases

Retailer feeds are inconsistent about brand spelling. “LG Electronics”, “LG”, “lg”, “LGE” — all the same brand, all show up in different feeds. We maintain a brand_aliases table that maps variants to the canonical brand name.

The aliases table is conservative — only verified equivalents land. New aliases require:

• The variant appears in ≥3 retailer feeds for the same product family
• The brand’s own AU site uses one of the canonical spellings
• A manual-review row in manual_review_queue with explicit approval

alias_text         canonical_brand
'lg'               'LG Electronics'
'lge'              'LG Electronics'
'lg electronics'   'LG Electronics'
'sony'             'Sony'
'sony electronics' 'Sony'

Fallback heuristics

When a retailer offer arrives without a GTIN (some marketplace listings; Kogan’s private-label catalogue), we cascade through three fallback heuristics:

• Brand + MPN (manufacturer part number)— exact-match against the canonical product’s MPN field. High confidence when MPN is unambiguous.
• Token-overlap + word-similarity (pg_trgm + word_similarity) — the offer title is tokenised and matched against canonical product titles via Postgres trigram similarity, plurality-aware ("phone" ↔ "phones"). Medium confidence.
• Embedding distance (pgvector, Voyage AI embeddings) — 512-dim embeddings of the offer title compared to canonical product embeddings via cosine distance. Low confidence; used only when the prior two miss.

If all three fallbacks miss with confidence above the floor, the offer is routed to manual_review_queueand never surfaces in search. We’d rather miss a listing than show a wrong one.

Confidence tiers

Every match emits a confidence tier surfaced on every offer:

• high — GTIN-exact match, checksum-verified. Default for well-formed feeds.
• medium — Brand + MPN exact, no GTIN; OR retailer’s GTIN coverage is reliable but matcher hit on fallback.
• low — Token-overlap or embedding fallback only. Surfaced with a UI badge.

How to verify a match

Any canonical product’s match graph is auditable. Visit /products/[id] on the public site and inspect:

• Header: brand + MPN + GTIN (the canonical fields)
• “All offers” table: every retailer offer collapsed onto this canonical row, with per-offer confidence badges
• “Same-SKU vs different-SKU” distinction made explicit (Phase 12.19 ships this)

If you spot a wrong match — two offers that shouldn’t collapse, or two products that should — flag it via the /about contact path. Dedup correctness is one of the few claims we hold ourselves to absolute accuracy on.