The sample comparison matrix: scoring samples across factories
A weighted sample comparison matrix you can copy: criteria, scoring arithmetic worked across three factories, and what to do with the samples that lose.
InventoryX is in development and not available today. ClearStock is the live Shopify app.
Score samples in a spreadsheet, not in your head. When three factories' samples arrive three weeks apart, memory rewrites itself: the plating that scratched becomes "fine, probably cured by production," the packaging that arrived crushed becomes "a shipping thing, not a factory thing." A weighted matrix freezes the judgment at the moment you have the goods in hand, makes the trade-offs visible on one page — Factory C's finish beat A's, A's packaging beat C's, and that's why B won — and gives you something to show a co-founder or a rep instead of a feeling. Below: a copy-ready matrix with weights, the scoring arithmetic worked on three enamel-pin factories, and the cleanup for samples that lose. How to order samples and what a sample predicts about a supplier is owned by our guide to ordering product samples; this article is the side-by-side evaluation. And the other "sample" problem — units that never sell once the sourcing round is over: rep samples kept after the win, demo stock, PR pulls — is a stock-accounting job, owned by the sample inventory guide.
Why gut feel loses to a scorecard
Three failure modes, all common: samples arrive spread across weeks, so memory is the only instrument; attributes trade off against each other, and nobody holds six dimensions in mind at once; and motivated reasoning bends the process toward the factory with the best price — you want the cheap one to be good, so the scratch becomes "probably cured by production." A matrix fixes all three with the same three rules: same criteria for every factory, weights locked before any box is opened, and everything scored the same day.
The matrix: weights and the scoring rule
| Criterion | Weight | What a 5 looks like |
|---|---|---|
| Material & build quality | 25 | Feels at or above spec weight; no flash, no rough seams |
| Finish fidelity | 15 | Plating even, enamel fill flush, print registration clean |
| Color accuracy | 10 | Matches the Pantone reference side by side (Pantone, accessed November 2026) |
| Dimensions & weight | 10 | Within stated tolerance of the spec sheet |
| Retail-ready packaging | 15 | Carded cleanly, barcode prints, survives a desk-height drop |
| Durability quick tests | 15 | Rub, scratch, and bend tests pass as designed |
| Communication during sampling | 10 | Answers within a day; flagged problems before you found them |
Scoring rule: rate each criterion 1–5, then weighted total = Σ(score × weight) ÷ 100, which lands on a 1.00–5.00 scale. Keep criteria to eight or fewer — beyond that, the middle weights blur together. Set your weights before opening any package, score everything on arrival day, and have a second person score blind where you have one.
Worked: three enamel pin factories scored
Illustrative scenario: PIN-EN-045, a 38mm hard-enamel pin. Three quotes, three sample sets, unit costs of A $0.38, B $0.42, C $0.51.
| Criterion (weight) | Factory A | Factory B | Factory C |
|---|---|---|---|
| Material & build (25) | 3 | 4 | 5 |
| Finish fidelity (15) | 4 | 3 | 5 |
| Color accuracy (10) | 4 | 5 | 4 |
| Dimensions & weight (10) | 5 | 4 | 3 |
| Retail-ready packaging (15) | 2 | 4 | 3 |
| Durability (15) | 3 | 3 | 5 |
| Communication (10) | 3 | 4 | 2 |
The arithmetic, worked per factory:
- Factory A: (3×25) + (4×15) + (4×10) + (5×10) + (2×15) + (3×15) + (3×10) = 75 + 60 + 40 + 50 + 30 + 45 + 30 = 330 → 3.30
- Factory B: (4×25) + (3×15) + (5×10) + (4×10) + (4×15) + (3×15) + (4×10) = 100 + 45 + 50 + 40 + 60 + 45 + 40 = 380 → 3.80
- Factory C: (5×25) + (5×15) + (4×10) + (3×10) + (3×15) + (5×15) + (2×10) = 125 + 75 + 40 + 30 + 45 + 75 + 20 = 410 → 4.10
Decision rule: set the advance cutoff before scoring — say 4.00 — so only Factory C advances, B is the runner-up, and A declines. Cost enters after the cutoff, never inside it. If you're tempted to compare price-per-quality-point, you can: B works out to $0.42 ÷ 3.80 = $0.111 per point and C to $0.51 ÷ 4.10 = $0.124 per point (illustrative), so B looks cheaper per unit of quality. But B missed the cutoff on the two things a customer's hand finds first — finish and durability — and a $0.09 unit-cost saving doesn't survive a shelf of pins that customers scrape. The better move when margin is tight is a second sample round against an amended spec, not a quiet downgrade. And when two factories miss the same criterion, suspect your spec rather than their skill: that's a tech pack problem to fix before round two.
What losing samples get: the hygiene
- Log the verdict. Archive the matrix row and photos per factory in your sourcing folder. Next sourcing round, you quote three factories with a clean slate instead of rediscovering three you already rejected and why.
- Keep the runner-up reachable. B stays warm. If C's production run disappoints, B is your known-quantity fallback — with scores you can reread.
- Decline politely, don't ghost. A two-line decline ("the round went another way; we may re-spec and come back") costs thirty seconds. Sourcing communities are small, and suppliers remember how they lose.
- Salvage what the loss taught you. Two factories failing the same criterion means the spec was ambiguous, not that the industry can't do it. Fix the spec first.
- Check the brand before you commit. If winning samples carry your brand name or logo, confirm the mark is clear for use in your class via the USPTO trademark search before the PO (USPTO, accessed November 2026) — a factory rejection you can recover from; a trademark problem you can't.
- Fold sample spend into unit economics. Sample and freight cost per factory is a sourcing cost, not a write-off — amortize it into the winning SKU's first-order math.
Mistakes that sink a scoring round
- Weights set after opening the boxes. That's motivated reasoning in spreadsheet form; the numbers will always confirm the factory you already preferred.
- Scoring weeks apart, or all at once from memory. Same day, side by side, or the comparison silently becomes three separate judgments.
- Price inside the scorecard. Cost isn't quality. Apply it after the cutoff, where it can't inflate a cheap factory's score.
- Comparing unequal stages. A pre-production sample against a production-grade sample isn't a comparison — confirm what stage each set is before scoring (Alibaba's buyer help center documents the common sample types, accessed November 2026).
- Skipping the communication column. How a factory handles a sample complaint predicts how it handles a production problem. Two-day reply times during sampling become two-week silences during a short shipment.
Common questions
What is a sample comparison matrix? A weighted scorecard for comparing product samples from multiple factories on the same criteria, scored the same day. Each criterion gets a 1–5 score, weights multiply it, and the totals rank factories on evidence instead of memory. It doubles as your written record of why a factory won or lost the business.
What weights should I use? Start with the template's: heaviest on material and build (25), then finish, packaging, and durability (15 each). Shift weight toward what your customer touches and what product photos can't show. Total the weights to 100 and lock them before opening any sample box.
Should unit price be a scored criterion? No. Score quality first, set an advance cutoff, then apply cost to the factories that passed. Mixing price into the score hides quality trade-offs and rewards the cheapest factory with points it didn't earn on the goods themselves. Price-per-point math is fine after the cutoff, as a tiebreaker.
How many factories should I sample? Three is the working minimum for a real comparison; four if the category is new to you. Below three, every difference looks enormous because there's no middle reference point. The ordering logistics and cost expectations are in our guide to ordering product samples.
What do I do with samples from losing factories? Log the matrix and the reason, decline politely, and keep the runner-up reachable as your fallback. Don't discard anything before the winner's first production order arrives. If two factories failed the same criterion, fix your spec before round two rather than resampling the same ambiguity.
Next step
Copy the weights table into a sheet today and set your own weights for the next sourcing round — before requesting samples, because deciding the weights first is half the discipline. When a factory wins, the next paperwork is the purchase order; the steps are in our purchase order process guide.