Measured, not claimed

The corpus is small right now. We publish these numbers anyway, because the protocol matters more than the sample size: every figure below is reproducible, and this page is regenerated as the corpus grows. A benchmark you can rerun beats a percentage you have to trust.

The deeper point: scores from a single LLM call are not trustworthy, and we can prove it, on our own system. What makes ContentRank defensible is not the judge, it is the machinery around the judge: double-pass order swapping, verbatim quote verification, and a consolidation step that has to reconcile two independent judgments before anything reaches the score.

1. Position bias: measured on ourselves

Every match is judged twice: once with text A presented first, once with text B presented first. Comparing the two passes, axis by axis, measures how much the judge's verdict depends on mere presentation order. On the last run:

Nearly all direction changes happen on "wins slightly" verdicts: when the judge hesitates, it leans toward whichever text it read first. Clear-cut verdicts are stable. This is exactly why no single LLM call ever reaches a ContentRank score directly: a third, consolidating judgment sees both passes and must reconcile them, and axes it cannot reconcile are excluded from the rating update.

If a tool scores content with one LLM call and no order control, roughly a quarter of its close calls are decided by presentation order. Ask your vendor for this number.

2. Quote verification: every citation is checked

Every verdict must be anchored on quotes from the texts. Each quote the judge produces is programmatically verified against the source: if the exact passage is not in the text, the quote is rejected. An axis whose evidence is entirely rejected is excluded from the score.

• 138 quotes produced by the judge on the last run
• 30 rejected by verbatim verification (22%): paraphrases, reference markers cited as quotes, passages not found in the source
• 1 axis (of 45) excluded from scoring because no valid evidence survived

The rejection rate is a feature, not a bug: it is the measured frequency at which an LLM, explicitly instructed to quote verbatim, still does not. Unverified quotes never reach a score.

3. Audit it yourself, on any match

This is the strongest guarantee we can offer, and it requires no trust at all: open any public match, expand any axis, and check any quote against the source text. The reasoning, the quotes, and the per-axis verdicts are all published. If you find a quote that is not in the source, the system failed and we want to know.

Cost and latency, for the record: a full match (two judging passes plus consolidation) costs about €0.012 in LLM calls and takes ~49s.

4. Protocol

1. Fixed set of public URL pairs (same-topic, contrasting sources: encyclopedia vs vulgarization, official docs vs tutorial).
2. Each pair runs through the full production pipeline: extraction, two independent judging passes with the presentation order swapped, verbatim quote verification, consolidation, scoring.
3. Pass-level outputs are archived per match and diffed run over run, so any prompt or pipeline change is measured against the previous run before it ships.

Full methodology, axis definitions and known limits are on the methodology page.

5. What we measure next

Planned, not yet measured. They will appear here with numbers when they run, not before.

• Self-match probe. A text compared to itself must tie on every axis. Any deviation is measured bias.
• Repeatability. The same match rerun several times: how stable are verdicts call to call, per axis and per verdict strength.
• Human anchor pairs. Pairs with a documented expected outcome, annotated under published guidelines: agreement rate between ContentRank and human judgment.
• Cross-model agreement. The same pairs judged by independent LLMs from different vendors: how often they designate the same winner.