How ContentRank works

ContentRank gives any URL a quality score from 0 to 100. The score emerges from a network of matches, the same way PageRank emerges from a network of links. Each URL is a node in that graph, each match is an edge. Glicko-2 makes the graph converge into stable per-axis ratings. NLP tooling handles text extraction, normalisation, hashing, embedding-based concordance. In-house statistical tools aggregate the signals into the final ContentRank. A LLM-as-a-judge sits inside this stack as one of the components: on each match, it produces the local qualitative comparison, the kind of structured judgement modern LLMs are reliably good at.

The score reflects how a text holds up across many independent confrontations on a network. The same shape as PageRank or chess Elo, applied to argumentation quality instead of link authority or game wins.

This page is the audit page. If you want to dismantle the system, this is where you look. We try to be precise about what we measure, what we don't, and where the limits are.

1. The conceptual core

Quality, in this system, is what survives confrontation against peers. A URL that holds up well across many comparisons converges to a high rating, one that doesn't, doesn't. The architecture is the same as PageRank (quality emerges from the structure of a graph), except instead of citation links between pages, the edges of our graph are matches between articles on the same topic.

A ContentRank rests on four pillars, each doing one job:

• The match graph. Each URL is a node, each match is an edge weighted by concordance. The score emerges from the graph the way PageRank emerges from links. A single match informs the structure, the structure produces the rating.
• LLM-as-a-judge. On each match, it does the local qualitative comparison: given two extracted texts, it produces per-axis verdicts anchored on exact passages. Natural-language judgement, with citations. This is the component where natural-language understanding is needed, and modern LLMs are reliably good at it under structured prompting.
• The match algorithm. Deterministic in-house math: handles URL canonicalisation, text extraction, normalisation, hashing, embedding-based concordance, anti-hallucination checks (do the cited passages actually exist?), and conversion of qualitative verdicts into Glicko inputs weighted by concordance.
• Glicko-2. The statistical method behind chess Elo, validated since 2012. Makes the match graph converge. Each URL accumulates one rating per axis as matches roll in. The single ContentRank surfaced on a URL profile is a deterministic derivation of those nine ratings.

The 9 axes are designed for this division of labour: each one is something a LLM-as-a-judge can compare two texts on (does this one cite more sources? is this one more internally coherent?), the kind of relative judgement on text that the technique handles well.

2. The 9 axes

Quality isn't one-dimensional. We split it into nine axes grouped in three families. The 3×3 structure surfaces a text's quality profile, strong on Foundation but weak on Form, or the inverse, rather than collapsing everything into a single dimension.

Foundation: the substance markers

Sourcing

Presence and quality of mentioned sources, explicit citations, bibliographic references, verifiable hyperlinks. Distinct from factuality: a sourced text can be wrong, an unsourced text can be right. We measure whether the text bothers to ground its claims.

Not applicable: explicitly subjective or creative texts.

Factuality

Correctness of verifiable factual claims, judged comparatively. For each factual claim in one text, the other text confirms it, contradicts it, or ignores it. The LLM compares A and B; it does not check the outside world.

Not applicable: purely opinion or creative texts.

Internal Coherence

Absence of internal contradictions. Do the claims square with each other? Do the conclusions follow from the arguments? This is the axis the LLM measures most reliably, internal coherence is intrinsically verifiable from the text alone.

Form: the reader-experience markers

Clarity

Accessibility to the apparent target audience. Defined terms, illustrative examples, sentence-level legibility. Distinct from structure: clarity operates at the paragraph or sentence level; structure operates at the document level.

Structure

Logical organisation of the text as a whole. Introduction, development, transitions, conclusion, does the text take the reader somewhere, or does it wander?

Conciseness

Expression efficiency, does the text say what's essential without dilution? Conciseness is not brevity. A long text can be concise; a short text can be padded. This axis exists in part to mechanically counter the verbosity bias of LLM-judges, who tend to reward verbose output.

Context: the depth markers

Depth

Breadth and depth of subject treatment. Depth is not length. A 1000-word article can treat one aspect deeply; a 5000-word article can skim ten. Pairs naturally with conciseness, a productive trade-off that forces nuanced judgement.

Freshness

Currency of the information presented. Does the text engage with recent developments, or does it rely on dated references?

Not applicable: timeless subjects (philosophy, established mathematics).

Epistemic Honesty

Lucidity of the text about its own limits. Does it hedge appropriately? Does it acknowledge counter-arguments? Does it signal where the evidence is weak? This is a rare axis, most quality systems don't measure it, and it's a strong differentiator. Texts that score high here tend to be the ones written by people who know their subject well enough to know what they don't know.

3. The match: one edge in the graph

A match connects two URLs in the graph and produces the structured signal that Glicko-2 will later aggregate. Two texts go in, nine axis verdicts come out, each anchored on exact passages from one or both texts. The match is the unit of evidence, the graph is what produces the rating.

Multi-stage pipeline, scoped LLM calls

A single match is processed through several sequential stages. Each stage has one narrow job, and the LLM is called only where natural-language understanding is genuinely needed.

1. Extraction: URL canonicalisation, content extraction, normalisation, deduplication hashing.
2. Graph registration: look up each URL in the match graph. If it's new, create a node with default Glicko ratings on each of the nine axes. The match will become an edge between these two nodes.
3. Concordance pre-check: cheap embedding-based similarity check that refuses obviously incomparable pairs before any expensive call.
4. Qualitative comparison (LLM-as-a-judge): produces nine axis verdicts as labels (e.g. A_WINS_SLIGHTLY, TIE) with citations anchored on exact passages. For URL matches, this stage runs twice with text A and text B swapped, to neutralise the position bias modern LLMs carry.
5. Anti-hallucination verification: every cited passage is checked against the source text. Unverified citations invalidate the corresponding axis.
6. Score quantification: verdict labels are converted into Glicko-2 input scores in [0, 1] via a fixed mapping (A_WINS_CLEARLY → 1.0, A_LOSES_CLEARLY → 0.0, etc.), weighted by the match's concordance score. The qualitative judgement becomes a number here.
7. Glicko-2 update: the quantified scores feed the per-axis ratings of the two nodes. The new edge updates the graph, the network converges a little further.
8. Post-work consolidation (LLM-as-a-judge): re-reads the qualitative comparison output(s) and produces the short verdict line (≤140 chars), the longer verdict narrative (4-6 sentences), a summary of each text, and a clean display title. Consolidation and narration for display.

Per match, 2-3 scoped LLM-as-a-judge calls produce labels, citations, and display copy. Everything else is math. The rating is built by the graph.

Anchored evidence is mandatory and verified

Every verdict cites exact passages from one or both texts. Citations are short verbatim fragments, not page numbers or paraphrases. After the LLM call, an independent verifier checks that each cited passage actually appears in the source text (whitespace-normalised substring match). If a citation can't be verified, the axis is invalidated: relevance forced to 0, no Glicko update from it. The verifier is independent of the LLM call.

Verdict labels, not numbers

Each axis lands on one of five labels: A_WINS_CLEARLY, A_WINS_SLIGHTLY, TIE, A_LOSES_SLIGHTLY, A_LOSES_CLEARLY. We tested integer scales during the prompt-validation phase and found them empirically unstable, sometimes inverting the label relative to the reasoning. Semantic labels generate the encoding from meaning rather than from a cognitive jump to a number scale. Conversion of those labels into Glicko inputs is handled downstream by an in-house deterministic mapping (see §4).

Position randomisation

Modern LLMs exhibit position bias: depending on the model, the first or the last item presented tends to be favoured. Before each call we randomise which text is presented as A and which as B, then re-map the verdicts back to the user's display order. The randomisation seed and the original orientation are persisted in the database, so any match can be replayed in the exact orientation the LLM saw.

Graceful failure

A match can also fail cleanly. A match_validity field lets the LLM declare two texts fundamentally incomparable, in which case no Glicko update occurs and the user sees an "incommensurable" message. The match is recorded but doesn't pollute the rankings. Concordance scoring (§5) catches most of these before the LLM even runs, the validity flag is the safety net for the rest.

4. The ranking: Glicko-2

Each match is one edge between two URL nodes. As edges accumulate across the graph, Glicko-2 makes the rating of each node converge: the same convergence behaviour as PageRank on a citation graph, applied here to a graph of qualitative comparisons rather than links. Glicko-2 is the rating system behind chess Elo, a successor to Elo that explicitly models uncertainty. Each node carries three numbers per axis: a rating (the current best estimate), a rating deviation (how confident the system is in that estimate), and a volatility (how erratic the entity is in matches). After each match, all three update.

We run nine independent Glicko-2 instances per URL, one per axis. A URL that's strong on Sourcing but weak on Conciseness has a high rating on the first instance and a low rating on the second. The single visible ContentRank 0-100 score is derived deterministically from those nine ratings, it's a presentation artefact, not a primitive of the system.

Two consequences of this graph-based structure:

• ·The signal is structural. A URL's rating reflects how it has held up against many peers across many independent matches. The convergence is what produces the rating, individual edges contribute incrementally.
• ·The graph is the audit trail. A user inspecting a URL profile can see every match that contributed to its current rating, with the citations and reasoning that anchored each verdict. Every input that fed the score is inspectable.

Verdict to score conversion

Each verdict maps to a Glicko score in [0, 1] for each side:

Reliability tiers

Each rating carries a Glicko rating deviation (RD). High RD means few matches, low confidence. Low RD means the rating is settled. We translate the average RD across the nine axes into a five-tier reliability indicator. Each URL surfaces its tier in two places : prominently as a star bar and label on its /url/{hash} profile, and discreetly on every /match/{hash} page (Provisional or Stable badge under the score).

The "Stable" tag (top two tiers) is shown on the score banner. "Provisional" stays in the body of the match (under the concordance line) and on the URL profile. We surface the positive signal where it counts and keep the negative one in context, so a reader can calibrate their own confidence in the score they see.

Low-confidence URLs are also less defensible against a single biased match. We don't hide that : a freshly-evaluated URL gets a small star bar and a "Very provisional" label, which is honest information for the reader.

Median axis marker

On each per-axis bar (visible on every /url/{hash} profile and /match/{hash} result), a small grey dot marks the median across all rated URLs on that axis : line up every URL we've ever evaluated by their score on this axis, the dot is the one in the middle. Half score above it, half below.

The marker turns each individual rank into a relative reading. A URL whose bar reaches past the dot scores above the median on that axis ; a bar that doesn't reach the dot is below. The dot moves with the catalogue : as more URLs are evaluated and their ratings settle, the median on each axis migrates, and so does the contextual reading of every other URL. This is the same compositional logic as PageRank, applied to per-axis quality.

Updates weighted by concordance

Glicko updates are weighted by the match's concordance score (see §5). A low-concordance match, two texts that aren't really comparable, moves both ratings less than a high-concordance match. This prevents noisy comparisons from distorting the ranking.

Reference: Glickman, M. E. (2012). Example of the Glicko-2 system. The implementation in this codebase has been validated against the worked example in that paper.

5. Concordance: when matches are comparable

Not every pair of texts is meaningfully comparable. A philosophy article on Kant and a Stack Overflow tutorial on regex might both score on our 9 axes, but the comparison is noise. We need a way to measure how comparable two texts are, and to weight or refuse the match accordingly.

Concordance is a composite score in [0, 1], computed from three independent signals:

concordance = 0.33 × embedding_similarity
            + 0.33 × mean_relevance
            + 0.33 × subject_similarity_score

• ·Embedding similarity: cosine similarity of the two text embeddings (a pre-LLM, cheap signal).
• ·Mean relevance: the LLM's per-axis relevance scores, averaged and normalised. If the LLM marks most axes as not-applicable, the texts probably aren't comparable.
• ·Subject similarity: the LLM's labelled judgement of comparability, highly comparable, broadly comparable, loosely comparable, weakly comparable, incomparable, mapped to a numeric score.

We refuse matches where the embedding similarity alone is below 0.3, the cheap pre-filter, to save LLM cost on obviously-mismatched pairs. For matches that pass the pre-filter, the full concordance score is computed after the LLM call and used to weight the Glicko update.

6. What we don't claim

ContentRank is built to be honest about its limits. The most important ones:

We don't measure truth

Pure substance evaluation, judging whether a thesis is correct, whether a specific factual claim matches reality, requires fact-checking against the outside world. A LLM cannot reliably do that. What we measure is upstream: whether a text exhibits the markers of solid argumentation, sources cited, claims internally coherent, depth maintained, counter-arguments acknowledged. A high ContentRank earns your reading time. The verdict on truth stays yours.

The score is imperfect

A ContentRank of 73 vs. 71 means very little. A 73 vs. 35 means a lot. The score's value lives in the gap, not the absolute. We aggregate noisy qualitative inputs into a single number; the aggregation is principled, but the inputs aren't perfect, and reducing nine dimensions to one is lossy by design.

The LLM has biases we inherit

The underlying LLM was trained on the public web, which skews toward mainstream consensus. Heterodox or anti-mainstream texts may be systematically penalised on axes like Factuality, where the LLM's prior favours the consensus view. Order randomisation mitigates position bias; it does not solve prior-distribution bias. We acknowledge this as an open problem.

Some genres don't fit

Creative writing, poetry, fiction, and pure-opinion pieces don't sit cleanly in our framework. Our axes assume non-fiction informational texts. If you submit a poem, it'll score poorly on Sourcing, and that score reflects the framework misfit, not the poem's quality.

We're an upstream filter

ContentRank tells you which articles are worth your reading time based on how they hold up under structured comparison. It's a triage signal. It doesn't replace your judgement on whether a high-scoring article is right, useful, or applicable to your case.

7. Traceability: open vs. closed

Our promise is traceability, not abstract transparency. Every score on a URL profile points to inspectable data: the matches that contributed, the citations they were anchored on, the per-axis verdicts and their reasoning. The full reasoning chain is auditable end-to-end.

What's open:

• ·The 9 axes and their definitions
• ·The five verdict labels and their semantics
• ·The Glicko-2 update logic (Glickman 2012, public paper)
• ·The concordance composite formula (33/33/33)
• ·The reliability tier table
• ·For each match: the citations, sub-scores, reasoning text, every input that fed the score

What's closed:

• ·The exact prompt text. Two reasons: anti-injection (page authors should not be able to craft content tuned to specific phrasings) and anti-cloning (the prompt is months of iteration). The prompt's structure is described in this page; the verbatim wording stays closed.
• ·Cache TTLs and rate-limit thresholds, for anti-abuse reasons.

It's a deliberate trade-off. Open on the principles. Reserved on the levers.

8. Disputed answers

The standard objections, and our responses.