Reading a duplicate cluster and picking the page that survives

Two pages are compared on the text that is genuinely theirs, after the shared regions are set aside. That last part is the whole reason a score can be trusted: if the header, nav and footer were included, every page on your site would be 70 percent similar to every other page, and the report would find nothing but itself.

What comes out is a proportion, and proportions have a threshold behind them. Above roughly 95 percent you are looking at the same document with a different wrapper. Between about 85 and 95 you are looking at two documents assembled from the same parts, which is a different problem with a different fix. Below that, pages on the same subject are supposed to resemble each other, and a report that flagged them would be reporting that you have a topic.

So the score is not a severity. It is a classifier, and reading it as "how bad is this" rather than "what kind of thing is this" is how a cluster of two good city pages ends up merged into one.

In an exact cluster the choice looks arbitrary and is not. Four criteria, in this order, and the order matters because they conflict.

• The one with external links pointing at it. This outranks everything else on the list. Links are the asset you cannot recreate, and choosing a different survivor means asking a redirect to carry them, which works and is still a cost.
• The one already being shown in results. If a search engine has already picked a representative, agreeing with it costs you nothing. Disagreeing means a period of instability while the new preference is worked out.
• The cleanest address. No parameters, no session fragment, the path you would type. This is where most people start and it belongs third, because a pretty URL with no links pointing at it is worth less than an ugly one with forty.
• The one your own navigation links to. Not because it is inherently better, but because if you pick anything else you now have to change every internal link, and a site that links to a URL while canonicalising away from it is contradicting itself.

When the first two criteria disagree, follow the links and change the rest. When they agree and the URL is ugly, keep the ugly one and stop thinking about it. The address nobody sees is not worth a migration.

It affects it before anything else does. Deduplication is one of the first passes in any pipeline that assembles documents at scale, because a corpus containing the same text nine times weights that text nine times. Clusters are collapsed, a representative is chosen, and the rest are discarded rather than ranked lower.

Which puts the stakes differently from search. In a results page, the versions that lost are still on the site and can still be found by someone who looks. In a collapsed corpus, they were merged into one record and the record has one URL attached to it. If that URL is your print view, your citations point at your print view.

The practical implication is that the survivor decision in the section above is not only about which page ranks. It is also, silently, about which address gets credited when something quotes you, and that makes the criteria about external links and existing visibility more important rather than less.

Before merging anything, look at what each page actually contains, stripped of markup. Two pages that a report scored at 97 percent usually differ in one paragraph, and that paragraph is what you are about to delete.

# crude text extraction, good enough to diff two of your own pages
strip() {
  curl -s "$1" \
    | sed -e 's/<script[^>]*>.*<\/script>//g' \
          -e 's/<style[^>]*>.*<\/style>//g' \
          -e 's/<[^>]*>/ /g' \
    | tr -s ' \n' ' ' \
    | tr ' ' '\n' | sed '/^$/d'
}

strip https://example.com/plumbers/leeds > /tmp/a.txt
strip https://example.com/plumbers/york  > /tmp/b.txt

# how much of page A is not in page B
comm -23 <(sort -u /tmp/a.txt) <(sort -u /tmp/b.txt) | head -40

wc -l /tmp/a.txt /tmp/b.txt

If the words unique to one page are a city name, a phone number and nothing else, the cluster is real and the pages were never differentiated. If forty words come back, somebody wrote something and you are about to lose it.

Run this before the redirect, not after. A 301 is easy to place and awkward to reason about six months later, when the only record of what the old page said is in an archive.