Is there a duplicate content penalty?

No. Google has stated repeatedly that there is no penalty for duplicate content in the normal case, and the belief that there is causes more harm than the duplication does. The real cost is that signals split across several URLs and the engine picks one to show, which may not be the one you would choose.

How much overlap counts as duplicate?

There is no threshold, because the question is asked the wrong way round. Two URLs serving the same page are duplicates at any percentage; two articles sharing a boilerplate introduction are not duplicates at all. Look at whether the URLs are variants of one page, not at a similarity score.

Should I use a canonical or a redirect for duplicates?

A redirect where the duplicate should not exist, a canonical where it should stay reachable. A print view and a parameter variant need to keep working for people, so they get canonicals. An old HTTP address or a www variant should not be reachable at all, so they get redirects.

Where duplicate URLs come from, and the right remedy for each

Almost no duplicate content is copied. It is generated: by a parameter, a protocol, a print view, a filter or a trailing slash. Which means the remedy is chosen by the cause, and applying the wrong one leaves the problem in place with a tag on top of it.

The short answer

Duplicate content is almost never plagiarism. It is five ordinary causes: URL parameters, printer and AMP variants, HTTP alongside HTTPS, www alongside non-www, and syndicated or boilerplate text. There is no duplicate content penalty. What there is, is signal dilution and an engine choosing which URL to keep without your input.

CriticalAudit check · Duplicate content

Five sources, none of them plagiarism

The phrase makes people think of stolen text, and that is the rarest version of this by a wide margin. What an audit actually finds is your own page reachable at several addresses, because something in the stack manufactures addresses.

Query strings that do not change the page. Tracking parameters, session identifiers, sort orders and filter values that return the same items in a different order. Every distinct string is a distinct URL, and a campaign that appends a parameter has published a new one for every recipient who shares it.
The same page on two protocols or two hosts. HTTP and HTTPS, www and the bare domain. Four combinations, one page, and each is a separate URL until something enforces one. This is the oldest cause on the list and it still shows up on sites whose HTTPS migration was mostly finished.
Alternate renderings of one document.Print views, embed views, an AMP copy, a PDF of the same article, a preview route left open. Each is a full copy of the text with a different wrapper.
Faceted navigation. The largest generator by volume on any site that has it. Three filters with ten values each is a thousand URLs, most of them showing overlapping or empty result sets, all of them crawlable if you link to them.
Genuine republication.Syndicating your article to a partner, or accepting a manufacturer's product description that four hundred other retailers also publish. This one is real duplication and the only one on the list where the copy is not yours.

Notice what the first four have in common: nobody wrote a duplicate. A routing layer, a CMS feature or a marketing tool produced one, which is why you cannot fix this class of problem by editing pages.

Four remedies, and the cause each one fits

These are not interchangeable, and reaching for the familiar one is how a site ends up with canonical tags on pages that needed redirects.

remedy-by-cause.txt

cause                          remedy              why
-----------------------------  ------------------  -----------------------------
tracking / sort parameters     canonical tag       the variant must stay
                                                   reachable, so it can't 301
http, www, trailing slash      301 redirect        one address should exist,
                                                   permanently, at the edge
print / embed / preview copy   301, or noindex     301 if nobody needs it,
                                                   noindex if they do
faceted combinations           canonical + no      the useful facets stay, the
                               internal links      thousand others stop being
                                                   discoverable
syndicated to a partner        their canonical     points at you, set on their
                               points to you       page, agreed in advance
manufacturer description       rewrite it          no tag fixes text you share
                                                   with 400 competitors

The distinction that matters most is the first two rows. A canonical tag is a hint about which URL to index while leaving every variant working and reachable, which is exactly what you want for a parameterised link somebody already shared. A redirect removes the variant, which is exactly what you want for a protocol or host that should never have served anything. Using a canonical where a redirect belongs leaves four addresses alive and competing, each spending crawl budget, forever.

The last row is the one people resist. If your product pages carry the same description as four hundred other retailers, no markup will distinguish them, because nothing about the page is distinguishing. That is a uniqueness problem and its remedy is writing, which is slower and the only thing that works.

The penalty that does not exist

There is no duplicate content penalty. Google has said so directly and repeatedly, and the claim survives anyway because the observed symptoms look punitive: pages disappear from results, traffic drops, a URL you promoted is replaced by one you did not choose.

What is actually happening is selection, not punishment. Faced with several near-identical URLs, a search engine picks one to show and files the others as alternates. Nothing is demoted. One representative is chosen, and the choice is made by signals you may not have set deliberately, which is why the winner is so often the parameterised URL from a newsletter rather than the clean one you link to internally.

The distinction changes what you do. A penalty would mean pleading a case. Selection means the fix is to state a preference clearly enough that the choice is not left to inference, which is what a canonical tag, a redirect and a consistent internal link pattern all are.

The exception, and it is a real one: text copied at scale to manipulate rankings is a spam problem, and that does carry consequences.

Scraping other sites, spinning articles into variants, or generating hundreds of near-identical location pages with the city name swapped is treated as manipulation rather than as a technical duplicate. If that describes what you have, this check is the wrong lens: what you are looking at is thinness produced at volume, and consolidating the URLs does not address it.

Two URLs are duplicates only of each other

There is no field on a page that says it has a twin. The comparison is pairwise across everything you serve, which is why this is one of the checks the crawl exists for, and why a single fetch is the wrong instrument for it.

This one needs the full crawl

Two URLs are duplicates only relative to each other, which means comparing rendered bodies across the site. There is no property of a single page that says it has a twin.

The instant search on this site audits a single page, so rather than show you a verdict it cannot support, this guide sends you to the place the check actually runs.

Run the full audit, free

What the crawl gives you is groups. Working through a group, deciding which URL survives it, is the subject of the clusters guide rather than this one.

Does this affect AI search?

Yes, and the mechanism is more direct than it is for ranking. Systems that build a corpus have to deduplicate before they do anything else, because a document repeated six times distorts everything downstream. So they collapse near-identical fetches into one record and pick a representative, exactly as an index does, using whatever identity signals the documents provide.

Your reason to care is that the representative carries the attribution. If a model cites the version of your article that lives on a syndication partner's domain, the answer is correct, the source is credited, and the source is not you. That is the concrete cost, and it is why the row about syndication in the table above specifies the partner setting a canonical pointing back rather than leaving it to be worked out.

There is a second, quieter effect. Retrieval favours passages, and a passage that appears on nine of your own URLs is a passage with no single home. Consolidating does not make the text better, but it does mean the accumulated signals behind it, links, references, engagement, sit on one address rather than being divided nine ways.

Why this is the duplication check rated Critical

Three of the checks in this area are about duplication and only two are Critical. The line is drawn at whether pages are being kept out of the results, and here they are. When a search engine picks a representative, the pages it did not pick are effectively absent: not penalised, not ranked, not shown. A page you published and cannot be found is the definition this rating exists for.

The rating also reflects volume. Every other finding in an audit scales with the number of pages you wrote. This one scales with the number of parameters, filters and views your system can combine, which is multiplicative. A site with two hundred real pages can serve forty thousand URLs, and the crawl budget spent discovering thirty-nine thousand of them is budget not spent on anything you care about.

What keeps it from being alarming is that the fix is usually configuration rather than content. One redirect rule at the edge, one canonical helper in a template, one decision about which facets are linkable. Critical here means do it early, not that it is difficult.

Finding the variants your own site hands out

Start with the addresses you publish yourself, because internal links are the strongest hint you give about which URL is canonical, and the most common contradiction of it.

find-variants.sh

# does the site serve the same page on all four host and protocol combinations?
for u in http://example.com/ https://example.com/ \
         http://www.example.com/ https://www.example.com/; do
  printf '%s -> %s\n' "$u" "$(curl -s -o /dev/null -w '%{http_code} %{redirect_url}' "$u")"
done

# every internal link carrying a query string, grouped by parameter name
curl -s https://example.com/shop \
  | grep -o 'href="[^"]*?[^"]*"' \
  | sed 's/.*?//;s/=[^&"]*//g;s/"$//' \
  | sort | uniq -c | sort -rn

# does the print view exist, and is it indexable?
curl -s https://example.com/print/guides/sharpening \
  | grep -o '<meta name="robots"[^>]*>'

The first block should show three of the four redirecting to one. Anything returning a 200 twice is a host you are publishing in parallel, and no canonical tag is a substitute for fixing it at the edge.

The second is the one that surprises people. If your own navigation links to ?sort=price and ?sort=name, you are not just permitting those URLs, you are recommending them, and the canonical you set on them is arguing against your own markup.

Questions this check raises

Is there a duplicate content penalty?: No. Google has stated repeatedly that there is no penalty for duplicate content in the normal case, and the belief that there is causes more harm than the duplication does. The real cost is that signals split across several URLs and the engine picks one to show, which may not be the one you would choose.
How much overlap counts as duplicate?: There is no threshold, because the question is asked the wrong way round. Two URLs serving the same page are duplicates at any percentage; two articles sharing a boilerplate introduction are not duplicates at all. Look at whether the URLs are variants of one page, not at a similarity score.
Should I use a canonical or a redirect for duplicates?: A redirect where the duplicate should not exist, a canonical where it should stay reachable. A print view and a parameter variant need to keep working for people, so they get canonicals. An old HTTP address or a www variant should not be reachable at all, so they get redirects.