Skip to content

Find duplicates by size first, then hash. Never the other way

The method that takes seconds instead of hours, plus the folders you must never let a duplicate finder touch.

6 min read

Duplicates accumulate in a predictable way: the same photos imported twice, a folder copied before a reinstall, an archive extracted beside itself. They are real space, and the tools that find them are also the tools most likely to delete something that only looked like a copy.

The method that makes it fast

Two files of different sizes cannot be identical. That single fact turns an hour of hashing into a few seconds, because only groups of files that already share an exact size need to be read at all.

$root = "$env:USERPROFILE"

# 1. Group by exact size. Only groups with more than one member can contain duplicates.
$candidates = Get-ChildItem $root -Recurse -File -Force -EA SilentlyContinue |
  Where-Object Length -gt 1MB |
  Group-Object Length | Where-Object Count -gt 1

# 2. Hash only those, and group again on the hash.
$dupes = $candidates | ForEach-Object { $_.Group } |
  Get-FileHash -Algorithm SHA256 -EA SilentlyContinue |
  Group-Object Hash | Where-Object Count -gt 1

# 3. What the copies are costing, largest first
$dupes | ForEach-Object {
  $size = (Get-Item $_.Group[0].Path).Length
  [PSCustomObject]@{
    MB = [math]::Round($size * ($_.Count - 1) / 1MB, 1)
    Copies = $_.Count
    Path = $_.Group[0].Path
  }
} | Sort-Object MB -Descending | Select-Object -First 25

The MB column is what you would recover by keeping one copy, not the total size of the group. That distinction matters when deciding whether the exercise is worth the risk.

Where to point it, and where never to

FolderRun it hereWhy
Pictures, Videos, DocumentsYesWhere real duplicates accumulate
DownloadsYesThe same installer fetched repeatedly
C:\Windows and Program FilesNeverIdentical files there are meant to be identical
node_modules, package cachesNeverDuplication is how the tool works
OneDrive or Drive foldersCarefulA deletion syncs everywhere immediately

The trap: files that are already one copy

A hard link makes one set of bytes appear under two names. A duplicate finder reports it as two identical files, and deleting one frees nothing, because there was only ever one copy. Windows itself is built from these, which is why the component store misleads every size tool: hard links and junctions.

# LinkCount greater than 1 means other names point at the same data
(Get-Item 'C:\path\to\file' -Force).LinkCount

Deleting safely

  1. Sort by recovered space and work on the top ten only. The long tail is rarely worth the risk.
  2. Keep the copy in the folder you would look in first, not the one with the shortest path.
  3. Delete to the Recycle Bin, never with a permanent delete, so a mistake is a right click away.
  4. Empty the bin a week later, once you have used the machine normally.

A photo library is the usual reason people start this, and there the catalogue matters as much as the files: removing a duplicate behind an editor's back leaves a broken entry, which is covered in photo editors on Windows.

Common questions

How do I find duplicate files on Windows?

Group files by exact size first, then hash only the groups with more than one member. Two files of different sizes cannot be identical, so this avoids reading almost everything.

Are duplicate file finders safe?

The finding is safe; the automatic deleting is where damage happens. Review the largest groups by hand, delete to the Recycle Bin, and never run one over Windows, Program Files or package caches.

Why does deleting a duplicate free no space?

Because the two names may point at the same data through a hard link. Check with (Get-Item path -Force).LinkCount: anything above 1 means other names share the same bytes.

Where do duplicates usually come from?

Photos imported twice, a folder copied before a reinstall, archives extracted beside themselves, and the same installer downloaded repeatedly.

Read next