Finding duplicate files on a Mac without paying for an app

Duplicate finders are priced like they do something difficult. Here is how to find real duplicates with tools already on your Mac, and when an app is worth it.

  • Duplicates
  • Cleanup
  • Storage

6 min read

Duplicate finder apps are a healthy business, which is odd, because finding identical files is one of the more mechanical jobs a computer can do. You can do it with tools already installed.

It is worth saying first that on most Macs duplicates are not where the space went. Before spending an afternoon on this, check what is actually filling the drive. If the answer is caches or build output, deduplicating your Documents folder will not help.

Finding true duplicates

A duplicate is a file with identical content, which means comparing hashes rather than names or sizes. This finds them under any folder you point it at:

find ~/Documents -type f -size +1M -exec md5 -r {} + \
  | sort \
  | awk '{ if ($1 == prev) print; prev = $1 }'

It hashes every file over 1 MB, sorts by hash, and prints any whose hash matches the line before. The size filter matters: without it you spend most of the time hashing tiny files that are not worth reclaiming.

For a readable report showing which paths collide:

find ~/Documents -type f -size +1M -exec md5 -r {} + \
  | sort \
  | uniq -w32 -D \
  | sed 's/^[a-f0-9]* //'

Photos, which are a special case

Do not deduplicate the Photos library with shell commands. Photos stores originals, edits and thumbnails in a managed structure, and files that look duplicated to find are often an original and its derivative. Photos has its own duplicate detection in the sidebar, and it understands that structure.

When an app is worth the money

Being fair about it, a dedicated finder does three things the shell does not:

  • Near-duplicates. Two photos taken a second apart, or the same image at different resolutions, are not byte-identical and no hash will match them.
  • A review interface. Deciding which of forty pairs to keep is genuinely nicer in a list with previews than in a terminal.
  • Undo. A good one moves to the Trash and remembers what it did.

If your problem is a few large duplicated exports, the commands above are ten minutes. If it is ten thousand photos, buy the app.

The honest expectation

On a typical Mac, duplicates account for single-digit gigabytes. Caches, developer build output and app leftovers usually account for tens. Start where the space is.

Common questions

How do I find duplicate files on a Mac for free?

Use md5 to hash every file over a size threshold, sort the results, and print the ones whose hashes match. That finds byte-identical duplicates, which is what the paid apps look for too. It will not find near-duplicates such as two similar photos, because those are not identical files.

Are duplicate file finders worth paying for?

They are worth it if you have a large photo library or want a review interface with previews and undo, because near-duplicate detection is genuinely hard and worth paying for. For a handful of large duplicated files, the shell commands do the same job in ten minutes.

Do duplicate files slow down a Mac?

No. Duplicates take up space and nothing else. Performance on a Mac degrades when the drive approaches full, so removing anything large helps at that point, but a duplicate file is no worse for speed than any other file of the same size.

Read next