Xcode keeps a clone of every package repository you have used

Cleaning DerivedData is the advice everyone knows, and it misses the package caches entirely. Swift Package Manager keeps a clone of every repository your projects depend on in a shared location, and nothing prunes it when a project stops using a package.

Where to look

du -sh ~/Library/Caches/org.swift.swiftpm \
  ~/Library/Developer/Xcode/DerivedData 2>/dev/null
du -sh ~/Library/Caches/org.swift.swiftpm/* 2>/dev/null | sort -h

The repositories folder inside it is the shared clone cache. Each entry is a full git clone, so a dependency with a long history costs what that history costs, not what the source code costs.

Per project checkouts as well

On top of the shared cache, each project keeps resolved checkouts inside its DerivedData folder or its .build directory for command line builds. That means the same package can exist several times on one Mac: once in the shared clone cache and once per project.

find ~ -type d -name .build -maxdepth 5 2>/dev/null \
  -exec du -sh {} + | sort -h | tail -10

Clearing it safely

In Xcode, File, Packages, Reset Package Caches does this properly for the current project, re-resolving from the manifest afterwards. From the command line, swift package purge-cache clears the shared cache.

Everything here is fetched again from the network on the next resolve, so the cost is bandwidth and a slower first build. The one caveat is a project depending on a package whose repository has since disappeared, in which case the cache was the last copy. That is rare and worth a thought before clearing on an old project.

Where it sits among the Xcode folders

Package caches are usually the smallest of the four. DerivedData is the largest, archives and device support hold things that cannot be regenerated, and simulator runtimes are the ones that surprise people. Clearing all four in one pass is the difference between a few gigabytes and a few tens of gigabytes.