The notebook is small, the outputs and the datasets are not
Checkpoints, embedded outputs and downloaded datasets fill a data science Mac. What is derived, what is irreplaceable, and how to tell.
A notebook is a text file until you run it. Then it carries its outputs inside itself, including every image and table, and a notebook with a few hundred plots can be tens of megabytes on its own. Around it sit checkpoints, caches and whatever data you downloaded.
Find the space
find ~ -name '*.ipynb' -size +5M -exec ls -lh {} + 2>/dev/null | sort -k5 -h | tail
find ~ -type d -name '.ipynb_checkpoints' 2>/dev/null -exec du -sh {} + | sort -h | tail
du -sh ~/.cache/huggingface ~/scikit_learn_data ~/nltk_data 2>/dev/nullWhat each thing is
| Item | What it is | Safe to remove |
|---|---|---|
.ipynb_checkpoints | Autosave copies of notebooks | Yes |
| Embedded outputs in a notebook | Images and tables saved inside | Yes, by clearing outputs |
| Downloaded datasets | Fetched by a library or by you | Depends on the source |
| Model and library caches | Fetched on demand | Yes, redownloaded |
| Intermediate parquet or pickle files | Generated by your own code | Yes, if the code reruns |
Clearing outputs is the quiet win
Clearing outputs before committing a notebook makes it smaller, makes diffs readable, and removes embedded images that are regenerated in seconds by running the cells again:
jupyter nbconvert --clear-output --inplace notebook.ipynb
find . -name '*.ipynb' -exec jupyter nbconvert --clear-output --inplace {} +Datasets are the part to think about
A public dataset that will still be there next year is a download. A dataset that came from a source that has since changed, or that you produced by an expensive pipeline, is not. Sorting the data folder into those two categories once, and noting it in a readme, is worth more than any cleanup command.
Model weights are a separate and larger category with their own guide in local model files, and the Python side of the same setup is in what Python leaves on a Mac.
Common questions
Why are my Jupyter notebooks so large?
Because outputs are stored inside the notebook file, including every image and table. Clearing outputs with nbconvert --clear-output shrinks them dramatically and they regenerate when you rerun the cells.
What is the .ipynb_checkpoints folder?
Autosave copies Jupyter keeps beside your notebooks. They are safe to remove and are recreated as you work, and they are usually small unless notebooks carry heavy outputs.
Can I delete downloaded datasets?
It depends on the source. A public dataset that will still exist next year is effectively a download. Anything produced by your own pipeline or fetched from a source that has changed should be treated as irreplaceable.
Where do Python data libraries cache downloads?
Commonly ~/.cache/huggingface, ~/scikit_learn_data and ~/nltk_data, among others. All are fetched on demand and safe to clear, at the cost of downloading again.