• Windows
  • Data

The notebook is small, the outputs and the datasets are not

Checkpoints, embedded outputs and downloaded datasets on a Windows data science machine, and which of the three you can safely remove.

5 min read

A notebook is a text file until you run it. Then it carries its outputs inside itself, including every image and table, and a notebook with a few hundred plots is tens of megabytes on its own.

Finding the space

Get-ChildItem $env:USERPROFILE -Recurse -Filter *.ipynb -EA SilentlyContinue |
  Where-Object Length -gt 5MB |
  Sort-Object Length -Descending |
  Select-Object @{n='MB';e={[math]::Round($_.Length/1MB,1)}}, FullName -First 15
Get-ChildItem $env:USERPROFILE -Recurse -Directory -Filter .ipynb_checkpoints -Force -EA SilentlyContinue |
  Measure-Object | Select-Object Count

What each thing is

ItemWhat it isSafe to remove
.ipynb_checkpointsAutosave copiesYes
Embedded outputsImages and tables saved inside the notebookYes, by clearing outputs
Downloaded datasetsFetched by a library or by youDepends on the source
Model and library cachesFetched on demandYes, redownloaded
Intermediate parquet or pickle filesGenerated by your own codeYes, if the code reruns

Clearing outputs

jupyter nbconvert --clear-output --inplace notebook.ipynb
Get-ChildItem . -Recurse -Filter *.ipynb | ForEach-Object {
  jupyter nbconvert --clear-output --inplace $_.FullName
}

Clearing outputs before committing makes notebooks smaller, makes diffs readable, and removes embedded images that regenerate in seconds by rerunning the cells.

Datasets are the part to think about

A public dataset that will still be there next year is a download. One produced by your own pipeline, or fetched from a source that has since changed, is not. Sorting the data folder into those two categories once, and writing it down, is worth more than any cleanup command.

Where this sits on Windows

A data science machine usually also has Conda environments and model caches, and both are larger than the notebooks. Checking all three together is one pass.

Common questions

Why are my Jupyter notebooks so large?

Because outputs are stored inside the notebook file, including every image and table. Clearing outputs with nbconvert shrinks them dramatically and they regenerate when you rerun the cells.

What is the .ipynb_checkpoints folder?

Autosave copies Jupyter keeps beside your notebooks. They are safe to remove and are recreated as you work.

Can I delete downloaded datasets?

It depends on the source. A public dataset that will still exist next year is effectively a download. Anything from your own pipeline should be treated as irreplaceable.

How do I clear outputs from many notebooks at once?

Loop over the files with jupyter nbconvert --clear-output --inplace. It is worth doing before committing, since it also makes diffs readable.

Read next