How to clear Jupyter Notebook's output and metadata when using git commit?

Question

The output and metadata is not for code reviewing and it is annoying if committed. How to clear Jupyter Notebook's output and metadata when using git commit?

Answer 1

github post

similar approach in stackoverflow

the answer is based on the previous 2 posts.

My approach includes cleaning metadata at the same time.

Add this to your local .git/config

[filter "strip-notebook-output"]
clean = "jupyter nbconvert --ClearOutputPreprocessor.enabled=True --ClearMetadataPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR"

Create a .gitattributes file in your directory with notebooks, with this content:

*.ipynb filter=strip-notebook-output

Answer 2

You could use the command-line json processor jq to perform this task notably quicker than with nbconvert. This blog post shows how to get rid of metadata, outputs and execution counts via this command:

jq --indent 1 \
    '
    (.cells[] | select(has("outputs")) | .outputs) = []
    | (.cells[] | select(has("execution_count")) | .execution_count) = null
    | .metadata = {"language_info": {"name":"python", "pygments_lexer": "ipython3"}}
    | .cells[].metadata = {}
    ' 01-parsing.ipynb

You could also modify to just clean a specific part of the output, such as execution counts (recursively wherever they occur in the json), and then add this as a git filter:

[filter "nbstrip"]
    clean = jq --indent 1 '(.. |."execution_count"? | select(. != null)) = null'
    smudge = cat

And add the following to ~/.config/git/attributes to have the filter applied globally to all your local repos:

*.ipynb filter=nbstripout

There are some more ideas in this thread How to clear Jupyter Notebook's output in all cells from the Linux terminal?. There is also nbstripout which is made for this purpose, but it's a bit slower.