Jupyter Notebook

Intuition

A script runs top to bottom and produces output somewhere else. A notebook keeps the output next to the code that made it, and lets you re-run one piece without re-running the rest. For exploration that is exactly right: most of data analysis is looking at something, adjusting, and looking again, and a workflow that reloads a 2 GB dataframe on every adjustment is a workflow you will abandon.

Project Jupyter defines the artefact broadly: "a notebook is a shareable document that combines computer code, plain language descriptions, data, rich visualizations like 3D models, charts, graphs and figures, and interactive controls" [1, What is a notebook?].

The cost is bought with the same coin. The freedom to run cells in any order means the document you save is not a record of what produced it. Notebooks "combine detailed workflows with narrative text and visualization of results" [2, abstract] — and that combination is the reason they are both the best exploration tool available and a known reproducibility problem.

Formal definition

A notebook is a file and a protocol, and the split matters.

The file. "Jupyter Notebooks are structured data that represent your code, metadata, content, and outputs. When saved to disk, the notebook uses the extension .ipynb, and uses a JSON structure" [1, Architecture]. Every cell carries its source, its outputs, and its execution count — so the outputs are stored in the document rather than regenerated from it.

The kernel. Code does not run in the browser. A kernel "is a separate process which is responsible for running user code, and things like computing possible completions" [1, Architecture]. It holds all the state: your variables live in that process, not in the file.

The connection. "Frontends, like the notebook or the Qt console, communicate with the IPython Kernel using JSON messages sent over ZeroMQ sockets" [1, Architecture], mediated by a server — "the browser, notebook file on disk, and kernel cannot talk to each other directly. They communicate through the Jupyter server" [1, Architecture].

Three separate things, then: a document, a process, and a channel between them. Almost every confusing notebook behaviour comes from forgetting that the first two are not the same thing.

How it works

You run a cell; the frontend sends the source to the kernel; the kernel executes it against the state it already holds and returns outputs; the frontend writes those outputs back into the document [1, Architecture].

The execution count in the brackets increments once per execution, globally. That number is the only trace the file keeps of the order things actually happened in, and it is easy to make it disagree with the reading order — delete a cell whose variable is still live, edit and re-run cell 4 before cell 3, or restart the kernel while old outputs stay on screen. In each case the saved document is internally consistent and describes a computation that no longer exists.

The scale of the consequence has been measured. In an automated study of 27,271 notebooks from 2,660 GitHub repositories associated with 3,467 biomedical publications, 15,817 declared their dependencies; of the 10,388 whose dependencies could all be installed and which were then re-run, 1,203 ran through without any errors, and 879 produced results identical to those reported in the original notebook [3, abstract].

Read that last step against the first: fewer than one in twenty of the notebooks collected reproduced their own published results, and that is after excluding every notebook that could not even be installed.

Where it's used

• Exploratory data analysis — the case the tool is straightforwardly best at, and the one it was designed for.
• Teaching, where prose, code and output on one page is the whole point.
• Reproducible publication, when done deliberately: notebooks combined with software repositories and open licensing are described as powerful instruments for "transparent, collaborative, reproducible, and reusable data analyses" [2, abstract].
• Model prototyping — the left half of the ML workflow, before anything is packaged.
• Reporting and dashboards, via export to HTML or slides.
• Beyond Python. The kernel is a separate process speaking a documented protocol [1, Architecture], so R, Julia and others plug into the same frontend.

Why & when

Use a notebook while the question is still changing. Cheap iteration on expensive state is the thing it does that nothing else does.

Move to modules once the answer stops changing. Anything imported, tested, scheduled or reviewed belongs in a .py file. A notebook is a poor unit of software: it diffs badly, imports awkwardly, and tests not at all.

Treat "Restart and run all" as the definition of working. A notebook that only works in the order you happened to click is not a result — the measured reproduction rates above are what that habit looks like at scale [3, abstract].

Declare dependencies. In the same study, of the 22,578 Python notebooks only 15,817 had dependencies declared in standard requirement files, and only 10,388 of those could actually be installed [3, abstract]. Missing or uninstallable dependencies stopped 12,190 of them before they ran — more than the 9,185 that then failed with an exception. A notebook nobody can install has already failed before its logic is even examined.

Worked example

The audit worth running on any notebook you are about to trust — including your own. A notebook is JSON, so this needs no libraries at all.

import json

# A notebook is JSON. This is what the file actually looks like.
notebook = json.loads("""
{"nbformat": 4, "nbformat_minor": 5, "cells": [
  {"cell_type": "code", "execution_count": 1,
   "source": ["df = load('sales.csv')"], "outputs": []},
  {"cell_type": "code", "execution_count": 3,
   "source": ["df = df.dropna()"], "outputs": []},
  {"cell_type": "code", "execution_count": 2,
   "source": ["df['total'].mean()"],
   "outputs": [{"output_type": "execute_result", "data": {"text/plain": "84.7"}}]},
  {"cell_type": "code", "execution_count": null,
   "source": ["plot(df['total'])"], "outputs": []}
]}
""")

code = [c for c in notebook["cells"] if c["cell_type"] == "code"]
counts = [c["execution_count"] for c in code]
print("execution counts, in document order:", counts)

ran = [n for n in counts if n is not None]
print("never executed          :", counts.count(None))
print("executed out of order   :", ran != sorted(ran))
print("has output but stale    :",
      any(c["outputs"] and c["execution_count"] != i + 1
          for i, c in enumerate(code)))

execution counts, in document order: [1, 3, 2, None]
never executed          : 1
executed out of order   : True
has output but stale    : True

Every one of those flags is visible in the file, and none of them is visible in the rendered page a reader sees. The mean of 84.7 was computed at execution 2, before the dropna at execution 3 — so it is a mean over rows that the cleaned dataframe no longer contains. The notebook displays it confidently, directly below the cleaning step, in a layout that implies the opposite order.

Nothing here is a bug in Jupyter. The execution counts are recorded faithfully; they simply are not what anyone reads.

Pitfalls

Hidden state. Variables live in the kernel, not the document [1, Architecture]. A cell you deleted may still be holding the notebook up, and you will not find out until someone else runs it.

Out-of-order execution. The case above. The counters are the evidence; the layout is not.

Committing outputs. Outputs are stored in the JSON [1, Architecture], which makes notebooks diff badly, inflates repositories with base64 images, and is a straightforward way to commit data you meant to keep private.

Assuming it will run elsewhere. In the reproduction study, undeclared or uninstallable dependencies stopped more Python notebooks (12,190) than everything that went wrong while running (9,185) [3, abstract].

Letting it become the product. Notebooks were designed to be shareable documents [1, What is a notebook?], not deployable software. The migration to modules is not a formality anyone imposed on you; it is the point at which the tool stops matching the job.