Programming with Python and Jupyter

How Python runs analysis code and how Jupyter notebooks run it differently. Python's history from 1991 and the Python 3 break; why a language with slow loops became the language of data science; compilation to bytecode, names bound to objects and dynamic typing; the Jupyter kernel, cells and execution counts; and hidden state, shown by running the same three cells on 5,247 NHANES adults in different orders to get 16.85, 2.25 and 126.36. Ends with the evidence on notebook reproducibility (24% of 863,878 GitHub notebooks ran, 4% gave the same results) and the practices that address it.

Suppose an analysis notebook has three cells. The first selects the adults of the 2017 to 2018 National Health and Nutrition Examination Survey (NHANES) who have a systolic blood pressure reading, the second converts the readings from millimeters of mercury to kilopascals, and the third prints their mean:

bp = adults.loc[:, ["age", "sbp"]].dropna()     # cell A
bp["sbp"] = bp["sbp"] / KPA                     # cell B: mm Hg to kPa (KPA = 7.50062)
result = round(bp["sbp"].mean(), 2)             # cell C

Run from top to bottom, the notebook reports 16.85 kPa for the 5,247 adults. Run cell B a second time, as happens when someone edits it and presses Shift-Enter, and cell C reports 2.25, because the readings have now been divided twice. Run cell C before cell B and it reports 126.36, the mean in mm Hg, under a heading that says kilopascals. None of these runs raises an error, and the saved notebook may show any of the three numbers next to code that looks correct.

The example is small, but it captures what distinguishes programming for data analysis in Python and Jupyter from programming in general: the code is run interactively, a piece at a time, against data held in memory, and the order of execution is chosen by a person rather than fixed by the file. This article covers the language and the notebook from that point of view: where Python came from and why it is used for data science, what happens when Python runs code, and how the Jupyter notebook works, including the reproducibility problems its design invites and the practices that prevent them. The following articles take up the language itself (Python Data Types and Python Functions), installing and managing libraries (Data Science Packages) and the three libraries nearly every later article uses: NumPy, pandas and matplotlib.

All results in this article come from tools/experiments/data-science/python-and-jupyter.py in the site's repository, run with Python 3.13.5.

Python

Where it came from

Guido van Rossum created Python at the Centrum Wiskunde & Informatica (CWI) in the Netherlands (Python Software Foundation, 2025b), and released version 0.9.0 to the Usenet group alt.sources on 20 February 1991; version 1.0 followed on 26 January 1994 (van Rossum, 2009). The dates are sometimes given differently, with 1994 as the "initial release", but 1994 was version 1.0 of a language already in public use for three years.

The most consequential event in the language's history for anyone reading old analysis code is Python 3.0, released on 3 December 2008. It changed the language in ways that made much Python 2 code invalid or subtly different: print became a function, text strings became Unicode by default with a separate bytes type, and dividing two integers with / began to return a floating-point result rather than truncating (7 / 2 is 3.5 in Python 3 and was 3 in Python 2) (Python Software Foundation, 2025c). Python 2 remained in wide use for a decade afterwards; its last version, 2.7, reached the end of its support on 1 January 2020, and its final release, 2.7.18, came out in April 2020 (Peterson, 2008). Everything on this site uses Python 3, and the minor versions (3.11, 3.12, 3.13) add features while keeping old code working, with occasional changes in detail. One such change appears in Python Data Types: from Python 3.12 the built-in sum adds floating-point numbers more accurately, so the same line can print a different last digit in 3.11 and 3.13.

Why data science uses it

Python was not designed for numerical work, and a loop written in plain Python is slow: each step looks up the types of its operands at run time and allocates a new object for each intermediate result. The NumPy Package measures the cost on this site's machine: standardizing 5,232 blood pressure readings with a Python loop takes about 30 times as long as the same computation on a NumPy array, and a sum of squares of a million numbers several hundred times as long as NumPy's dot. Memory is similar: a Python list of 5,232 floating-point numbers occupies 172,728 bytes, counting the list and the number objects it points to, against 41,856 bytes (8 per number) in an array.

The language became the common language of data science in spite of this, for reasons the history of its libraries explains. Python is easy to call C and Fortran from, so the expensive inner loops could be written once in a compiled language and wrapped. Harris et al. (2020) trace the array libraries from Numeric in the mid-1990s through Numarray to NumPy in 2005, and describe the result as an ecosystem in which "NumPy underpins almost every Python library that does scientific or numerical computation". Python itself, in their words, is "an open-source, general-purpose interpreted programming language well suited to standard programming tasks such as cleaning data, interacting with web resources and parsing text", so the same program that fits a model can also download the data, parse its documentation and write the report. The combination, a readable general-purpose language for the glue and compiled libraries for the arithmetic, is the pattern every later article in this collection follows: the analysis is written in Python and the work is done by NumPy, pandas, scikit-learn and the rest.

The trade-offs are real. Code that cannot be expressed as operations on whole arrays stays slow; libraries differ in their conventions; and a program's correctness depends on the versions of dozens of packages, a problem Data Science Packages takes up.

How Python runs code

Compilation, then execution

Python is often called an interpreted language, and the description is close enough for most purposes, but the standard implementation, CPython, does not translate and run a file one line at a time. It first compiles the whole file (or the whole cell, in a notebook) into bytecode, "the internal representation of a Python program in the CPython interpreter" (Python Software Foundation, 2025a), and then a loop in the interpreter executes the bytecode instructions. The difference is visible whenever there is a syntax error late in a file:

print("line 1 ran")
x = (1 +

Executing this source never prints anything. It fails with SyntaxError: '(' was never closed, reported on line 2, before line 1 has run, because the compiler rejected the whole text. Errors that depend on values, such as dividing by zero or using a name that does not exist, are found only when the line containing them executes; syntax errors are found first, for the whole unit.

The bytecode is a short list of simple instructions. The function

def avg(a, b):
    return (a + b) / 2

compiles in Python 3.13 to six instructions: RESUME, LOAD_FAST_LOAD_FAST (load both arguments), BINARY_OP (add), LOAD_CONST (the 2), BINARY_OP (divide) and RETURN_VALUE, as the standard library's dis module shows. Nothing in the instructions says what kind of numbers a and b are. Each BINARY_OP discovers the types of its operands when it runs and dispatches to the right addition, integer, floating-point, or the concatenation of two lists. That run-time dispatch is what makes Python flexible, and repeated for every element of a large dataset, it is also what makes Python loops slow.

Names, objects and dynamic typing

An assignment such as predicted = 90 is often described as storing 90 "in" a variable. A more accurate model, and one that explains much of Python's behavior, is that the program contains objects, each with a type, a value and an identity, and names that are bound to objects. The assignment creates (or finds) an integer object with value 90 and binds the name predicted to it. A name has no type of its own: predicted = "ninety" later in the program binds the same name to a string object. This is what dynamic typing means. It does not mean that types are loose: "120" + 1 raises TypeError: can only concatenate str (not "int") to str, because a string and an integer cannot be added and Python does not guess. Python is dynamically but strongly typed.

Binding several names to one object is where the model matters. After

a = [120, 135]
b = a
b.append(150)

the list that a names is [120, 135, 150], because b = a bound a second name to the same list and append changed that list; a is b is True. The statement b = b + [160], by contrast, builds a new list and rebinds b to it, leaving a as it was. Python Data Types draws this picture and follows its consequences, which include several of the most common errors in analysis code, and Python Functions shows that passing an argument to a function binds a name in the same way.

Jupyter

From IPython to Jupyter

Interactive computing in Python began with the read-eval-print loop of the standard interpreter: type a statement, see its result, keep the variables. IPython, described by Pérez and Granger (2007), made that loop more capable for scientific work and later added a browser-based notebook. The language-independent parts were then separated into Project Jupyter, with IPython remaining its default Python kernel. The name comes from "the three programming languages the project originally supported: Julia (ju), Python (pyt) and R (r)" (Project Jupyter, 2025). Kluyver et al. (2016) present the notebook as a publishing format for research: a single document that holds code, its results and the explanation around them, readable by a person and executable by a computer. That states both the promise and, as the next sections show, the difficulty.

Kernels, cells and execution counts

A notebook is two programs working together. The editor (JupyterLab, the classic Notebook interface, or an editor such as Visual Studio Code) shows the document in a browser or window. The kernel is a separate operating-system process, one per open notebook, that runs the code. The Jupyter documentation describes each kernel as a read-eval-print loop that "holds the data and objects you create with your notebook code in a long-running program on your computer" (Project Jupyter, 2025). When a cell is run, the editor sends its text to the kernel, the kernel compiles and executes it in a single shared namespace, and the outputs are sent back and displayed under the cell.

Cells come in two kinds. Code cells contain code; each run is numbered by the kernel, which is the number shown as In [7]: beside the cell. Markdown cells contain formatted text, headings and, on most installations, mathematics. The file saved to disk, with the extension .ipynb, is a JSON document that records the cells, their outputs (text, tables, images) and, for each code cell, its execution_count, the number of the run that produced the stored output (nbformat, 2025). The file therefore holds both the code and a record of one particular history of running it, and the two need not agree.

Hidden state

Because the kernel keeps one namespace for the life of the session, the state of a running notebook is the result of every cell executed so far, in the order they were executed, including cells that have since been edited or deleted. Nothing in the document shows that history except the execution counts. This is called hidden state, and the opening example shows its three common forms. The script for this article simulates a kernel by executing cell texts in one shared namespace, on the 5,247 NHANES adults with a systolic reading:

Order of execution What happened Printed mean
A, B, C top to bottom 16.85
A, B, B, C cell B re-run after an edit 2.25
A, C, B cell C run before cell B 126.36

The fourth form is a deleted cell. Suppose the conversion is written with a helper function, to_kpa, defined in a cell D that is later deleted because it looked unused. The notebook keeps working, since the function still exists in the kernel, and reports 16.85. Restart the kernel and run all cells from the top, and the conversion cell fails with NameError: name 'to_kpa' is not defined. The notebook as saved could never have produced its own outputs.

The way to find all four is the same: restart the kernel and run every cell in order ("Restart Kernel and Run All Cells"). If the outputs are the same, the document is a faithful record of a computation; if not, the difference is the hidden state.

What happens to notebooks in practice

Pimentel, Murta, Braganholo and Freire (2019) measured how often notebooks reproduce. They collected 1,159,166 unique notebooks from 264,023 GitHub repositories and tried to re-execute the 863,878 valid Python notebooks whose execution order could be determined, following the order recorded in their execution counts. Of those, 24.11% ran without errors, and 4.03% produced the same results as the outputs stored in the file. Among notebooks with an unambiguous execution order, 36.36% had cells run out of order, and 76.90% had at least one gap in their execution counts, a sign of cells run and then deleted or re-run. The most common failures were missing modules (ImportError and ModuleNotFoundError, 29.23% of executions), names used before definition (NameError, 14.53%), and missing data files (FileNotFoundError and IOError, 12.59%), often because the notebook used an absolute path on its author's computer.

The figures need care in interpretation. The authors note that some notebooks were never meant to be rerun (a keyword search, which they caution is rough, suggests that about 22% are course exercises), that they ran cells in execution-count order rather than top to bottom, and that their environments could only approximate the original authors' (for repositories that declared no dependencies, they installed the whole Anaconda distribution). Still, the leading causes they identify, undeclared dependencies, hidden state and inaccessible data, are the same ones the opening example illustrates, and none of them is a property of Python; they are properties of how notebooks are used.

Practices that make a notebook reproducible

Pimentel et al. (2019) close with a list of practices. The list below takes the ones that matter most for analysis (restarting and running all, declaring pinned dependencies, relative paths, and moving code into tested modules) and adds two that follow from the examples in this collection, cells that are safe to re-run and seeded random numbers:

  • Restart and run all before sharing or saving results. It is the only check that the stored outputs come from the stored code, and it resets the execution counts to 1, 2, 3, ..., so that a reader of the file can see that the outputs came from a single run in document order.
  • Make cells safe to re-run. A cell that modifies its input in place, like cell B, gives a different answer each time. Writing the result to a new name (bp_kpa = bp["sbp"] / KPA) makes the cell idempotent.
  • Put imports at the top and declare dependencies with versions. Pimentel et al. recommend listing them in a requirements file and "pin the versions of all packages", then test the list in a clean environment; Data Science Packages shows how.
  • Use relative paths and include, or document, the data. A notebook that reads C:\Users\me\Downloads\data.csv runs on one computer. Data Engineering describes how this site's scripts download each public file and check it against a recorded hash, so that the data, not only the code, are pinned.
  • Move reusable code into functions and modules, and test them. Functions defined in a notebook can be tested only by running the notebook; a function in a .py file can be imported by the notebook and by a test.
  • Seed every random number generator. A bootstrap or a train-test split without a seed gives different numbers on every run. The NumPy Package shows NumPy's seeded generators.

Notebooks can also be executed without the editor, top to bottom in a fresh kernel, with jupyter nbconvert --to notebook --execute analysis.ipynb, which makes "restart and run all" something a scheduled job or a continuous-integration check can do.

Notebooks, scripts and the editor

Notebooks are best at what the opening paragraph described: exploring data a step at a time, with results, plots and notes beside the code that produced them. Scripts, plain .py files run from start to finish by python analysis.py, have no hidden state, because every run starts from nothing, and they work naturally with version control, testing and scheduling. Most data scientists use both, exploring in a notebook and moving settled code into modules and scripts, and modern editors blur the line by running parts of a script interactively against a kernel.

This site takes the script side for everything it publishes. Each article's numbers come from a script in tools/experiments/data-science/, run from a fresh interpreter, which downloads its data, checks them, seeds its random numbers and prints every value the article quotes. A reader can rerun the script and compare. The same discipline can be kept in a notebook, but the script makes it the default rather than a habit.

References