The same two lines of pandas,
df = pd.DataFrame({"sbp": [202.0, 121.0, 101.0], "age": [66, 18, 75]})
df["sbp"][df["age"] > 60] = 0.0
set the systolic pressures of the two people over 60 to zero under pandas 2.2.3, printing two warnings, and change nothing under pandas 3.1.1, printing a different warning. Both runs used Python 3.13.5 on the same computer; only the installed version of one library differed. An analysis that relies on the first behavior gives a different answer, without an error, when someone reruns it after an upgrade.
Python's role in data science rests on libraries that are not part of Python: NumPy, pandas, matplotlib, SciPy, scikit-learn and many more, each released on its own schedule and each depending on others. This article explains how Python loads them, how they are installed, how environments keep them separate, and how versions are recorded so that an analysis can be rerun with the libraries it was written for. It ends with a map of the core packages. Programming with Python and Jupyter describes the notebook side of the same reproducibility problem.
The results here come from tools/experiments/data-science/python-packages-for-data-science.py in the site's repository, run in two environments on the same machine: the system installation of Python 3.13.5, and a virtual environment, also Python 3.13.5, with newer libraries.
Modules and packages
A module is a file of Python code, statistics.py for instance, whose functions and variables become available when it is imported. A package is a directory of modules with an __init__.py file, which may contain further packages: matplotlib is a package and matplotlib.pyplot a module inside it. Python ships with a large standard library of modules (math, statistics, json, sqlite3, csv), and everything else is installed separately. What is installed is a distribution, the unit published on the Python Package Index; its name can differ from the package it provides, as the distribution scikit-learn provides the package sklearn.
What import does
import numpy as np
from pandas import read_csv
The first statement makes the module numpy available under the name np; the second binds only the name read_csv. Either way the whole module is loaded, since read_csv needs the rest of pandas to work; from ... import changes which names are bound, not how much is loaded. The aliases np, pd and plt are conventions rather than rules, but conventions strong enough that other names make code harder to read. from module import *, which binds every public name of a module, is best avoided in analysis code, because it hides where names come from and can silently replace names already defined.
Importing proceeds in steps that explain several common errors (Python Software Foundation, 2025a):
- Python looks the name up in
sys.modules, the cache of modules already imported in this process. If it is there, import binds the name and stops. A secondimport numpyin the same session returns the same module object (the script checks thatsys.modules["numpy"] is np), so editing a module's file has no effect on a running session or notebook kernel until the kernel is restarted, or the module reloaded withimportlib.reload. - Otherwise Python searches the directories in
sys.pathin order, and loads and executes the first matching module or package it finds, which runs all of its top-level code.
Importing pandas is not small. In a fresh interpreter, import pandas loaded 543 further modules under pandas 2.2.3 and 565 under pandas 3.1.1, among them 295 or 294 modules of pandas itself and 99 or 100 of NumPy, which pandas imports for its own use.
The search path, and shadowing
The order of sys.path matters because the first match wins. Its first entry is "the directory that contains the input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a -c command, or -m module" (Python Software Foundation, 2025b). The standard library and installed packages come later. A file in the working folder with the same name as a library therefore replaces the library. The script demonstrates this in a temporary folder containing a file statistics.py that defines its own mean. There,
python -c "import statistics; print(statistics.mean([1, 2, 3]))"
prints local statistics.py, not the standard library instead of 2. Run in isolated mode, python -I -c ..., which leaves the script's directory and the current directory out of the path, the same command prints 2.
The usual victims are scripts named after what they are about: random.py, statistics.py, numpy.py, pandas.py. A script named pandas.py that begins with import pandas imports itself. The consequence reached this collection directly. The articles' scripts live in one folder, and the script for the NumPy article could not be called numpy.py, because every other script in that folder imports NumPy and would have found it first; it is called numpy-package.py, and the pandas and matplotlib scripts are named the same way. A name with a hyphen cannot be imported at all, which makes it safe for a script.
Installing packages
pip and the Package Index
pip installs distributions from the Python Package Index (PyPI), the public repository of Python packages, and resolves their dependencies: installing pandas also installs the packages pandas declares it needs. Those declarations are part of the installed metadata and can be read with importlib.metadata.requires("pandas"). For pandas 2.2.3 they are NumPy (at least 1.26.0 on Python 3.12 and later), python-dateutil, pytz and tzdata; for pandas 3.1.1 they are NumPy 2.0.2 or later, python-dateutil and, on Windows, tzdata. pytz is gone, and the minimum NumPy has moved to the 2.x series. Most distributions are published as wheels, prebuilt archives for each platform and Python version, so installing NumPy does not require a C compiler.
The pip documentation writes its commands as python -m pip install pandas rather than pip install pandas, explaining that this form "executes pip using the Python interpreter you specified as python" (Python Packaging Authority, 2025). It installs into that interpreter's environment, whereas a bare pip command is whichever one the shell finds first, which on a machine with several Pythons may belong to a different installation from the one the notebook kernel runs. The mismatch produces the confusing case of a package that installs successfully and then raises ModuleNotFoundError on import. In a notebook, the %pip install magic command installs into the kernel's own environment for the same reason, and the kernel must usually be restarted before a newly installed version of an already imported package takes effect, since step 1 above returns the cached module.
Virtual environments
Installing every project's packages into one Python installation fails as soon as two projects need different versions of the same package. A virtual environment is a directory with its own site-packages folder, linked to a base Python installation. The standard library's venv module "supports creating lightweight 'virtual environments', each with their own independent set of Python packages installed in their site directories" (Python Software Foundation, 2025c):
python -m venv .venv # create
.venv\Scripts\activate # activate (Windows); source .venv/bin/activate elsewhere
python -m pip install pandas==3.1.1 # installs into .venv only
Inside an environment sys.prefix differs from sys.base_prefix, which is how the script reports whether it is running in one. Some operating systems now refuse to let pip install into the system Python at all: PEP 668 lets an installation mark itself as "externally managed", in which case pip should "neither install nor remove packages into the interpreter's global context, by default, and should instead guide the end user towards using a virtual environment" (Thomas et al., 2021). The intent is the practice this section recommends, one environment per project.
conda
conda, the package manager of the Anaconda and Miniconda distributions and of the community conda-forge channel, takes a different approach. It installs packages of any language, not only Python, so an environment can include compiled libraries such as BLAS, CUDA or GDAL that pip cannot manage, and it manages environments and the Python interpreter itself. Its packages come from conda's own channels rather than from PyPI. conda and pip can be mixed in one environment, but each tracks only what it installed itself, so mixing them makes the environment harder to reproduce; the usual advice is to install as much as possible with one tool.
Versions and pinning
Packages number their releases by the scheme of PEP 440, usually as major.minor.micro, and declare and accept ranges of versions with specifiers (Coghlan and Stufft, 2013). How the common specifiers match five versions of pandas, as the script computes with the packaging library that pip itself uses:
| Specifier | 2.2.3 | 2.2.4 | 2.3.1 | 3.0.0 | 3.1.1 |
|---|---|---|---|---|---|
==2.2.3 |
yes | ||||
==2.2.* |
yes | yes | |||
~=2.2 |
yes | yes | yes | ||
>=2.2,<3 |
yes | yes | yes | ||
>=2.2 |
yes | yes | yes | yes | yes |
The compatible release operator ~=2.2 means "2.2 or later within the 2 series", >= 2.2, == 2.*. It encodes the assumption, common but not guaranteed, that a change in the first number is the only kind that breaks code. pandas 3.0 is such a change: its release notes recommend upgrading to pandas 2.3 first and fixing every warning before moving to 3.0 (pandas development team, 2026).
Pinning records the exact version of every installed package, including dependencies of dependencies, usually in a requirements.txt file written by python -m pip freeze > requirements.txt and restored with python -m pip install -r requirements.txt. The pip documentation describes a frozen requirements file as one that "contains a pinned version of everything that was installed", for the purpose of repeatable installs (Python Packaging Authority, 2025). Pimentel et al. (2019), who tried to rerun 863,878 Jupyter notebooks from GitHub, found that missing modules were the most common single cause of failure (29.23% of executions) and recommend declaring dependencies in requirements files with all versions pinned, then testing them in a clean environment. This site pins more lightly. The production requirements.txt lists the four packages the deployed site depends on directly, each with == and an exact version, but not their own dependencies, which pip resolves at install time; a separate requirements-dev.txt adds pinned tools used only for local work, such as matplotlib for the figures, and is never installed on the server.
Two environments, one script
The script was run in both environments with no changes:
| Package | System Python 3.13.5 | Virtual environment, Python 3.13.5 |
|---|---|---|
| NumPy | 2.2.5 | 2.5.4 |
| pandas | 2.2.3 | 3.1.1 |
| matplotlib | 3.10.1 | 3.11.2 |
| SciPy | 1.15.2 | 1.18.1 |
| scikit-learn | 1.6.1 | not installed |
| seaborn | not installed | 0.13.2 |
The pandas section of the script prints:
| Code | pandas 2.2.3 | pandas 3.1.1 |
|---|---|---|
pd.Series(["a", "b"]).dtype |
object |
str |
sub = df["sbp"]; sub.iloc[0] = 0 |
changes df (with SettingWithCopyWarning) |
leaves df unchanged |
df["sbp"][df["age"] > 60] = 0 |
changes df (FutureWarning, SettingWithCopyWarning) |
leaves df unchanged (ChainedAssignmentError warning) |
df.loc[df["age"] > 60, "sbp"] = 0 |
changes df |
changes df |
The first row is pandas 3's new default string type and the next two are its copy-on-write rules; The pandas Package explains both. The last row is the form that works in every version, and it is what both kinds of warning recommend. Two lessons follow. State the versions with the results, as this collection's articles do when a number depends on them. And treat warnings as information: pandas 2.2 announced the change in behavior with a FutureWarning on exactly the line that later changed.
Pinning code is half of reproducibility. The other half is pinning data, recording exactly which input files an analysis read, which Data Engineering covers with the cryptographic hashes this site's scripts check before using any public file.
The core packages
Harris et al. (2020) draw the scientific Python ecosystem as layers: a foundation of the Python language, NumPy and IPython or Jupyter; on top of it, technique-specific libraries such as SciPy for algorithms, matplotlib for plots, pandas and statsmodels for statistics and scikit-learn for machine learning; and domain-specific libraries above those. The packages a data scientist meets first, and where this collection uses them:
NumPy (import numpy as np) provides the n-dimensional array, fixed-width numerical types, fast element-wise arithmetic, linear algebra and random number generation. Almost every other package here stores its numbers in NumPy arrays. The NumPy Package.
pandas (import pandas as pd) provides labeled tables (DataFrame) and columns (Series), with reading and writing of files and databases, alignment by label, missing-value handling, grouping, reshaping and joins. The pandas Package introduces it, and the Data Wrangling articles use it throughout, from Introduction to Data Wrangling to Enriching Data; Queries in Python connects it to SQL databases.
matplotlib (import matplotlib.pyplot as plt) draws figures, from quick plots to publication graphics, and saves them as PNG, SVG or PDF. Every figure on this site is drawn with it. The matplotlib Package.
seaborn (import seaborn as sns) is a declarative statistical plotting library built on matplotlib, which takes a DataFrame and column names and computes summaries such as density estimates and confidence intervals as it draws. Tools for Visualizing Data compares it with matplotlib, plotly and others.
SciPy (import scipy) collects scientific algorithms: optimization, integration, interpolation, signal processing, sparse matrices and, in scipy.stats, probability distributions and statistical tests.
statsmodels (import statsmodels.api as sm) fits statistical models with an emphasis on inference: regression tables with standard errors, confidence intervals and tests, generalized linear models and time series models. The regression articles, from Simple Linear Regression to Logistic Regression, cover the inference these tables report.
scikit-learn (import sklearn) implements machine learning with a uniform interface: every model has fit and predict, every transformation fit and transform, and pipelines, cross-validation and model selection are built from those parts. The model evaluation and machine learning articles, such as Cross-Validation and Random Forests, use it.
The division between the last two is one of emphasis, not of capability. statsmodels is usually the choice when the question is about a coefficient and its uncertainty, scikit-learn when it is about predictions on new data, and the two often fit the same model: least squares and logistic regression are in both.
References
- Coghlan, A., and Stufft, D. (2013). PEP 440: Version Identification and Dependency Specification. Python Software Foundation. https://peps.python.org/pep-0440/ record
- Thomas, G., Klose, M., Laíns, F., Stufft, D., Chung, T., Rivera, S., Hashman, E., and Gedam, P. (2021). PEP 668: Marking Python base environments as "externally managed". Python Software Foundation. https://peps.python.org/pep-0668/ record
- Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., ... and Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585, 357–362. https://doi.org/10.1038/s41586-020-2649-2 record · 3 pages
- pandas development team (2026). What's new in 3.0.0 (January 21, 2026). pandas documentation. https://pandas.pydata.org/docs/whatsnew/v3.0.0.html record · 2 pages
- Pimentel, J. F., Murta, L., Braganholo, V., and Freire, J. (2019). A large-scale study about quality and reproducibility of Jupyter notebooks. In Proceedings of the 16th International Conference on Mining Software Repositories (MSR 2019) (pp. 507–517). IEEE. https://doi.org/10.1109/MSR.2019.00077 record · 3 pages
- Python Packaging Authority (2025). User Guide. pip documentation. https://pip.pypa.io/en/stable/user_guide/ record
- Python Software Foundation (2025a). The Python Language Reference, section 5, "The import system". https://docs.python.org/3/reference/import.html record · 2 pages
- Python Software Foundation (2025b). The initialization of the sys.path module search path. Python 3 documentation. https://docs.python.org/3/library/sys_path_init.html record
- Python Software Foundation (2025c). venv: Creation of virtual environments. Python 3 documentation. https://docs.python.org/3/library/venv.html record