The pandas Package

An introduction to pandas through NHANES data, with the behavior that differs between pandas 2.2 and 3.1. Series, DataFrame and Index, and the automatic alignment by label that McKinney designed in (dividing 8,704 readings by 9,254 ages gives 9,254 results, 6,302 of them numbers); dtypes, NaN in float columns, nullable Int64 and the new str dtype; loc against iloc, where a sorted table makes loc[0:2] return 1,022 rows; Boolean masks and operator precedence; copy-on-write and the end of chained assignment; method chains; and row-wise apply, about 300 times slower than vectorized arithmetic.

The 2017 to 2018 NHANES demographics file has one row for each of the 9,254 people interviewed, with their age. The blood pressure file has one row for each of the 8,704 who were examined, with up to three systolic readings; the first reading is present for 6,302 of them. Put each file's column in a pandas Series labeled by the respondent number, divide the first systolic reading by age, and the result has 9,254 entries: one for every respondent in either file, with a number for the 6,302 who have both a reading and an age, and a missing value for the rest. The rows were matched by respondent number, not by position, and nothing had to be sorted or joined first. The same division on the bare NumPy arrays fails, operands could not be broadcast together with shapes (8704,) (9254,), and if the lengths had happened to agree, it would have divided each reading by some other person's age without complaint.

Matching by label is the central idea of pandas. Wes McKinney, who began the library for financial data at AQR Capital Management, described it in the paper that introduced it: "the data have been automatically aligned based on their labels and added together. The result object contains the union of the labels between the two objects so that no information is lost" (McKinney, 2010). Tables with labeled rows and columns, missing values and columns of different types are the form in which most real data arrive, and pandas is the standard Python library for them.

This article introduces pandas' data structures, its types, how to select data, the copy-on-write rules of pandas 3, and the style of writing transformations as chains of methods. The Data Wrangling articles use pandas throughout and are the place for its larger operations: grouping and pivot tables in Exploring Data with Groups and Pivot Tables, reshaping and types in Structuring Data, missing values in Cleaning Data, and joins in Enriching Data. All results come from tools/experiments/data-science/pandas-package.py in the site's repository, run twice on the same machine with Python 3.13.5: once with pandas 2.2.3 and NumPy 2.2.5, once with pandas 3.1.1 and NumPy 2.5.4. Where the two differ, both results are given.

Series, DataFrame and Index

A Series is a one-dimensional array of values with an index, a label for each value. A DataFrame is a table: an ordered collection of columns, each a Series, sharing one row index, with a second index of column names. The name comes from "panel data, which is a term for 3-dimensional data sets encountered in statistics and econometrics" (McKinney, 2010); the three-dimensional Panel object of the early versions was removed in pandas 0.25.0 in 2019 (pandas development team, 2019), and the two-dimensional DataFrame became the library's center.

The Index is more than a list of row names. McKinney (2010) explains that an Index stores its labels both as an array and as a hash table mapping labels to positions, so that looking up a label or testing membership takes constant time; its labels should be unique for that reason, and age.index.is_unique is True for the respondent numbers. age.loc[93705] returns 66, the age of respondent 93705, by label. The default index, when none is given, is the integers 0, 1, 2, ..., which look like positions but are labels, a distinction that matters for loc below.

Alignment

Every operation between two pandas objects first aligns them on their labels. In the opening example, sys1 / age produced a Series whose index is the union of the two indexes, age.index.union(sys1.index), which the script confirms: here every examined person was also interviewed, so the union is the 9,254 interviewed respondents. Reversing the order of one Series before an operation makes no difference: sys1 - sys1.iloc[::-1] is zero everywhere, since each value is matched with itself by label. Arithmetic between a DataFrame and a Series aligns the Series' index with the DataFrame's columns by default, and assigning a Series to a DataFrame column aligns it with the DataFrame's rows.

Alignment is why pandas code can combine data from different sources so concisely, and also why it can surprise. Values whose labels have no partner become missing rather than raising an error, so a typo in an index, or respondent numbers stored as floats in one table and integers in another, produces a column of missing values instead of a failure. The habit that catches this is to count missing values after every combining step, as the wrangling articles do.

Types

Each column has one dtype. Reading a file assigns them, and they are often not what the data mean. Every column of an NHANES file arrives as float64, including the respondent number, which reads as 93703.0, because of the file format (Structuring Data explains and converts them).

Missing values in numerical columns

pandas marks a missing number with NaN, the floating-point "not a number" value. McKinney (2010) gave the reasons and the cost: NaN was chosen "for performance reasons", because it "propagates in floating-point operations in a natural way", but it "cannot be used in integer-type arrays". So a column of whole numbers with any missing value becomes float64. The first systolic reading, all even whole numbers, is float64 with 2,402 NaN values, and s.astype("int64") fails with IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.

pandas later added nullable dtypes that keep a separate mask of missing positions: Int64 (capital I), Float64, boolean and string. s.astype("Int64") converts the readings, shows missing values as <NA>, and computes the mean of the 6,302 present readings as 121.33. The missing-value marker of these types, pd.NA, propagates through comparisons as well as arithmetic: pd.NA + 1 is <NA>, and so is pd.NA == pd.NA, which is neither True nor False but unknown: the nullable types follow three-valued logic, as SQL does for NULL (pandas development team, 2026c; Special Operators and Clauses). df.convert_dtypes() converts a whole table to the nullable types; on four columns of the adults table it gives Int64 for the respondent number, Float64 for systolic pressure, string for smoking status and boolean for a below-detection-limit flag.

Strings and categories

Before pandas 3, text columns had dtype object, an array of references to arbitrary Python objects. pandas 3.0 made a dedicated string dtype the default: "pandas will start inferring columns containing string data as the new str data type", which can hold only strings or missing values, and is "backed by PyArrow under the hood, if installed, otherwise falling back to being backed by NumPy object-dtype" (pandas development team, 2026). The script shows the change directly: pd.Series(["a", "b"]).dtype is object under pandas 2.2.3 and str under 3.1.1, and the adults' smoking-status column likewise. Without PyArrow installed, as here, the memory used is the same, 301,986 bytes for the 5,533 values in both versions. Code that tests dtype == object to find text columns behaves differently under pandas 3.

A column with a few distinct values repeated many times is better stored as a category: an array of small integer codes plus a table of the distinct values. Smoking status, with three values, takes 5,806 bytes as a category in pandas 2.2.3 and 5,698 in 3.1.1, about 2% of the string column, with codes of dtype int8. Categories can also carry an order (for age groups, say), which sorting and plotting respect.

Selecting data

Columns, loc and iloc

df["sbp"] selects one column as a Series; df[cols], with a list such as cols = ["age", "sbp"], selects a DataFrame of those columns. Rows and columns together are selected with two indexers that differ in what they read:

  • loc selects by label. Slices of labels include both ends.
  • iloc selects by integer position, like Python lists and NumPy arrays. Slices exclude the end.

On a table with the default index the two look interchangeable, apart from the endpoint: adults.loc[0:2] returns 3 rows, labels 0, 1 and 2, and adults.iloc[0:2] returns 2. They stop looking alike after any operation that reorders or filters rows, because the labels travel with the rows. Sorted by systolic pressure in descending order, the table's first three labels are 3174, 846 and 5176; srt.iloc[0:3] returns those three rows, the highest pressures, while srt.loc[0:2] returns the 1,022 rows that lie between label 0 and label 2 in the new order. srt.loc[labels], with a list labels = [0, 1, 2], returns the three rows labeled 0, 1 and 2, wherever they now are. The rule is to use iloc for positions and loc for labels, and never to rely on the default index meaning both.

Boolean masks

A comparison on a column gives a Boolean Series, and indexing with it keeps the rows where it is True: adults[adults.age >= 60]. Conditions are combined with & (and), | (or) and ~ (not), element by element. Each condition needs its own parentheses, because & and | bind more tightly than comparisons. Without them,

adults[adults.age >= 60 & adults.sex == 2]

is read as adults.age >= (60 & adults.sex) == 2, a chained comparison, which Python evaluates with an implicit and between two Series and fails with ValueError: The truth value of a Series is ambiguous. The same error comes from writing and instead of &. Parenthesized, (adults.age >= 60) & (adults.sex == 2) selects 996 women aged 60 and over, as does the string form adults.query("age >= 60 and sex == 2").

Negation and missing values need care. A comparison with NaN is False, so adults.sbp >= 130 is True for 1,900 adults and False for the other 3,633, including the 286 with no reading. ~(adults.sbp >= 130) therefore selects 3,633 adults, while adults.sbp < 130 selects 3,347. "Not high" and "low" differ by exactly the missing values.

Copy-on-write

In pandas 2 and earlier, whether a selection was a view sharing memory with its source or an independent copy depended on the operation, the dtypes and the memory layout, and the documentation described the behavior as "tricky to understand" (pandas development team, 2026b). The script's example, on a three-row copy of the adults table:

col = df["sbp"]
col.iloc[0] = 0.0

Under pandas 2.2.3 this changes df as well, with a SettingWithCopyWarning, because col happened to be a view. Under pandas 3.1.1 df is unchanged.

pandas 3.0 adopted copy-on-write as its only mode. Its rule is simple to state: "any subset or returned series/dataframe always behaves as a copy of the original, and thus never modifies the original" (pandas development team, 2026a). Underneath, pandas still shares memory wherever it can and copies only when one of the objects is modified, which is where the name comes from; the sharing is invisible, and only the behavior of a copy is promised. Two consequences show up in everyday code.

Chained assignment no longer works. df["sbp"][df["age"] >= 60] = 0 selects a column and then assigns into it, two indexing steps. Under pandas 2.2.3 it changed df, with a FutureWarning and a SettingWithCopyWarning; under pandas 3.1.1 it leaves df unchanged and issues a ChainedAssignmentError warning. The form that works in every version is a single loc assignment: df.loc[df["age"] >= 60, "sbp"] = 0. The SettingWithCopyWarning itself is gone in pandas 3, and the defensive .copy() calls written to silence it are no longer needed.

Arrays taken from a table are read-only when they share its memory. adults["age"].to_numpy() is writeable under pandas 2.2.3 and read-only under 3.1.1, since writing into it would change the table behind the user's back.

The inplace=True argument of many methods predates these rules. It modifies the object and returns None (as sort_values(inplace=True) does in the script), so it cannot be used in a chain. Under copy-on-write, assigning the result back to the same name, df = df.sort_values("sbp"), is the clearer form: once the old object is no longer referenced, later changes to df need no copy (pandas development team, 2026b). pandas 3.1 goes further and deprecates inplace for sort_values and the other methods that never really worked in place (pandas development team, 2026d); under pandas 3.1.1 the script's call prints a deprecation warning.

Method chains

Because most methods return a new object, a sequence of transformations can be written as one expression, each step applied to the result of the last:

table = (
    adults
    .dropna(subset=["sbp"])
    .assign(age_group=lambda d: pd.cut(d.age, [17, 39, 59, 80], labels=["18-39", "40-59", "60+"]),
            sex=lambda d: d.sex.map({1: "male", 2: "female"}))
    .groupby(["age_group", "sex"], observed=True)
    .sbp.agg(["count", "mean"])
    .round(1)
    .unstack("sex")
)
Age group Women Men Women: mean systolic Men: mean systolic
18 to 39 921 824 110.8 119.3
40 to 59 836 743 126.2 126.5
60 and over 943 980 138.8 134.9

The counts add up to the 1,745, 1,579 and 1,923 adults of the three age groups in Introduction to Data Wrangling. Younger women have lower systolic pressure than younger men, the difference vanishes in middle age, and over 60 it reverses.

assign adds columns and takes functions (here lambdas) that receive the intermediate table, so each step can refer to columns created earlier in the chain without naming a temporary variable; pipe(f) passes the table to any function f. A chain reads in the order the operations happen, leaves no half-finished intermediate tables in a notebook's namespace, and never modifies adults, which makes it safe to re-run (Programming with Python and Jupyter). It is harder to inspect in the middle; when a step needs checking, the chain can be cut there and the intermediate result examined.

Vectorized, not row by row

pandas operations on whole columns run in compiled code over NumPy arrays (The NumPy Package). DataFrame.apply(f, axis=1) instead calls a Python function once per row, building a Series for each row first. Computing body mass index from weight and height for 5,434 adults:

row = sub.apply(lambda r: r.weight / (r.height / 100) ** 2, axis=1)
vec = sub.weight / (sub.height / 100) ** 2

gives the same values both ways, and agrees with the BMI that NHANES publishes to within 0.05 (the published value is rounded to one decimal). In four runs of the script the row-wise version took between 280 and 330 times as long as the vectorized one under pandas 2.2.3, and between 340 and 380 times as long under pandas 3.1.1. As with NumPy, the times depend on the machine, and the ratio is the useful part: apply along rows is a last resort for logic that cannot be written with column operations, np.where or Series.map.

Missing values, briefly

Most pandas reductions skip missing values by default: adults.sbp.mean() is 126.36, the mean of the 5,247 values present, while adults.sbp.mean(skipna=False) is NaN, and adults.sbp.count() reports the 5,247 that were used out of 5,533 rows. Skipping is convenient and silent, so a reported mean should come with its count. df.isna().sum() counts missing values per column; in the adults table the most incomplete are C-reactive protein (387 missing), cholesterol and HDL cholesterol (357 each) and waist circumference (349). Deciding what to do about them is a question about the data rather than about pandas, and Cleaning Data takes it up.

References