Tools for Visualizing Data

The software used to make charts and the ideas that separate it. Drawing commands against declarative specifications; the grammar of graphics of Wilkinson and Wickham, in which a pie chart is a stacked bar in polar coordinates; matplotlib's figures, axes and output formats, where one scatter plot of 5,247 points is 763 KB as vector SVG and 107 KB with its points rasterized; pandas plotting, seaborn and its semantic mappings, plotly for interactive charts, ggplot2 and Vega-Lite; spreadsheets and business intelligence tools, including the gene names that spreadsheets turn into dates; and how to keep figures reproducible.

A scatter plot of systolic blood pressure against age, colored by sex, for 5,247 NHANES adults, takes eight lines of matplotlib, one line of seaborn or plotly, and a few clicks in a spreadsheet. The results look similar, but the tools differ in what they think a chart is. In matplotlib the chart is the result of a sequence of drawing commands: make a figure, draw points here in this color, label this axis. In seaborn, plotly or R's ggplot2 the chart is a description: map age to the horizontal position, pressure to the vertical, sex to color, and let the software work out the scales, colors and legend. In a spreadsheet or a dashboard tool the chart is an object built by hand on top of a table and kept with it. Those differences decide how quickly a chart can be changed, how it can be reproduced, who can make it, and where it can be shown.

This article describes the main tools a data scientist meets, organized by the idea behind them rather than by feature lists, which change with every release. Version numbers are left out on purpose, except where a number in the article depends on one. The Python packages appear again in The matplotlib Package and The pandas Package in the Python for Data Science topic; this article is about choosing among them and understanding what they share.

Two ways to describe a chart

An imperative (or procedural) plotting library gives the user drawing operations: create axes, draw a line through these points, put a rectangle there, write this text. Everything not drawn explicitly is absent, and every mapping from data to appearance, which group gets which color, where the legend goes, is the user's code. This gives complete control and makes anything drawable, at the cost of code that grows with each feature of the chart. Waskom (2021) gives a typical case: in matplotlib, a scatter plot whose marker size shows one numerical variable and whose marker shape shows a categorical one requires converting the values to point sizes and looping over the categories, calling the plotting function once per marker type.

A declarative library takes a specification of what to show and derives the drawing. The specification names the data, says which variables map to which visual properties, and chooses a kind of mark; the library supplies scales, axes, legends and colors, usually with sensible defaults. Changing a chart means changing the specification, often one word. Most declarative tools rest on the same theory, the grammar of graphics.

The grammar of graphics

Wilkinson's The Grammar of Graphics (1999; 2nd ed., Wilkinson, 2005) set out to describe statistical graphics by their components rather than by their names. Instead of a fixed menu of "scatter plot", "histogram" and "pie chart", a graphic is specified by independent parts that can be combined freely, so that named charts become particular combinations and new charts fall out of new combinations. Wickham (2010) reworked the grammar around layers and implemented it as the R package ggplot2. In his formulation a plot consists of:

  • a default dataset and set of aesthetic mappings from variables to visual properties (x position, y position, color, size, shape);
  • one or more layers, each with a geometric object (point, line, bar, area), a statistical transformation of the data (identity, binning, counting, smoothing), a position adjustment (stacking, dodging side by side, jittering) and optionally its own data and mappings;
  • one scale for each aesthetic, which converts data values to visual values and draws the axis or legend that converts them back;
  • a coordinate system, usually Cartesian;
  • a facet specification, which splits the data into panels (small multiples).

The components are independent, so one can be changed while the others stay fixed (Wickham, 2010, §3). A histogram, for example, is not a separate kind of chart but a bar geometry applied to the output of a binning transformation, which creates a new variable, the count per bin, for the bars' heights. A scatter plot with a fitted curve is two layers on the same data and mappings: points with the identity transformation, and a line with a smoothing transformation. Wickham notes that a statistical transformation must be location-scale invariant, commuting with shifts and rescalings of the data: $f(x + a) = f(x) + a$ and $f(b x) = b f(x)$, since moving or stretching a graphic should not change what it shows.

Coordinates: a pie is a stacked bar

The coordinate system is the component that best shows the grammar's power. Wickham (2010, §6.2) observes that a pie chart is a single stacked bar drawn in polar coordinates, with the bar's height mapped to angle; mapping the height to radius instead gives a "bullseye" chart; and a set of side-by-side bars in polar coordinates is a coxcomb chart, the form Florence Nightingale used for deaths in the Crimean War.

One layer, four coordinate systems. The blood pressure categories of 1,578 NHANES adults aged 40 to 59 (normal 36.0%, elevated 14.5%, stage 1 25.9%, stage 2 23.6%) as a single stacked bar; the same bar in polar coordinates with height mapped to angle, which is a pie chart; with height mapped to radius, a bullseye; and side-by-side bars in polar coordinates, a coxcomb. Only the coordinate system differs between the first three.
One layer, four coordinate systems. The blood pressure categories of 1,578 NHANES adults aged 40 to 59 (normal 36.0%, elevated 14.5%, stage 1 25.9%, stage 2 23.6%) as a single stacked bar; the same bar in polar coordinates with height mapped to angle, which is a pie chart; with height mapped to radius, a bullseye; and side-by-side bars in polar coordinates, a coxcomb. Only the coordinate system differs between the first three.

The figure draws the shares of the four blood pressure categories among 1,578 NHANES adults aged 40 to 59 in all four ways. Seeing the pie as a transformed bar makes its weakness concrete: the transformation turns lengths along a common baseline into angles around a center, which readers judge less accurately (Best Practices for Visualizing Data). It also shows why coxcombs distort: the bar's value becomes a radius, but the eye reads the wedge's area, which grows with the square of the radius. In code, the change from the first panel to the second is one component, coord_polar(theta = "y") in ggplot2.

Where the grammar lives now

The grammar's influence extends well beyond R. ggplot2 is its reference implementation; plotnine ports ggplot2's interface to Python. Vega-Lite (Satyanarayan, Moritz, Wongsuphasawat and Heer, 2017) expresses a grammar of graphics, with interaction added, as JSON specifications that a web browser renders, and Altair writes those specifications from Python. seaborn's newer objects interface is organized around marks, statistical transformations and moves (position adjustments). Tableau grew out of the Polaris research system (Stolte, Tang and Hanrahan, 2002), whose drag-and-drop interface built visual specifications from the same kind of algebra over table columns. Learning the grammar once makes all of these easier to learn.

matplotlib

matplotlib (Hunter, 2007) is the foundation of most plotting in Python: pandas plotting and seaborn draw with it, and its output formats and fine control are used even when the chart is specified elsewhere. Its object model has three levels. A Figure is the whole image; it contains one or more Axes, each a plotting region with its own coordinate system, axes and ticks; and everything visible, every line, patch, text and tick, is an Artist belonging to an Axes or the Figure. There are two ways to drive it. The pyplot interface keeps a "current" figure and axes and lets calls such as plt.plot and plt.xlabel act on them, which is quick for interactive work. The object-oriented interface creates the objects explicitly (fig, ax = plt.subplots()) and calls methods on them (ax.plot, ax.set_xlabel), which is clearer for anything with more than one panel and is the style used for reusable code.

matplotlib draws through backends: interactive ones that open a window or render inside a notebook, and file backends for raster formats (PNG) and vector formats (SVG, PDF). The choice of format has consequences. A vector file stores every mark as a geometric object, which keeps text and lines sharp at any size, but grows with the number of marks; a raster file stores pixels, and its size depends on the image's dimensions, not on the data. Saved from the same matplotlib figure, a scatter plot of the 5,247 NHANES adults is 763 KB as SVG, 121 KB as a PNG at 150 dots per inch, and 107 KB as an SVG in which only the points are rasterized while the axes and text stay vector. The figures on this site use the last approach for any layer with more than 500 marks.

pandas plotting

pandas (McKinney, 2010) attaches plotting methods to its tables. DataFrame.plot and its variants (.plot.scatter, .plot.hist, .plot.box, .plot.hexbin), along with DataFrame.hist, DataFrame.boxplot and pandas.plotting.scatter_matrix, draw with matplotlib by default and return matplotlib objects, so they can be adjusted afterwards. Their value is speed during exploration: a histogram of every numerical column is one call, adults.hist(), and the table's column names become axis labels. The plotting backend is configurable, and other libraries, plotly among them, can register themselves so that the same .plot calls produce their charts instead. For anything beyond a quick look, most users move to seaborn or to matplotlib directly.

seaborn

seaborn (Waskom, 2021) is a declarative interface to matplotlib for statistical graphics. Its functions take a dataset and the names of variables, and they assign roles: x, y, and the semantic mappings hue, size and style, which seaborn turns into colors, marker sizes and marker or line styles, with a legend, choosing a categorical or a continuous mapping according to the type of the variable. Many functions also compute a statistic before drawing, such as a mean with a bootstrap confidence interval, a kernel density estimate or a regression line.

Two kinds of function divide the work. Axes-level functions (scatterplot, histplot, kdeplot, boxplot, violinplot) draw into one matplotlib Axes, which can be passed in, so they combine with other matplotlib code. Figure-level functions (relplot, displot, catplot, and the matrix and grid functions pairplot and clustermap) create a whole figure and can facet the data into panels by the row and col variables; each corresponds to several axes-level functions, selected by a kind argument (Waskom, 2021). seaborn works most naturally with "long-form" or tidy data, one row per observation and one column per variable (Structuring Data), in which any column can be assigned to any role without reshaping.

Its defaults carry statistical choices that should be known, not just accepted. kdeplot and violinplot smooth with a bandwidth from a rule of thumb and draw past the data by default; grouped density curves are scaled by group size by default (Visualizing Data with One Feature discusses both). Error bars on bar and point plots are intervals computed by bootstrap resampling, which need to be named in any caption that shows them.

plotly

plotly produces interactive charts: the output is a web page in which the chart is drawn by the JavaScript library plotly.js, and the reader can hover over a mark to see its values, zoom into a region, pan, and show or hide groups by clicking the legend. The plotly.express module offers one-call functions similar to seaborn's (px.scatter, px.histogram, px.box), with color, size, symbol, facet_row and facet_col arguments and a hover_data argument that adds columns to the tooltip; the lower-level graph_objects module builds charts trace by trace.

Interactivity is most useful in exploration: hovering over an unusual point to read its identifier is faster than finding it in a table, and zooming lets a single chart show both the bulk of the data and its tail. The costs follow from the format. The chart is an HTML document that needs a browser, and a self-contained file embeds the plotly.js library, which makes even a small chart several megabytes unless the library is loaded from the web instead. Static images for print require a separate export step. Very large numbers of points slow a browser down, which brings back the binning and sampling of Visualizing Data with Multiple Features. And a reader of a printed report cannot hover, so anything the reader must know has to be visible without interaction.

R and ggplot2

In R, ggplot2 is the standard plotting system, and its code reads as the grammar described above:

library(ggplot2)
ggplot(adults, aes(x = age, y = sbp, colour = sex)) +
  geom_point(alpha = 0.3) +
  geom_smooth() +
  facet_wrap(~ smoking)

Each + adds a component: a layer of points, a layer with a smoothing transformation, a facet specification. Changing geom_point to geom_hex or adding coord_polar() changes one component and leaves the rest. Wickham (2010, §5) points out a design decision that also applies to the Python libraries: by relying on the host language for data manipulation, ggplot2 can leave out the parts of Wilkinson's grammar that transform and reshape data, so that the data are prepared, and checked, before they reach the plot.

Spreadsheets and business intelligence tools

Spreadsheets such as Excel, Google Sheets and LibreOffice Calc are the most widely used data tools, and charting in them is immediate: select a range, insert a chart. For a quick look at a small table, or for a reader who will open nothing else, they are hard to beat. Their weaknesses are the ones that matter for analysis. A chart built by clicking leaves no record of how it was made, so it cannot be rerun on updated data or checked by someone else. Cells can be edited without a trace. And spreadsheets interpret what is typed or imported, sometimes silently. Ziemann, Eren and El-Osta (2016) screened supplementary files of genomics papers and found that gene symbols such as SEPT2 (Septin 2) and MARCH1 had been converted to dates ("2-Sep", "1-Mar") in files from 704 published articles; among the journals they examined, 19.6% of papers with Excel gene lists in their supplementary data were affected.

Business intelligence (BI) applications such as Tableau, Power BI and Looker sit between spreadsheets and code. They connect to databases and files, let analysts build charts by dragging variables onto roles (rows, columns, color, size), and assemble charts into dashboards that refresh as the data change. They are the usual choice for recurring reports to an organization, where the same views are consulted repeatedly by people who do not write code. Their limits are analytical depth, since statistical modeling happens elsewhere, the fixed set of chart types each product offers, and dependence on a commercial product whose files and skills do not transfer between vendors.

Reproducible figures

A figure in a report is a claim about data, and like a number it should be possible to regenerate and check. Code-based tools make this straightforward if a few habits are kept: the script that makes the figure reads the data from a known source, does all its transformations in code, sets every random seed (for jitter, sampling or bootstrap intervals), and writes the file. On this site, for example, every figure is drawn by a script in tools/figures/ from numbers written by a script in tools/experiments/, with a fixed style module that sets fonts, colors and a fixed identifier seed for SVG output so that rerunning a script reproduces its file byte for byte. A chart made by hand in an image editor or a spreadsheet cannot be checked this way.

Choosing a tool

Need Usually suited
A quick look during exploration pandas .plot, seaborn
Statistical graphics with groups, panels and summaries seaborn, ggplot2
Complete control of a publication figure matplotlib (often after seaborn)
Interactive exploration or a web page plotly, Vega-Lite and Altair
Recurring dashboards for an organization BI tools (Tableau, Power BI, Looker)
A one-off chart for a reader who uses only a spreadsheet a spreadsheet, with the data checked first

The choice is rarely exclusive. A common pattern is to explore with seaborn or plotly, then produce the final figure with matplotlib calls on the axes seaborn created, and to hand recurring views to a dashboard once they have settled.

Corrections to the course notes

The unit's notes describe three-dimensional plots as difficult or impossible in matplotlib and ggplot2; matplotlib includes a three-dimensional toolkit (mpl_toolkits.mplot3d, used through projection="3d"), although its plots are static and plotly's interactive rotation is often easier to read. The notes also say that the spreadsheet gene-name errors required published corrections; the study they refer to (Ziemann et al., 2016) documented the errors in the supplementary files of 704 published articles, and the scale it reports is the figure to cite.

In Python

The same chart in three Python tools, each tested on the NHANES table described in Performing Exploratory Data Analysis (adults, with sex_label holding "men" or "women"):

# matplotlib: drawing commands
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6, 4))
for sex, colour in [("men", "tab:blue"), ("women", "tab:orange")]:
    d = adults[adults.sex_label == sex]
    ax.scatter(d.age, d.sbp, s=4, alpha=0.3, color=colour, label=sex)
ax.set_xlabel("Age (years)")
ax.set_ylabel("Systolic pressure (mm Hg)")
ax.legend()
fig.savefig("sbp-age.svg")

# seaborn: a specification, faceted by smoking status
import seaborn as sns
g = sns.relplot(data=adults, x="age", y="sbp", hue="sex_label", col="smoking", s=8, alpha=0.3)

# seaborn objects interface: layers, with a fitted quadratic per group
import seaborn.objects as so
(so.Plot(adults, x="age", y="sbp", color="sex_label")
   .add(so.Dots(alpha=0.3))
   .add(so.Line(), so.PolyFit(order=2))
   .facet(col="smoking"))

# plotly: interactive, with the respondent number in the tooltip
import plotly.express as px
fig = px.scatter(adults, x="age", y="sbp", color="sex_label", facet_col="smoking",
                 opacity=0.3, hover_data=["seqn"])
fig.write_html("sbp-age.html", include_plotlyjs="cdn")   # load plotly.js from the web

The file sizes quoted above are computed by tools/experiments/data-science/data-visualization-tools.py in the site's repository, which also prints the matplotlib version they were measured with, and the coordinate-system figure is drawn by tools/figures/data-visualization-tools.py.

References

  • Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3), 90–95. record · 2 pages
  • McKinney, W. (2010). Data structures for statistical computing in Python. In Proceedings of the 9th Python in Science Conference (pp. 56–61). record · 2 pages
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographics (DEMO_J) and Blood Pressure (BPX_J) data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • Satyanarayan, A., Moritz, D., Wongsuphasawat, K., and Heer, J. (2017). Vega-Lite: A grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics, 23(1), 341–350. record
  • Stolte, C., Tang, D., and Hanrahan, P. (2002). Polaris: A system for query, analysis, and visualization of multidimensional relational databases. IEEE Transactions on Visualization and Computer Graphics, 8(1), 52–65. record
  • Waskom, M. L. (2021). seaborn: Statistical data visualization. Journal of Open Source Software, 6(60), 3021. https://doi.org/10.21105/joss.03021 record
  • Wickham, H. (2010). A layered grammar of graphics. Journal of Computational and Graphical Statistics, 19(1), 3–28. record · 2 pages
  • Wilkinson, L. (2005). The Grammar of Graphics (2nd ed.). Springer. record
  • Ziemann, M., Eren, Y., and El-Osta, A. (2016). Gene name errors are widespread in the scientific literature. Genome Biology, 17, 177. https://doi.org/10.1186/s13059-016-1044-7 record