Most of this collection teaches one method at a time: a join, a regression, a tree. A real project strings dozens of them together, and most of its decisions are not about methods at all. What exactly is the question? Which data could answer it, and what is wrong with them? What would count as a good enough answer? Who will use the result, and what happens when the world it was built for changes? Descriptions of the data science lifecycle organize these decisions into phases, so that a team can say where a project stands and what remains to be done.
This article describes the most widely used of these descriptions, CRISP-DM, and two others, then walks one project from this site through every phase: the screening of adults for undiagnosed diabetes with a blood test, which the Applied Statistics course works through in its unit on Screening and Diagnostic Tests. The numbers are computed by tools/experiments/data-science/data-science-lifecycle.py, which reuses that unit's data code.
CRISP-DM
The Cross-Industry Standard Process for Data Mining was conceived in late 1996 by three companies with data mining practices, Daimler-Benz, the software company ISL (later part of SPSS) and NCR, who wanted a process model that would be "non-proprietary and freely available"; its 1.0 guide appeared in 2000 (Chapman et al., 2000). It describes "the life cycle of a data mining project" in six phases.
- Business understanding "focuses on understanding the project objectives and requirements from a business perspective, then converting this knowledge into a data mining problem definition and a preliminary plan". The guide asks for the objectives to be stated in the client's terms and for "data mining success criteria" to be set in advance. In this collection, the decision about which of the three kinds of question is being asked, description, prediction or a causal question, belongs here; Why Data Science explains why the choice matters.
- Data understanding "starts with an initial data collection and proceeds with activities in order to get familiar with the data, to identify data quality problems, to discover first insights into the data or to detect interesting subsets to form hypotheses". This is Performing Exploratory Data Analysis, Visualizing Data with One Feature and the provenance questions of Introduction to Data Wrangling.
- Data preparation "covers all activities to construct the final dataset ... from the initial raw data", tasks that "are likely to be performed multiple times and not in any prescribed order". This is the Data Wrangling topic: Cleaning Data, Structuring Data, Enriching Data, and SQL for extraction.
- Modeling: "various modeling techniques are selected and applied and their parameters are calibrated to optimal values"; because techniques have their own requirements on the data, "stepping back to the data preparation phase is often necessary". This is regression, supervised and unsupervised learning, trees, ensembles and neural networks.
- Evaluation looks again at a model that "appears to have high quality from a data analysis perspective" and asks whether "it properly achieves the business objectives". A key objective is "to determine if there is some important business issue that has not been sufficiently considered". The technical side is the Evaluating Model Performance topic, beginning with Training, Validation, and Test Sets; the other side is a return to the original question.
- Deployment may be "as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise". Among its tasks the guide lists a plan for "monitoring and maintenance", which "are important issues if the data mining result becomes part of the day-to-day business and its environment" (Chapman et al., 2000, pp. 13–14, 33).
The guide is explicit that the order is not fixed: "The sequence of the phases is not rigid. Moving back and forth between different phases is always required." And the process does not end with deployment: "The lessons learned during the process and from the deployed solution can trigger new, often more focused business questions."
Other descriptions of the process
CRISP-DM came from data mining in industry. Two later descriptions came from other settings, and comparing them shows what all of them agree on.
| CRISP-DM (2000) | OSEMN (2010) | Team Data Science Process (2016) |
|---|---|---|
| Business understanding | Business understanding | |
| Data understanding | Obtain | Data acquisition and understanding |
| Data preparation | Scrub | Data acquisition and understanding |
| Explore | Data acquisition and understanding | |
| Modeling | Model | Modeling |
| Evaluation | Interpret | Modeling; customer acceptance |
| Deployment | Deployment; customer acceptance |
OSEMN, an acronym for obtain, scrub, explore, model and interpret, was proposed by Hilary Mason and Chris Wiggins in a 2010 blog post, "A Taxonomy of Data Science" (Mason and Wiggins, 2010). It is a list of activities rather than a project plan: it has no phase for framing the question or for deployment.
Microsoft's Team Data Science Process (TDSP) describes "five major stages that are executed iteratively": business understanding, data acquisition and understanding, modeling, deployment and customer acceptance (Microsoft, 2016). It is aimed at projects "that lead to building data products and intelligent applications", and notes that exploratory and one-off analyses may not need every step. Its business understanding stage asks for "sharp" questions and lists five kinds that data science typically answers: "How much or how many? (regression)", "Which category? (classification)", "Which group? (clustering)", "Is this weird? (anomaly detection)" and "Which option should be taken? (recommendation)".
All three put the same activities in the same order, and all three draw them as a cycle. The differences are of emphasis. CRISP-DM and TDSP start with the client's problem and end with a result in use; OSEMN describes the analyst's own work between those points.
Reproducibility
Iteration has a cost: each return to an earlier phase must reproduce everything downstream of it. If cleaning changes, every model must be refitted and every number in the report recomputed. Doing that reliably requires that every step be recorded in a form a computer can rerun. Three practices used throughout this site make it possible.
- Scripts rather than manual steps. Every number in this collection comes from a script in
tools/experiments/, and every figure from a script intools/figures/. Programming with Python and Jupyter shows what goes wrong when the record is a notebook run out of order. - Pinned data. The scripts download public files once and check each against a SHA-256 hash recorded in the code, so that a silent change to a published file stops the analysis instead of changing its results. Data Engineering explains the technique.
- Seeds and versions. Random numbers come from generators created with fixed seeds (The NumPy Package), and Data Science Packages shows the same five lines of pandas giving different answers under two versions of the library, which is why environments are pinned too.
A project walked through the phases
The unit Screening and Diagnostic Tests asks whether HbA1c, the share of hemoglobin carrying glucose, can find adults who have undiagnosed diabetes. It was written as a sequence of statistics articles, but it is also a complete project, and laying it over CRISP-DM shows what each phase involves on real data.
Business understanding
The client in this case would be a screening program. Its question is whether a blood test that needs no fasting can replace or precede one that does. The success criteria have to be set by the client, not the analyst: a program that will confirm every positive result with a second test can accept more false positives than one that will start treatment on a single result, and the costs of the two errors decide the best cut-off, as ROC Curves for Diagnostic Tests shows. The question is also a prediction, not a causal question: nothing in it asks what would happen if anyone's HbA1c were changed.
The phase also fixes the reference standard, what "having diabetes" will mean in the data. The unit uses the American Diabetes Association's fasting criterion, a fasting plasma glucose of 126 mg/dL or more, applied to a single measurement. That is a definition, not an independent fact about each person, and Sensitivity and Specificity shows how the same table reads differently when the other test is taken as the truth.
Data understanding
The data come from the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018: public files from the National Center for Health Statistics for demographics, fasting glucose, HbA1c and a questionnaire on diabetes, with body measurements for the comparison screens of the unit. Understanding them means reading their documentation. Fasting glucose was measured only on a subsample, participants aged 12 and over examined in the morning session, and the fasting subsample's weight variable is zero for the 325 of them with no result or who had not fasted for 8 to 24 hours; the file reader used throughout the site reads those zeros as $5.4 \times 10^{-79}$, a data quality problem described in Cleaning Data that has to be repaired before the weight can be used as a filter.
Data preparation
Preparation turns 9,254 survey participants into the 1,975 adults the analysis needs. Each step has a reason, and counting the rows at each one is the cheapest check that nothing unintended happened.
| Step | Rows kept |
|---|---|
| Participants in the demographics file | 9,254 |
| In the fasting glucose file | 3,036 |
| Fasted 8 to 24 hours (fasting weight above zero) | 2,711 |
| With a fasting glucose result | 2,711 |
| Aged 18 or over | 2,387 |
| Not pregnant | 2,371 |
| With an HbA1c result | 2,368 |
| Answered the diabetes question (yes, no or borderline) | 2,366 |
| Never told they have diabetes | 1,975 |
The last step defines the screening population: people already diagnosed are not the ones a screening program is for. Of the 1,975, 115 (5.8%) have a fasting glucose of 126 mg/dL or more. The step "with a fasting glucose result" removes no one, because the fasting weight is zero exactly for those without a result; a step that removes nothing is worth keeping in the table, because it shows that the two conditions agree.
Modeling
Here the model is a single cut-off: flag everyone whose HbA1c is at or above some value. Choosing the cut-off is the modeling decision. Over all cut-offs, the area under the ROC curve is 0.889, the probability that a randomly chosen adult with a diabetic fasting glucose has a higher HbA1c than a randomly chosen adult without one. The cut-off that maximizes Youden's index, sensitivity plus specificity minus one, is 6.0%, with sensitivity 0.748 and specificity 0.883. The American Diabetes Association's diagnostic cut-off is 6.5%. Both values agree with ROC Curves for Diagnostic Tests, which develops them.
Evaluation
At the cut-off of 6.5%, the 1,975 adults fall into a 2 by 2 table:
| Fasting glucose ≥ 126 | Fasting glucose < 126 | Total | |
|---|---|---|---|
| HbA1c ≥ 6.5% | 54 | 25 | 79 |
| HbA1c < 6.5% | 61 | 1,835 | 1,896 |
| Total | 115 | 1,860 | 1,975 |
Sensitivity is $54/115 = 0.470$, specificity $1{,}835/1{,}860 = 0.987$, the positive predictive value $54/79 = 0.684$ and the negative predictive value $1{,}835/1{,}896 = 0.968$. The test flags 4.0% of the adults.
Evaluation in CRISP-DM's sense asks whether these numbers meet the objective, and here the answer depends on the objective. The test misses more than half of the adults with a diabetic fasting glucose; a positive result is right about two times in three. For a program that wants a cheap first step before confirmatory testing, a lower cut-off such as 6.0% finds more cases at the price of more false alarms. For a program that will act on a single result, the high specificity of 6.5% matters more. The statistics cannot settle that choice; they show its consequences. This is the "important business issue" CRISP-DM's evaluation phase is meant to catch.
Deployment and monitoring
Suppose the screen is deployed. Its sensitivity and specificity were measured on one population, and the people it now sees may differ. Lipton, Wang and Smola (2018) distinguish two simple ways a population can change. Under covariate shift, the distribution of the inputs changes but the probability of the outcome given the inputs does not. Under label shift, the share of people with the condition changes but the inputs given the condition do not: diseases cause symptoms, and the symptoms of a given disease stay the same. Label shift is the natural model for a screening test moved to a clinic where diabetes is more or less common.
If label shift holds, sensitivity and specificity carry over, but the predictive values do not, because they depend on prevalence through Bayes' theorem (Predictive Values and Bayes' Theorem). The share of positive results changes too, and that share can be monitored without knowing anyone's fasting glucose:
| Prevalence in the new population | Share flagged positive | PPV | NPV |
|---|---|---|---|
| 2.0% | 2.26% | 0.416 | 0.989 |
| 5.8% (the survey) | 4.00% | 0.684 | 0.968 |
| 15.0% | 8.19% | 0.860 | 0.913 |
A monitor that watches the positive rate is close kin to the check that Sculley et al. (2015) call prediction bias. In a system working as intended, they write, "the distribution of predicted labels is equal to the distribution of observed labels", and a change in that comparison "can help to detect cases in which the world behavior suddenly changes, making training distributions drawn from historical data no longer reflective of current reality". A screen's positive rate can be compared with the rate seen when it was built, before any confirmatory results arrive. Under label shift the positive rate also gives back the new prevalence, by the Rogan–Gladen correction derived in Predictive Values and Bayes' Theorem.
Real shifts are less tidy. Treating two age groups of the same survey as if each were a new clinic tests the assumption on real data:
| Ages 18 to 39 | Ages 60 and over | |
|---|---|---|
| Adults | 743 | 604 |
| Fasting glucose ≥ 126 | 12 (1.6%) | 62 (10.3%) |
| Flagged positive | 10 (1.35%) | 40 (6.62%) |
| Prevalence estimated from the positive rate (Rogan–Gladen) | 0.0% | 11.6% |
| PPV observed | 0.600 | 0.750 |
| PPV predicted from overall sensitivity and specificity | 0.364 | 0.800 |
| Sensitivity in the group | 0.500 | 0.484 |
| Specificity in the group | 0.9945 | 0.9815 |
The monitor works in the direction it should: the positive rate is a third of the survey's in the younger group and two-thirds higher in the older one. The corrections built on label shift work less well. In the older group the corrected prevalence, 11.6%, is close to the true 10.3%. In the younger group the positive rate, 1.35%, is almost exactly the overall false positive rate, so the correction returns a prevalence of essentially zero, and the predicted PPV of 0.364 is far from the observed 6 true positives among 10. The reason is visible in the last row: the test's specificity is higher in young adults than overall, so the label shift assumption, that the test behaves the same given the condition, does not hold. Twelve cases are also too few for any estimate to be precise. Predictive Values and Bayes' Theorem analyzes these age groups in more detail.
The lesson for deployment is the one CRISP-DM puts in its monitoring plan. A cheap signal, such as the positive rate, can show that something has changed. Deciding what has changed, and whether the model still meets its objective, requires new labeled data: in this case, confirmatory fasting glucose results for a sample of the people screened. Monitoring sends the project back to evaluation, and sometimes to the question.
Communication
Every phase ends in communication with someone: the client in business understanding, the data's owners in data understanding, the eventual users in deployment. CRISP-DM's deployment phase makes the point that "even if the purpose of the model is to increase knowledge of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it", and that in many cases "it is the customer, not the data analyst, who carries out the deployment steps". The screening example shows how much the form matters: the same test is described by a sensitivity of 0.470, by "misses more than half", and by a positive predictive value that depends on where it is used, and each description suits a different reader. Likelihood Ratios for Diagnostic Tests gives a fourth, designed for clinicians.
References
- Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., and Wirth, R. (2000). CRISP-DM 1.0: Step-by-step data mining guide. SPSS. record · 2 pages
- Lipton, Z. C., Wang, Y.-X., and Smola, A. J. (2018). Detecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Machine Learning (PMLR 80). arXiv:1802.03916. record
- Mason, H., and Wiggins, C. (2010). A taxonomy of data science [Blog post]. dataists. record
- Microsoft (2016). Team Data Science Process lifecycle. Azure/Microsoft-TDSP repository,
Docs/lifecycle-detail.md. https://github.com/Azure/Microsoft-TDSP record - National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: DEMO_J, GLU_J, GHB_J, DIQ_J and BMX_J data files. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28 (NeurIPS 2015). record · 2 pages