Factor Analysis

Explaining the correlations among many measurements by a few unobserved factors. Spearman's one-factor model solved by hand from three correlations, the general model and the covariance structure it implies, communality and uniqueness, maximum likelihood fits with one to four factors, why factors can be rotated and what varimax does, factor scores, and how factor analysis differs from principal component analysis, on nine mental ability tests taken by 301 schoolchildren in 1939.

In 1939 Karl Holzinger and Frances Swineford published the scores of 301 seventh- and eighth-grade children, from two schools near Chicago, on a battery of mental ability tests (Holzinger and Swineford, 1939). Nine of those tests have become a standard example: three spatial tests (visual perception, cubes and lozenges), three verbal tests (paragraph comprehension, sentence completion and word meaning) and three speeded tests (addition, counting dots, and telling straight from curved capital letters). Their correlations, shown below, fall into a pattern. The three verbal tests correlate between 0.70 and 0.73 with each other, but at most 0.23 with any of the speeded tests. The speeded tests correlate 0.34 to 0.49 with each other, and the spatial tests 0.30 to 0.44.

Correlations among the nine tests for the 301 children, shaded by size. The lines mark off the spatial, verbal and speeded tests. The verbal block is the strongest; correlations between blocks are mostly below 0.25, with exceptions involving visual perception and the capitals test.
Correlations among the nine tests for the 301 children, shaded by size. The lines mark off the spatial, verbal and speeded tests. The verbal block is the strongest; correlations between blocks are mostly below 0.25, with exceptions involving visual perception and the capitals test.

A natural explanation is that the tests draw on a few underlying abilities, one verbal, one of speed and one spatial, and that two tests are correlated to the extent that they draw on the same ones. None of these abilities is measured directly. Factor analysis turns that explanation into a statistical model: each measured variable is a weighted sum of a small number of unobserved factors, shared by all the variables, plus a part unique to itself. The factors account for the correlations; the unique parts account for everything else. Fitted to these data, a three-factor model reproduces all 36 correlations to within 0.042, and the factors it finds are the verbal, speed and spatial groupings above.

Factor analysis is a latent variable model, one of the families of unsupervised learning, and its closest relative is principal component analysis (PCA). The two are often confused, but they answer different questions, and the last part of this article compares them on the same data.

Spearman's single factor

The method began with one factor. Charles Spearman (1904) studied the correlations among schoolchildren's standing in school subjects such as Latin, Greek and mathematics, and their scores on sensory discrimination tasks such as judging pitch, and concluded that "all branches of intellectual activity have in common one fundamental function (or group of functions), whereas the remaining or specific elements of the activity seem in every case to be wholly different from that in all the others". He called the common element general intelligence. The statistical claim inside that conclusion, one common factor plus specific parts unrelated to each other, is the one-factor model, and it can be solved by hand.

The model

Standardize each test score to mean 0 and variance 1, and call the standardized score of test $j$ $x_j$. The one-factor model says

$$x_j = \lambda_j f + e_j,$$

where $f$ is the child's unobserved factor score, the same for every test, $\lambda_j$ is the loading of test $j$ on the factor, and $e_j$ is the part of the score specific to test $j$. The assumptions are what give the model content:

  • $f$ has mean 0 and variance 1, which fixes its scale;
  • each $e_j$ has mean 0 and variance $\psi_j$;
  • the specific parts are uncorrelated with $f$ and with each other.

Two consequences follow from the rules for variances and covariances of sums. The variance of a test is

$$\operatorname{var}(x_j) = \lambda_j^2 \operatorname{var}(f) + \operatorname{var}(e_j) = \lambda_j^2 + \psi_j = 1,$$

and the covariance of two different tests, which is their correlation because both are standardized, is

$$\operatorname{corr}(x_j, x_k) = \operatorname{cov}(\lambda_j f + e_j,\ \lambda_k f + e_k) = \lambda_j \lambda_k,$$

since every term involving a specific part vanishes. All the correlations among the tests are explained by the factor: two tests correlate because both depend on $f$, and only for that reason. The share of a test's variance that comes from the factor, $\lambda_j^2$, is its communality, and the remainder, $\psi_j = 1 - \lambda_j^2$, is its uniqueness. The loading is also the correlation between the test and the factor, since $\operatorname{cov}(x_j, f) = \lambda_j$ and both have variance 1.

Three tests by hand

Take the three verbal tests. Their correlations are

$$r_{12} = 0.7332 \ (\text{paragraph, sentence}), \quad r_{13} = 0.7045 \ (\text{paragraph, word meaning}), \quad r_{23} = 0.7200 \ (\text{sentence, word meaning}).$$

The model says $r_{12} = \lambda_1 \lambda_2$, $r_{13} = \lambda_1 \lambda_3$ and $r_{23} = \lambda_2 \lambda_3$. Multiplying the first two and dividing by the third eliminates $\lambda_2$ and $\lambda_3$:

$$\lambda_1^2 = \frac{r_{12}\, r_{13}}{r_{23}} = \frac{0.7332 \times 0.7045}{0.7200} = 0.7174, \qquad \lambda_1 = 0.8470.$$

In the same way $\lambda_2^2 = r_{12} r_{23} / r_{13} = 0.7493$ and $\lambda_3^2 = r_{13} r_{23} / r_{12} = 0.6918$. So:

Test Loading $\lambda_j$ Communality $\lambda_j^2$ Uniqueness $\psi_j$
paragraph comprehension 0.8470 0.7174 0.2826
sentence completion 0.8656 0.7493 0.2507
word meaning 0.8317 0.6918 0.3082

About 70% of the variation in each verbal test is shared with the other two through the common factor, and about 30% is specific to the test, including its measurement error. A maximum likelihood fit, described below, returns the same loadings and uniquenesses to four decimals. With three tests there are three correlations and three loadings, so the model fits them exactly and cannot be tested. With four or more tests the one-factor model makes predictions that can fail: for example, $r_{12} r_{34}$ must equal $r_{13} r_{24}$, since both are $\lambda_1 \lambda_2 \lambda_3 \lambda_4$. On all nine tests it does fail, as the next sections show. One factor cannot produce the block pattern in the figure.

The general model

Several factors

With $m$ factors, each standardized variable is

$$x_j = \lambda_{j1} f_1 + \lambda_{j2} f_2 + \dots + \lambda_{jm} f_m + e_j,$$

or, for all $p$ variables at once, $x = \Lambda f + e$, where $\Lambda$ is the $p \times m$ loading matrix, $f$ the vector of $m$ factors and $e$ the vector of specific parts (Hastie, Tibshirani and Friedman, 2009, §14.7.1, eq. 14.80). The assumptions extend those above: the factors have mean 0, variance 1 and are uncorrelated with each other, so $\operatorname{Cov}(f) = I$; the specific parts are uncorrelated with the factors and with each other, so $\operatorname{Cov}(e) = \Psi$, a diagonal matrix of uniquenesses. The covariance matrix of $x$ then follows in one line,

$$\Sigma = \operatorname{Cov}(\Lambda f + e) = \Lambda \operatorname{Cov}(f) \Lambda^{\mathsf{T}} + \operatorname{Cov}(e) = \Lambda \Lambda^{\mathsf{T}} + \Psi$$

(Hastie et al., 2009, eq. 14.81). The off-diagonal entries, the correlations, come entirely from $\Lambda \Lambda^{\mathsf{T}}$: $\operatorname{corr}(x_j, x_k) = \sum_{l} \lambda_{jl} \lambda_{kl}$. The diagonal splits each variable's variance into its communality, $h_j^2 = \sum_l \lambda_{jl}^2$, and its uniqueness $\psi_j$, with $h_j^2 + \psi_j = 1$ for standardized variables.

The model is economical. A correlation matrix of $p$ variables has $p(p-1)/2$ correlations, 36 for nine tests. A model with $m$ factors explains them with $pm$ loadings, of which $m(m-1)/2$ can be fixed freely for a reason explained under rotation, and $p$ uniquenesses. Counting all $p(p+1)/2 = 45$ distinct entries of the correlation matrix against the free parameters, a three-factor model for nine tests has $27 + 9 - 3 = 33$ parameters and $45 - 33 = 12$ degrees of freedom left over: 12 independent constraints the data could violate. A one-factor model has 18 parameters and 27 constraints.

Fitting by maximum likelihood

If the factors and specific parts are normally distributed, the variables are multivariate normal with covariance $\Sigma = \Lambda\Lambda^{\mathsf{T}} + \Psi$, and the model can be fitted by maximum likelihood (Hastie et al., 2009, §14.7.1). Maximizing the likelihood is equivalent to minimizing the discrepancy

$$F(\Lambda, \Psi) = \log \lvert \Sigma \rvert + \operatorname{tr}\!\left(R\, \Sigma^{-1}\right) - \log \lvert R \rvert - p$$

between the model's $\Sigma$ and the sample correlation matrix $R$, which is zero when $\Sigma = R$ and positive otherwise. Only $R$ enters, so the fit depends on the data only through their correlations. There is no closed form; software iterates. Older methods, such as principal axis factoring, fit $\Lambda\Lambda^{\mathsf{T}}$ to $R$ with estimated communalities on its diagonal by least squares, and usually give similar loadings.

Fitted with one to four factors, the nine tests give:

Factors $m$ Discrepancy $F$ Degrees of freedom Largest residual correlation RMS residual correlation
1 1.036 27 0.450 0.160
2 0.433 19 0.216 0.085
3 0.076 12 0.042 0.019
4 0.017 6 0.023 0.010

A residual correlation is the difference between an observed correlation and the one the model implies, $r_{jk} - \sum_l \lambda_{jl}\lambda_{kl}$; the root mean square (RMS) is taken over the 36 pairs. With two factors, several residuals remain above 0.2: addition and counting dots are 0.216 more correlated than the model allows, cubes and lozenges 0.203. These are the speed and spatial blocks, which two factors cannot both represent. With three factors no residual exceeds 0.042, and a fourth factor buys little. The eigenvalues of $R$ point the same way: 3.217, 1.637 and 1.366, then a drop to 0.699 and below, so three exceed 1, the threshold of the Kaiser criterion discussed in Principal Component Analysis. Under normality, the likelihood also supports a formal test of whether $m$ factors suffice, but with a large sample such a test can reject a model whose remaining discrepancies are too small to matter, and the size of the residuals is often the more useful guide.

Rotation

Why the loadings are not unique

The three-factor solution as the fitting algorithm returns it is in the first columns of the table below. Every test loads on the first factor, between 0.23 and 0.84, and the other two factors mix positive and negative loadings in no obvious pattern. Nothing is wrong. The fit determines $\Lambda\Lambda^{\mathsf{T}}$, not $\Lambda$ itself. If $T$ is any orthogonal $m \times m$ matrix, so that $T T^{\mathsf{T}} = I$, then

$$(\Lambda T)(\Lambda T)^{\mathsf{T}} = \Lambda T T^{\mathsf{T}} \Lambda^{\mathsf{T}} = \Lambda \Lambda^{\mathsf{T}},$$

so the loadings $\Lambda T$ fit the data exactly as well as $\Lambda$, and correspond to factors $T^{\mathsf{T}} f$ that are still uncorrelated with variance 1 (Hastie et al., 2009, eq. 14.79). Geometrically, the factors are a set of perpendicular axes that can be turned freely in the space they span. This freedom is why $m(m-1)/2$ of the loadings could be fixed when counting parameters: an orthogonal $m \times m$ matrix has that many free angles.

The freedom is used to make the loadings easier to read. The ideal, known as simple structure, is that each variable loads strongly on one factor and near zero on the others, so that each factor is identified by a distinct group of variables.

Varimax

Varimax (Kaiser, 1958) chooses the rotation that maximizes the variance of the squared loadings within each factor, summed over the factors:

$$V(\Lambda T) = \sum_{l=1}^{m} \left[ \frac{1}{p} \sum_{j=1}^{p} \lambda_{jl}^4 - \left( \frac{1}{p} \sum_{j=1}^{p} \lambda_{jl}^2 \right)^{2} \right], \quad \text{where the } \lambda_{jl} \text{ are the entries of } \Lambda T.$$

The variance of the squared loadings in a column is large when they are a mix of values near zero and large values, and small when they are all middling, so maximizing it pushes each factor toward a few large loadings and many small ones. Kaiser's "normal" varimax first divides each row of $\Lambda$ by $h_j$, so that variables with low communality count equally, and multiplies back afterwards; the raw form is used here, and on these data the two agree to within 0.054 in every loading.

Test Unrotated $F_1$ $F_2$ $F_3$ Varimax: verbal speed spatial Communality Uniqueness
visual perception 0.488 0.313 0.389 0.320 0.130 0.607 0.487 0.513
cubes 0.244 0.173 0.402 0.135 −0.041 0.481 0.251 0.749
lozenges 0.272 0.407 0.466 0.080 0.113 0.662 0.457 0.543
paragraph 0.835 −0.152 −0.032 0.838 0.078 0.113 0.721 0.279
sentence 0.839 −0.209 −0.097 0.867 0.071 0.032 0.757 0.243
word meaning 0.823 −0.129 0.016 0.815 0.066 0.162 0.695 0.305
addition 0.231 0.484 −0.460 0.104 0.696 −0.063 0.499 0.501
counting dots 0.270 0.621 −0.267 0.077 0.703 0.175 0.530 0.470
capitals 0.376 0.561 0.023 0.169 0.511 0.408 0.456 0.544
Absolute loadings of the nine tests on three factors: as fitted (left), after varimax rotation (middle), and, for comparison, the correlations of the tests with the first three principal components after the same rotation (right). Rotation turns the unrotated pattern, in which every test loads on the first factor, into three groups. PCA finds the same groups with larger numbers, because it counts each test's unique variance as explained.
Absolute loadings of the nine tests on three factors: as fitted (left), after varimax rotation (middle), and, for comparison, the correlations of the tests with the first three principal components after the same rotation (right). Rotation turns the unrotated pattern, in which every test loads on the first factor, into three groups. PCA finds the same groups with larger numbers, because it counts each test's unique variance as explained.

After rotation the structure in the correlation matrix is plain. The verbal tests load between 0.815 and 0.867 on the first factor and below 0.17 elsewhere; addition and counting dots load 0.70 on the second; the spatial tests load 0.48 to 0.66 on the third. The capitals test, which asks children to sort letters quickly by shape, loads on both speed (0.511) and spatial (0.408), and visual perception has a modest verbal loading as well (0.320). The varimax criterion rose from 0.104 to 0.169.

The communalities and uniquenesses did not change, and neither did any implied correlation: rotation moves the axes, not the fit. The communality of paragraph comprehension is $0.838^2 + 0.078^2 + 0.113^2 = 0.721$ after rotation, the same as $0.835^2 + 0.152^2 + 0.032^2$ before. The uniquenesses are informative in their own right. Cubes has a uniqueness of 0.749: three quarters of its variation is not shared with the other tests, either because the test measures something specific or because it measures it unreliably.

The names are an interpretation

The factors come out of the fit as unnamed columns; "verbal", "speed" and "spatial" are names given afterwards because of which tests load on each, and they are only as good as the tests behind them. Rotation adds a second layer of choice, since a different criterion would give different loadings that fit equally well. Hastie et al. (2009, §14.7.1) note that this freedom to search for interpretable rotations leaves "a certain subjectivity in the use of factor analysis" and has left many analysts skeptical of the method. The safeguard is that the structure here was visible in the raw correlations before any model was fitted, and was predicted by the design of the tests.

Varimax keeps the factors uncorrelated. Oblique rotations, such as promax, allow correlated factors, which is often more plausible for abilities: children who do well on verbal tests also tend to do well on spatial ones, as the positive correlations between the blocks show. A further step is confirmatory factor analysis, in which the analyst states in advance which loadings are zero and tests that restricted model, instead of letting every test load on every factor. The nine-test subset used here is, as the documentation of the lavaan software records, the one Jöreskog (1969) analyzed with the 145 children of the Grant-White school in "A general approach to confirmatory maximum likelihood factor analysis", and it remains the standard example for that method in lavaan (Rosseel, 2012; lavaan documentation, n.d.).

Factor scores

The model describes the variables in terms of the factors, but a child's factor scores are never observed, and unlike principal component scores they cannot be computed exactly, because each test score mixes the factors with an unknown specific part. They can only be estimated. scikit-learn's transform returns the expected factor values given the test scores under the fitted normal model. Two consequences show in the estimates for these children. Their standard deviations are 0.937, 0.835 and 0.812 for the verbal, speed and spatial factors, not 1, because an estimate shrinks toward zero when the tests measure a factor imperfectly; the verbal factor, whose tests have the highest communalities, shrinks least. And although the factors are uncorrelated in the model, the estimated scores correlate slightly, between 0.034 and 0.101. Factor scores are useful summaries, but they carry measurement error, and analyses that use them as if they were the factors inherit it.

Factor analysis and principal component analysis

PCA and factor analysis both summarize many correlated variables by a few new ones, and both produce a table of loadings. The difference is in what they try to explain.

PCA is a transformation of the data. Its components are weighted sums of the observed variables, chosen to capture as much of their total variance as possible, and it has no notion of variance that is specific to one variable. Factor analysis is a model of the data. It tries to explain only the shared variance, the correlations, and sets each variable's specific variance aside in $\Psi$. Hastie et al. (2009, §14.7.1) put the distinction this way: because of the separate disturbances, factor analysis models the correlation structure of the variables rather than their covariance structure. A practical consequence is that rescaling a variable changes the principal components of the covariance matrix but not the factor model, whose loadings simply rescale.

The difference shows on the same correlation matrix. The correlations between the tests and the first three principal components, rotated by varimax for comparability, give the same three groups (the right panel of the figure), but with larger numbers, and the implied communalities are larger too:

Test Communality, 3 principal components Communality, 3 factors
visual perception 0.587 0.487
cubes 0.545 0.251
lozenges 0.630 0.457
paragraph 0.815 0.721
sentence 0.825 0.757
word meaning 0.793 0.695
addition 0.722 0.499
counting dots 0.692 0.530
capitals 0.611 0.456
mean 0.691 0.539

PCA credits each component with part of every test's own variance. Cubes, which shares little with the other tests, has a PCA communality more than twice its factor-analysis communality. In return, the components reproduce the correlations between tests less well: three components leave residual correlations as large as 0.212 (RMS 0.083), against 0.042 (RMS 0.019) for three factors. The components are the best summary of the scores themselves; the factors are the better account of why the tests correlate.

So the choice depends on the question. To compress data, draw a map or prepare inputs for another method, PCA is simpler, has a unique solution up to sign, and needs no distributional assumptions. To explain correlations by underlying traits, with each measurement allowed its own error, factor analysis states that model, though it then requires choices of the number of factors and of a rotation. Some programs offer a "principal component method" of factor extraction, which takes the factor loadings from the eigenvectors of $R$ scaled by the square roots of the eigenvalues. It is a quick approximation to factor analysis that inherits PCA's inflated communalities, as the table shows, and it is not the same as fitting the factor model.

Before running a factor analysis

  • Look at the correlation matrix first. Factor analysis can only find structure that the correlations contain. If the variables are nearly uncorrelated there is nothing for common factors to explain; if one block of variables dominates, the method will find it.
  • Use enough data. The estimates rest on the correlations, and correlations from small samples are noisy. Rules of thumb ask for several instances per variable; 301 children for nine tests is comfortable.
  • The model is linear. Each variable is a weighted sum of the factors. Ordinal items such as five-point survey responses violate this mildly, and specialized methods exist for them.
  • Expect to make judgments. The number of factors, the rotation and the names are choices, and different defensible choices give different solutions. Reporting the residual correlations and the uniquenesses alongside the rotated loadings lets a reader judge them.

In Python

scikit-learn fits the model by maximum likelihood with FactorAnalysis, which can also apply varimax.

import numpy as np
from sklearn.decomposition import FactorAnalysis

# X: the 301 x 9 test scores, in the order of the tables above
Z = (X - X.mean(0)) / X.std(0)                 # covariance of Z is the correlation matrix
fa = FactorAnalysis(n_components=3, rotation='varimax').fit(Z)

L = fa.components_.T                           # 9 x 3 loadings (column order and signs may differ)
psi = fa.noise_variance_                       # uniquenesses
print(np.round((L ** 2).sum(1) + psi, 3))      # communality + uniqueness: 1 for every test, to rounding
implied = L @ L.T + np.diag(psi)               # the model's correlation matrix
print(np.abs(np.corrcoef(Z.T) - implied).max())   # about 0.042
scores = fa.transform(Z)                       # estimated factor scores

components_ holds one factor per row, so it is transposed to put the tests in rows. The factors come out in no particular order and with arbitrary signs, so the script that produced this article's numbers sorts them by their sums of squared loadings and makes each factor's largest loading positive. That script, tools/experiments/data-science/factor-analysis.py, also checks the scikit-learn fit against a direct numerical minimization of the discrepancy $F$ and reproduces the varimax rotation independently. The data are the Rdatasets copy of the Holzinger and Swineford scores (sem::HS.data, originally from the MBESS package), columns visual, cubes, flags, paragrap, sentence, wordm, addition, counting and straight, loaded by tools/experiments/data-science/dsdata.py. The test names in this article follow lavaan's HolzingerSwineford1939, whose variables x1 to x9 are these nine tests and whose correlations agree with these to the third decimal; lavaan calls the third test "lozenges" where this copy's column is flags. Harman's widely reprinted correlation matrix of 24 tests for the Grant-White children, which labels the same test "Flags", differs from the correlations computed from these raw scores for the same children by more than 0.05 in two of the 36 entries (by 0.19 for cubes and flags), so the numbers here come from the raw scores, not from a published matrix.

References