After a linear trend and an annual cycle are removed from Iowa's weekly deaths from all causes, what remains is not noise. Autocorrelation and Stationarity found that the remainder has autocorrelation 0.18 between consecutive weeks, and that a Ljung–Box test rejects independence decisively. A week with more deaths than the season predicts tends to be followed by another. Which model describes that dependence, how are its parameters estimated, and how much does it improve a forecast of the next year's deaths?
The standard answer is the family of autoregressive moving average (ARMA) models. An autoregression predicts the present from a weighted sum of the recent past plus a fresh shock; a moving average builds the present from the current and recent shocks. Combined, they describe the autocovariance of a stationary series with a few parameters, they forecast by a simple recursion, and their fitted shocks give a check of the model.
Here is the route. First we solve the first-order autoregression and find when it has a stationary solution. Then we compute the autocovariances of moving averages, and prove the conditions on the roots of the model's polynomials under which an ARMA model is causal and invertible. Next we derive the Yule–Walker equations, treat prediction as projection, derive the Durbin–Levinson recursion and the partial autocorrelations, and prove an asymptotic distribution for the estimate of a first-order autoregression. Finally we forecast, choose models, and apply all of it to Iowa: fit 2014 to 2018, forecast 2019, and check the intervals. The article assumes the definitions, the notation and the results of Autocorrelation and Stationarity (stationarity, autocovariance, white noise $Z_t \sim \mathrm{WN}(0, \sigma^2)$, the backshift operator $B$, the $m$-dependent central limit theorem and Lemma 1 there).
Filters and the definition of an ARMA model
A linear filter with coefficients $a_j$ turns a series $\{Y_t\}$ into $\sum_ja_jY_{t-j}$. Written with the backshift operator $B$, which shifts a series back one step ($BY_t = Y_{t-1}$, so $B^jY_t = Y_{t-j}$), the filter is $a(B)Y_t$ with $a(z) = \sum_ja_jz^j$. ARMA models are built from filters, so we first establish when an infinite filter makes sense.
Lemma 1 (filters). Let $\{Y_t\}$ be stationary with mean 0 and let $\sum_j\lvert a_j\rvert < \infty$, the sum over all integers $j$.
- $\sum_ja_jY_{t-j}$ converges in mean square, and the result is stationary with mean 0.
- If also $\sum_j\lvert b_j\rvert < \infty$, applying the filter $b$ and then the filter $a$ is the same as applying the single filter $c$ with $c_m = \sum_ja_jb_{m-j}$, the coefficients of the product $a(z)b(z)$, and $\sum_m\lvert c_m\rvert < \infty$.
Proof. (1) By Theorem 1 of the stationarity article, $\lvert E(Y_sY_t)\rvert \le \gamma_Y(0)$. So for $m < n$, $E\big(\sum_{m<\lvert j\rvert\le n}a_jY_{t-j}\big)^2 \le \gamma_Y(0)\big(\sum_{m<\lvert j\rvert\le n}\lvert a_j\rvert\big)^2$, which tends to 0 as $m \to \infty$. The partial sums form a Cauchy sequence in mean square, which converges because the space of variables with finite second moment is complete. The limit has mean 0, and its covariances are limits of $\sum_{j,k}a_ja_k\gamma_Y(h - k + j)$ over finite ranges, which depend only on $h$. (2) The double series $\sum_j\sum_ka_jb_kY_{t-j-k}$ converges absolutely in mean square norm, since $\sum_j\sum_k\lvert a_j\rvert\lvert b_k\rvert\sqrt{\gamma_Y(0)} < \infty$; an absolutely convergent double series can be summed in any order, and grouping terms with $j + k = m$ gives $\sum_mc_mY_{t-m}$. Finally $\sum_m\lvert c_m\rvert \le \sum_j\lvert a_j\rvert\sum_k\lvert b_k\rvert$. $\square$
Definition (ARMA). $\{X_t\}$ is an ARMA($p$, $q$) process if it is stationary and $$X_t - \phi_1X_{t-1} - \dots - \phi_pX_{t-p} = Z_t + \theta_1Z_{t-1} + \dots + \theta_qZ_{t-q}, \qquad Z_t \sim \mathrm{WN}(0, \sigma^2),$$ that is, $\phi(B)X_t = \theta(B)Z_t$ with the autoregressive polynomial $\phi(z) = 1 - \phi_1z - \dots - \phi_pz^p$ and the moving average polynomial $\theta(z) = 1 + \theta_1z + \dots + \theta_qz^q$. A process with mean $\mu$ is ARMA when $X_t - \mu$ is. ARMA($p$, 0) is an autoregression AR($p$); ARMA(0, $q$) is a moving average MA($q$). With these signs, positive $\phi_j$ and $\theta_j$ both make neighboring values positively correlated. The unit's main text, Brockwell and Davis (2016), has a chapter on ARMA models (chapter 3) and one on modeling and forecasting with them (chapter 5).
The process is causal if $X_t = \sum_{j\ge0}\psi_jZ_{t-j}$ with $\sum_j\lvert\psi_j\rvert < \infty$, so that the present depends only on present and past shocks, and invertible if $Z_t = \sum_{j\ge0}\pi_jX_{t-j}$ with $\sum_j\lvert\pi_j\rvert < \infty$, so that the shocks can be recovered from the present and past of the series.
The first-order autoregression
Here is the idea. Substituting the equation $X_t = \phi X_{t-1} + Z_t$ into itself $k$ times gives $X_t = Z_t + \phi Z_{t-1} + \dots + \phi^{k-1}Z_{t-k+1} + \phi^kX_{t-k}$. When $\lvert\phi\rvert < 1$ the last term fades, and the solution is a weighted sum of past shocks. When $\lvert\phi\rvert > 1$ the substitution must run forward in time instead. When $\lvert\phi\rvert = 1$ nothing fades, and there is no stationary solution.
Theorem 1 (stationary solutions of the AR(1) equation). Let $Z_t \sim \mathrm{WN}(0, \sigma^2)$ with $\sigma^2 > 0$.
- If $\lvert\phi\rvert < 1$, the equation $X_t = \phi X_{t-1} + Z_t$ has exactly one stationary solution, $X_t = \sum_{j=0}^\infty\phi^jZ_{t-j}$. It is causal, with $\gamma(h) = \sigma^2\phi^{\lvert h\rvert}/(1 - \phi^2)$ and $\rho(h) = \phi^{\lvert h\rvert}$.
- If $\lvert\phi\rvert > 1$, it has the stationary solution $X_t = -\sum_{j=1}^\infty\phi^{-j}Z_{t+j}$, which depends on future shocks.
- If $\lvert\phi\rvert = 1$, it has no stationary solution.
Proof. (1) The series converges in mean square by Lemma 1, since $\sum_j\lvert\phi\rvert^j < \infty$, and it solves the equation: $\phi X_{t-1} + Z_t = \sum_{j\ge0}\phi^{j+1}Z_{t-1-j} + Z_t = \sum_{j\ge0}\phi^jZ_{t-j}$. For uniqueness, let $\{Y_t\}$ be any stationary solution. Iterating the equation $k$ times, $Y_t - \sum_{j=0}^{k-1}\phi^jZ_{t-j} = \phi^kY_{t-k}$, whose mean square is $\phi^{2k}E(Y_{t-k}^2)$, a constant times $\phi^{2k}$ by stationarity, which tends to 0. So $Y_t$ equals the series. Its autocovariance is $\operatorname{Cov}(\sum_j\phi^jZ_{t-j}, \sum_k\phi^kZ_{t+h-k}) = \sigma^2\sum_{j\ge0}\phi^j\phi^{j+h} = \sigma^2\phi^h/(1 - \phi^2)$ for $h \ge 0$, since only the pairs with $k = j + h$ share a shock. (2) The series converges by Lemma 1, and $\phi X_{t-1} + Z_t = -\sum_{j\ge1}\phi^{1-j}Z_{t-1+j} + Z_t = -\sum_{j\ge2}\phi^{1-j}Z_{t-1+j} = -\sum_{k\ge1}\phi^{-k}Z_{t+k} = X_t$, since the $j = 1$ term is $-Z_t$. (3) If $\{Y_t\}$ were a stationary solution, iteration would give $Y_t - \phi^kY_{t-k} = \sum_{j=0}^{k-1}\phi^jZ_{t-j}$. The right side has variance $k\sigma^2$, because $\lvert\phi\rvert = 1$. The left side has variance at most $(\sqrt{\gamma_Y(0)} + \sqrt{\gamma_Y(0)})^2 = 4\gamma_Y(0)$ by the triangle inequality for standard deviations, whatever $k$. For $k > 4\gamma_Y(0)/\sigma^2$ this is impossible. $\square$
Part 3 includes the random walk ($\phi = 1$), which Autocorrelation and Stationarity showed has a variance growing with $t$, and Random Walks and Martingales studies. Part 2 shows that $\lvert\phi\rvert > 1$ is not a failure of stationarity but of causality: the solution exists and is stationary, but it is built from the future. Its autocovariances are $\sigma^2\phi^{-2}\phi^{-\lvert h\rvert}/(1 - \phi^{-2})$, the same as those of a causal AR(1) with coefficient $1/\phi$ and noise variance $\sigma^2/\phi^2$ (by the computation in part 1 with $\phi^{-1}$ in place of $\phi$). Second moments cannot tell the two apart, so nothing is lost by restricting attention to causal models.
Moving averages
Theorem 2 (the autocovariances of a moving average). If $X_t = \sum_{j=0}^q\theta_jZ_{t-j}$ with $\theta_0 = 1$ and $Z_t \sim \mathrm{WN}(0, \sigma^2)$, then $$\gamma(h) = \sigma^2\sum_{j=0}^{q-\lvert h\rvert}\theta_j\theta_{j+\lvert h\rvert} \text{ for } \lvert h\rvert \le q, \qquad \gamma(h) = 0 \text{ for } \lvert h\rvert > q.$$ More generally, a causal process $X_t = \sum_{j\ge0}\psi_jZ_{t-j}$ has $\gamma(h) = \sigma^2\sum_{j\ge0}\psi_j\psi_{j+\lvert h\rvert}$.
Proof. $\operatorname{Cov}(X_t, X_{t+h}) = \sum_j\sum_k\psi_j\psi_k\operatorname{Cov}(Z_{t-j}, Z_{t+h-k})$, where the covariance is $\sigma^2$ when $k = j + h$ and 0 otherwise (for an infinite sum, covariance is continuous under mean square limits). For a moving average, $\psi_j = \theta_j$ for $j \le q$ and 0 beyond, so no pair matches when $h > q$. $\square$
So the autocorrelation function of an MA($q$) cuts off after lag $q$, which is how a moving average is recognized in a correlogram. For example, the MA(2) process $X_t = Z_t + 0.4Z_{t-1} + 0.3Z_{t-2}$ with $\sigma^2 = 1$ has $\gamma(0) = 1 + 0.16 + 0.09 = 1.25$, $\gamma(1) = 0.4 + 0.4 \times 0.3 = 0.52$, $\gamma(2) = 0.3$, and $\gamma(h) = 0$ beyond.
Two moving averages with the same autocovariances. The MA(1) process $Z_t + \theta Z_{t-1}$ with noise variance $\sigma^2$ has $\gamma(0) = \sigma^2(1 + \theta^2)$ and $\gamma(1) = \theta\sigma^2$. Replace $\theta$ by $1/\theta$ and $\sigma^2$ by $\theta^2\sigma^2$: then $\gamma(0) = \theta^2\sigma^2(1 + \theta^{-2}) = \sigma^2(\theta^2 + 1)$ and $\gamma(1) = \theta^{-1}\theta^2\sigma^2 = \theta\sigma^2$, the same. With Gaussian noise the two processes have the same distribution: each finite stretch of either is multivariate normal with the same mean and covariance matrix, and these determine the distribution (Theorem 1 of The Multivariate Normal Distribution). So no data can distinguish them. The script confirms that $\theta = 0.5$ with $\sigma^2 = 1$ and $\theta = 2$ with $\sigma^2 = 0.25$ both give $\gamma(0) = 1.25$ and $\gamma(1) = 0.5$. The convention that resolves the ambiguity is invertibility, which the next section shows picks $\lvert\theta\rvert < 1$. For an invertible MA(1), $Z_t = X_t - \theta Z_{t-1} = \sum_{j\ge0}(-\theta)^jX_{t-j}$, by the argument of Theorem 1 with the roles of $X$ and $Z$ exchanged.
Causality and invertibility of ARMA models
Here is the idea. The AR(1) result says that $1 - \phi z$ can be inverted by a causal filter when its root $1/\phi$ lies outside the unit circle. A polynomial of degree $p$ factors into $p$ such linear pieces, one per root, and each can be inverted in the same way when its root lies outside the circle. The converse holds because a causal solution forces an identity between power series.
Theorem 3 (causality). Let $\phi(z)$ and $\theta(z)$ have no common zeros, and let $Z_t \sim \mathrm{WN}(0, \sigma^2)$ with $\sigma^2 > 0$. Then $\phi(B)X_t = \theta(B)Z_t$ has a causal stationary solution if and only if $\phi(z) \ne 0$ for every complex $z$ with $\lvert z\rvert \le 1$. The solution is then unique, and $X_t = \sum_{j\ge0}\psi_jZ_{t-j}$ with $\psi(z) = \sum_j\psi_jz^j = \theta(z)/\phi(z)$ for $\lvert z\rvert \le 1$.
Proof. If. Factor $\phi(z) = \prod_{i=1}^p(1 - z/r_i)$ over its roots $r_1, \dots, r_p$, which is possible because $\phi(0) = 1$; by assumption $\lvert r_i\rvert > 1$. Each factor $1 - z/r_i$ has the inverse power series $\sum_{j\ge0}r_i^{-j}z^j$, with absolutely summable coefficients. Let $\chi(z) = \sum_j\chi_jz^j$ be the product of these $p$ series. By Lemma 1(2), $\chi$ has absolutely summable coefficients, and as a power series $\chi(z)\phi(z) = 1$. Its coefficients are real, because they are the coefficients of the real function $1/\phi(z)$ near 0. Define $X_t = \chi(B)\theta(B)Z_t$. By Lemma 1 it is stationary, and $\phi(B)X_t = \phi(B)\chi(B)\theta(B)Z_t = \theta(B)Z_t$, because the composed filter $\phi(B)\chi(B)$ has coefficients of $\phi(z)\chi(z) = 1$. So $X_t$ is a causal solution with $\psi(z) = \chi(z)\theta(z)$. For uniqueness, apply $\chi(B)$ to both sides of $\phi(B)Y_t = \theta(B)Z_t$ for any stationary solution $Y_t$: Lemma 1 gives $Y_t = \chi(B)\phi(B)Y_t = \chi(B)\theta(B)Z_t$.
Only if. Let $X_t = \psi(B)Z_t$ be a causal solution, $\sum_j\lvert\psi_j\rvert < \infty$. Then $\theta(B)Z_t = \phi(B)X_t = \xi(B)Z_t$, with $\xi(z) = \phi(z)\psi(z)$ by Lemma 1(2). The difference $\sum_j(\theta_j - \xi_j)Z_{t-j}$ is 0, and its variance is $\sigma^2\sum_j(\theta_j - \xi_j)^2$, because the $Z_t$ are uncorrelated. So $\theta_j = \xi_j$ for every $j$, that is, $\phi(z)\psi(z) = \theta(z)$ as power series, and so as functions on $\lvert z\rvert \le 1$, where $\psi(z)$ converges absolutely. If $\phi(z_0) = 0$ with $\lvert z_0\rvert \le 1$, then $\theta(z_0) = 0$, a common zero, which is excluded. $\square$
Theorem 4 (invertibility). Under the assumptions of Theorem 3, if $\theta(z) \ne 0$ for $\lvert z\rvert \le 1$, then any stationary solution satisfies $Z_t = \sum_{j\ge0}\pi_jX_{t-j}$ with $\pi(z) = \phi(z)/\theta(z)$ and $\sum_j\lvert\pi_j\rvert < \infty$. Conversely, if the solution is causal and invertible, then $\theta(z) \ne 0$ for $\lvert z\rvert \le 1$.
Proof. The first part is the "if" half of Theorem 3 with the roles of $\phi$ and $\theta$ exchanged: invert $\theta(B)$ by a causal absolutely summable filter $\eta(B)$ with $\eta(z)\theta(z) = 1$ and apply it to $\theta(B)Z_t = \phi(B)X_t$. For the converse, substitute $X_t = \psi(B)Z_t$ into $Z_t = \pi(B)X_t$: $Z_t = \pi(B)\psi(B)Z_t$, and the argument of the "only if" half of Theorem 3 gives $\pi(z)\psi(z) = 1$ on $\lvert z\rvert \le 1$. Multiplying by $\phi(z)$ and using $\phi(z)\psi(z) = \theta(z)$ gives $\pi(z)\theta(z) = \phi(z)$. A zero $z_0$ of $\theta$ in the closed disk would then be a zero of $\phi$, a common zero. $\square$
The condition "no common zeros" matters. If $\phi(z)$ and $\theta(z)$ share a factor $1 - cz$, it cancels from both sides, and the model is the same as one of lower order: an ARMA(1, 1) with $\phi_1 = -\theta_1$ is white noise. When fitted coefficients come close to such a cancellation, the model is close to a simpler one, and its coefficients are poorly determined. Iowa's fitted ARMA(1, 1) shows this below.
Example (an AR(2)). $X_t = 0.5X_{t-1} + 0.3X_{t-2} + Z_t$ has $\phi(z) = 1 - 0.5z - 0.3z^2$, whose roots are 1.174 and −2.840, both outside the unit circle. So it is causal, and its $\psi$ weights are the coefficients of $1/\phi(z)$. The script checks its autocovariances from those weights against the Yule–Walker equations of the next section.
How fast the autocovariances decay. For a causal ARMA model, $\psi(z) = \theta(z)/\phi(z)$ is analytic on a disk of radius $R = \min_i\lvert r_i\rvert > 1$. Its power series converges absolutely at any $z$ with $1 < \lvert z\rvert = s < R$, so its terms are bounded: $\lvert\psi_j\rvert \le Ms^{-j}$ for some $M$. By Theorem 2, $\lvert\gamma(h)\rvert \le \sigma^2M^2\sum_js^{-j}s^{-j-h} = Cs^{-h}$. The autocovariances of an ARMA model decay geometrically, and the long-run variance $\sum_h\gamma(h)$ of the stationarity article is finite. Iowa's raw series, whose correlogram is a cycle that does not decay, is not of this kind; its remainder after the cycle is removed can be.
The Yule–Walker equations
Theorem 5 (Yule–Walker equations). For a causal AR($p$) process $X_t = \phi_1X_{t-1} + \dots + \phi_pX_{t-p} + Z_t$, $$\gamma(0) - \sum_{j=1}^p\phi_j\gamma(j) = \sigma^2, \qquad \gamma(h) - \sum_{j=1}^p\phi_j\gamma(h - j) = 0 \text{ for } h \ge 1.$$
Proof. Multiply the equation $X_t - \sum_j\phi_jX_{t-j} = Z_t$ by $X_{t-h}$ and take expectations. The left side gives $\gamma(h) - \sum_j\phi_j\gamma(h - j)$. On the right, $E(Z_tX_{t-h}) = \sum_{k\ge0}\psi_kE(Z_tZ_{t-h-k})$ by causality, which is $\sigma^2\psi_0 = \sigma^2$ when $h = 0$ (only $k = 0$ matches) and 0 when $h \ge 1$ (no $k \ge 0$ makes $t - h - k = t$). $\square$
The equations for $h = 1, \dots, p$ form a linear system $\Gamma_p\boldsymbol\phi = \boldsymbol\gamma_p$, where $\Gamma_p = [\gamma(i - j)]_{i,j=1}^p$ and $\boldsymbol\gamma_p = (\gamma(1), \dots, \gamma(p))^{\mathsf T}$. Given the autocovariances it yields the coefficients, and given the coefficients the equations for $h > p$ continue the autocovariances recursively. They are named for Yule (1927), whose paper on Wolfer's sunspot numbers questioned, in the words of its abstract, the hypothesis that a periodicity is masked "solely by such more or less random superposed fluctuations", and Walker (1931), whose abstract describes Yule's idea as supposing "a certain amount of causal relationship between the successive annual numbers".
Example (the AR(2), continued). Divide the equations by $\gamma(0)$. At $h = 1$, $\rho(1) = 0.5 + 0.3\rho(1)$, so $\rho(1) = 0.5/0.7 = 0.7143$. At $h = 2$, $\rho(2) = 0.5\rho(1) + 0.3 = 0.6571$, and at $h = 3$, $\rho(3) = 0.5\rho(2) + 0.3\rho(1) = 0.5429$. The equation at $h = 0$ gives $\gamma(0) = \sigma^2/(1 - 0.5\rho(1) - 0.3\rho(2)) = 2.2436$ for $\sigma^2 = 1$. The script obtains the same values from the $\psi$ weights of the AR(2).
Best linear prediction
To forecast $X_{n+h}$ from $X_1, \dots, X_n$, we look for the linear combination $a_0 + \sum_ia_iX_i$ with the smallest mean squared error. For a stationary process with mean 0 the constant is 0, and the problem is a projection.
Theorem 6 (the best linear predictor). Let $\{X_t\}$ be stationary with mean 0 and $\Gamma_n = [\gamma(i - j)]_{i,j=1}^n$ nonsingular. Among linear combinations $\mathbf a^{\mathsf T}\mathbf X$ of $\mathbf X = (X_n, X_{n-1}, \dots, X_1)^{\mathsf T}$, the mean squared error $E(X_{n+h} - \mathbf a^{\mathsf T}\mathbf X)^2$ is minimized uniquely by the solution of $$\Gamma_n\mathbf a = \boldsymbol\gamma_n^{(h)}, \qquad \boldsymbol\gamma_n^{(h)} = (\gamma(h), \gamma(h + 1), \dots, \gamma(h + n - 1))^{\mathsf T},$$ equivalently by the $\mathbf a$ whose error is uncorrelated with each of $X_1, \dots, X_n$. The minimum is $\gamma(0) - \mathbf a^{\mathsf T}\boldsymbol\gamma_n^{(h)}$.
Proof. The covariance matrix of $\mathbf X$ (in the order given) is $\Gamma_n$, and $\operatorname{Cov}(\mathbf X, X_{n+h}) = \boldsymbol\gamma_n^{(h)}$. So $E(X_{n+h} - \mathbf a^{\mathsf T}\mathbf X)^2 = \gamma(0) - 2\mathbf a^{\mathsf T}\boldsymbol\gamma_n^{(h)} + \mathbf a^{\mathsf T}\Gamma_n\mathbf a$. With $\mathbf a^* = \Gamma_n^{-1}\boldsymbol\gamma_n^{(h)}$ this equals $\gamma(0) - \mathbf a^{*\mathsf T}\boldsymbol\gamma_n^{(h)} + (\mathbf a - \mathbf a^*)^{\mathsf T}\Gamma_n(\mathbf a - \mathbf a^*)$, by expanding the last term. $\Gamma_n$ is nonnegative definite (Theorem 1 of the stationarity article) and nonsingular, hence positive definite, so the last term is positive unless $\mathbf a = \mathbf a^*$. The normal equations $\Gamma_n\mathbf a = \boldsymbol\gamma_n^{(h)}$ say exactly that $\operatorname{Cov}(X_{n+h} - \mathbf a^{\mathsf T}\mathbf X, X_i) = 0$ for each $i$. $\square$
This is the projection of The Geometry of Least Squares, with the inner product $E(UV)$ of random variables in place of the dot product of data vectors: the error is perpendicular to every predictor. We write $P_nX_{n+h}$ for the best linear predictor. For a Gaussian process it is also the best predictor of all, the conditional expectation $E(X_{n+h} \mid X_1, \dots, X_n)$, because a normal vector's conditional mean is linear in the conditioning variables (Theorem 6 of The Multivariate Normal Distribution) and the conditional mean minimizes mean squared error (Conditional Expectation and the Laws of Total Expectation and Variance, Theorem 3, part 4). For the same reason the Gaussian prediction error is normal with variance equal to the minimum above, which gives prediction intervals.
The Durbin–Levinson recursion. Solving the normal equations afresh for each $n$ costs of order $n^3$ operations. Because $\Gamma_n$ is a symmetric Toeplitz matrix, the one-step predictors can be updated from $n - 1$ to $n$ in order $n$ operations. Write $P_nX_{n+1} = \sum_{j=1}^n\phi_{nj}X_{n+1-j}$ and $v_n = E(X_{n+1} - P_nX_{n+1})^2$, with $v_0 = \gamma(0)$.
Theorem 7 (Durbin–Levinson). If every $\Gamma_n$ is nonsingular, then $\phi_{11} = \gamma(1)/\gamma(0)$ and for $n \ge 2$ $$\phi_{nn} = \frac{\gamma(n) - \sum_{j=1}^{n-1}\phi_{n-1,j}\gamma(n - j)}{v_{n-1}}, \qquad \phi_{nj} = \phi_{n-1,j} - \phi_{nn}\phi_{n-1,n-j} \ (1 \le j \le n - 1), \qquad v_n = v_{n-1}(1 - \phi_{nn}^2).$$
Here is the idea. Predicting $X_{n+1}$ from $X_1, \dots, X_n$ adds one variable, $X_1$, to the predictor from $X_2, \dots, X_n$. Only the part of $X_1$ that the others cannot predict is new. Stationarity makes the backward prediction of $X_1$ from $X_2, \dots, X_n$ use the same coefficients, reversed, as the forward prediction of $X_{n+1}$ from $X_n, \dots, X_2$.
Proof. Let $H$ be the span of $X_2, \dots, X_n$, and write $P_H$ for the best linear predictor from them. Since $(X_2, \dots, X_{n+1})$ has the same covariances as $(X_1, \dots, X_n)$, $P_HX_{n+1} = \sum_{j=1}^{n-1}\phi_{n-1,j}X_{n+1-j}$ with error $U = X_{n+1} - P_HX_{n+1}$ of mean square $v_{n-1}$. The covariance matrix of $(X_1, \dots, X_n)$ is unchanged by reversing the order, because $\gamma$ is even, so the best linear predictor of $X_1$ from $X_2, \dots, X_n$ is $P_HX_1 = \sum_{j=1}^{n-1}\phi_{n-1,j}X_{1+j}$, with error $W = X_1 - P_HX_1$ also of mean square $v_{n-1}$. The variable $W$ is uncorrelated with every element of $H$, and $X_1, \dots, X_n$ span the same space as $H$ and $W$. Projecting onto $H$ and onto $W$ separately therefore gives $$P_nX_{n+1} = P_HX_{n+1} + aW, \qquad a = \frac{E(X_{n+1}W)}{E(W^2)},$$ which has an error uncorrelated with $H$ and with $W$, as Theorem 6 requires. Since $P_HX_{n+1}$ is uncorrelated with $W$, $E(X_{n+1}W) = E(X_{n+1}X_1) - \sum_j\phi_{n-1,j}E(X_{n+1}X_{1+j}) = \gamma(n) - \sum_j\phi_{n-1,j}\gamma(n - j)$, so $a$ is the stated $\phi_{nn}$. Expanding $aW = a(X_1 - \sum_j\phi_{n-1,j}X_{1+j})$, the coefficient of $X_1$ is $a$, and the coefficient of $X_{n+1-j}$ for $j < n$ is $\phi_{n-1,j} - a\phi_{n-1,n-j}$, since $X_{1+k} = X_{n+1-j}$ when $k = n - j$. Finally, the error is $U - aW$, and $E(UW) = E(X_{n+1}W) = av_{n-1}$, so $v_n = v_{n-1} - 2a^2v_{n-1} + a^2v_{n-1} = v_{n-1}(1 - a^2)$. $\square$
The recursion is due to Levinson (1946), in a paper on filter design and prediction, and Durbin (1960), in a paper on fitting time-series models; we read only their titles and bibliographic records. The script checks it against direct solution of the normal equations on Iowa's residual autocovariances (agreement to $10^{-10}$ at $n = 3$ and $n = 10$).
The partial autocorrelation function. The last coefficient $\phi_{hh}$ measures what $X_1$ adds to the prediction of $X_{h+1}$ beyond $X_2, \dots, X_h$. It is called the partial autocorrelation at lag $h$, $\alpha(h) = \phi_{hh}$, with $\alpha(1) = \rho(1)$. Since $\phi_{hh}$ is the coefficient of $W$, a scaled correlation of $X_{h+1}$ with the unpredicted part of $X_1$, and $v_h = v_{h-1}(1 - \alpha(h)^2) \ge 0$, every $\lvert\alpha(h)\rvert \le 1$.
Theorem 8 (the partial autocorrelations of an autoregression cut off). For a causal AR($p$) process, $\alpha(p) = \phi_p$ and $\alpha(h) = 0$ for $h > p$.
Proof. Let $h \ge p$. The candidate $\sum_{j=1}^p\phi_jX_{h+1-j}$ is a linear combination of $X_1, \dots, X_h$, and its error $X_{h+1} - \sum_j\phi_jX_{h+1-j} = Z_{h+1}$ is uncorrelated with $X_1, \dots, X_h$, because by causality each of these is a sum of $Z_s$ with $s \le h$. By Theorem 6 it is the best linear predictor. So $\phi_{hj} = \phi_j$ for $j \le p$ and $\phi_{hj} = 0$ for $p < j \le h$. At $h = p$ the last coefficient is $\phi_p$; for $h > p$ it is 0. $\square$
The two cutoffs mirror each other: the autocorrelations of an MA($q$) vanish beyond $q$ (Theorem 2), the partial autocorrelations of an AR($p$) beyond $p$. Neither cuts off for the other model. The AR(2) example has partial autocorrelations $\alpha(1) = 0.7143$, $\alpha(2) = 0.3$ and zeros beyond (Worked problem 3 computes them by hand), with one-step prediction variances $v_1 = 1.0989$ and $v_2 = 1$. For the MA(1) with $\theta = 0.8$ the script's Durbin–Levinson recursion gives $\alpha(1) = 0.4878$, $\alpha(2) = -0.3123$, $\alpha(3) = 0.2215$ and $\alpha(4) = -0.1652$, decaying with alternating signs but never zero. For the sample partial autocorrelations of an AR($p$) beyond lag $p$, the band $\pm1.96/\sqrt n$ is used as a guide in the same way as for the sample autocorrelations of noise. We do not prove a distribution for them; in the script, for 1,000 series of length 261 from the AR(2) above for each of 30 seeds, $n$ times the variance of $\hat\alpha(3)$ had median 0.98 and that of $\hat\alpha(4)$ median 0.97.
Estimation
Three estimators are in common use. All assume the order $(p, q)$ is given; choosing it comes later.
- Yule–Walker. For an AR($p$), replace the autocovariances in Theorem 5 by the sample autocovariances and solve $\hat\Gamma_p\hat{\boldsymbol\phi} = \hat{\boldsymbol\gamma}_p$, with $\hat\sigma^2 = \hat\gamma(0) - \hat{\boldsymbol\phi}^{\mathsf T}\hat{\boldsymbol\gamma}_p$. Theorem 5 of the stationarity article guarantees that $\hat\Gamma_p$ is positive definite when the data are not constant, so the system has one solution. The Durbin–Levinson recursion solves it for every order up to $p$ at once, giving the sample partial autocorrelations as a by-product. For $p = 1$, $\hat\phi = \hat\rho(1)$.
- Conditional least squares. For an AR($p$), regress $x_t$ on $x_{t-1}, \dots, x_{t-p}$ (and a constant) over $t = p + 1, \dots, n$, by ordinary least squares. This minimizes the sum of squared one-step errors, conditional on the first $p$ observations.
- Gaussian maximum likelihood. If the noise is Gaussian, $(X_1, \dots, X_n)$ is $N_n(\mu\mathbf 1, \Gamma_n)$ with $\Gamma_n$ a function of $(\boldsymbol\phi, \boldsymbol\theta, \sigma^2)$, and the log-likelihood is $-\frac12[n\ln(2\pi) + \ln\det\Gamma_n + (\mathbf x - \mu\mathbf 1)^{\mathsf T}\Gamma_n^{-1}(\mathbf x - \mu\mathbf 1)]$ (the density of The Multivariate Normal Distribution, Theorem 4). Writing $\Gamma_n = \sigma^2R_n$, the maximizing $\sigma^2$ for given $(\boldsymbol\phi, \boldsymbol\theta)$ is $\mathbf e^{\mathsf T}R_n^{-1}\mathbf e/n$ with $\mathbf e = \mathbf x - \mu\mathbf 1$; with regressors in place of $\mu\mathbf 1$, the maximizing coefficients for given $(\boldsymbol\phi, \boldsymbol\theta)$ are the generalized least squares estimate. What remains is maximized numerically. This is how the script fits every model in this article, with $R_n$ factored by its Cholesky factorization and the coefficients transformed so that only causal and invertible values are searched. The same likelihood is used when the noise is not Gaussian, as a criterion.
Theorem 9 (the least-squares estimate of an AR(1) is asymptotically normal). Let $X_t = \phi X_{t-1} + Z_t$ with $\lvert\phi\rvert < 1$ and $Z_t \sim \mathrm{IID}(0, \sigma^2)$ Gaussian, and let $\hat\phi = \sum_{t=2}^nX_{t-1}X_t/\sum_{t=2}^nX_{t-1}^2$ (the mean known to be 0). Then $$\sqrt n(\hat\phi - \phi) \xrightarrow{d} N(0, 1 - \phi^2).$$ The Yule–Walker estimate $\tilde\phi = \sum_{t=2}^nX_{t-1}X_t/\sum_{t=1}^nX_t^2$ has the same limit.
Here is the idea. The error $\hat\phi - \phi = \sum X_{t-1}Z_t/\sum X_{t-1}^2$ is a ratio. The denominator, divided by $n$, settles at $\gamma(0)$. The numerator is a sum of products $X_{t-1}Z_t$ that are uncorrelated but not independent; truncating $X_{t-1}$ to its last $m$ shocks makes them $m$-dependent, and the $m$-dependent central limit theorem with the approximation lemma of the stationarity article finishes the job.
Proof. Substituting $X_t = \phi X_{t-1} + Z_t$, $\hat\phi - \phi = \sum_{t=2}^nX_{t-1}Z_t/\sum_{t=2}^nX_{t-1}^2$, so $$\sqrt n(\hat\phi - \phi) = \frac{n^{-1/2}\sum_{t=2}^nX_{t-1}Z_t}{n^{-1}\sum_{t=2}^nX_{t-1}^2}.$$ Denominator. $n^{-1}\sum_{t=1}^nX_t^2 \to \gamma(0)$ in probability, by Step 2 of the proof of Theorem 4 in Autocorrelation and Stationarity, which applies to any stationary Gaussian process with $\sum_h\gamma(h)^2 < \infty$; removing the single term $X_n^2/n$ changes nothing in the limit.
Numerator. Let $X_{t-1}^{(m)} = \sum_{j=0}^{m-1}\phi^jZ_{t-1-j}$ and $Y_t = X_{t-1}^{(m)}Z_t$. The sequence $\{Y_t\}$ is strictly stationary (the same function of the shifted i.i.d. noise at every $t$) and $m$-dependent ($Y_t$ uses $Z_{t-m}, \dots, Z_t$). For $s > t$, $E(Y_tY_s) = E(Y_tX_{s-1}^{(m)})E(Z_s) = 0$, since $Z_s$ is independent of the earlier shocks; so the $Y_t$ are uncorrelated, with variance $E(X_{t-1}^{(m)2})\sigma^2 = \sigma^4(1 - \phi^{2m})/(1 - \phi^2)$. Theorem 6 of the stationarity article gives $n^{-1/2}\sum_tY_t \xrightarrow{d} N(0, \tau_m^2)$ with $\tau_m^2 = \sigma^4(1 - \phi^{2m})/(1 - \phi^2) \to \sigma^4/(1 - \phi^2) = \sigma^2\gamma(0)$. The remainder $n^{-1/2}\sum_t(X_{t-1} - X_{t-1}^{(m)})Z_t$ is a sum of uncorrelated terms by the same argument, each with variance $\sigma^2E(\sum_{j\ge m}\phi^jZ_{t-1-j})^2 = \sigma^4\phi^{2m}/(1 - \phi^2)$, so its mean square is at most $\sigma^4\phi^{2m}/(1 - \phi^2)$ for every $n$, which tends to 0 as $m \to \infty$. By Chebyshev's inequality and Lemma 1 of the stationarity article, $n^{-1/2}\sum_tX_{t-1}Z_t \xrightarrow{d} N(0, \sigma^2\gamma(0))$.
Ratio. By Slutsky's theorem the ratio tends to $N(0, \sigma^2\gamma(0)/\gamma(0)^2) = N(0, \sigma^2/\gamma(0)) = N(0, 1 - \phi^2)$. For $\tilde\phi$: $\tilde\phi = \hat\phi\cdot\sum_{t=2}^nX_{t-1}^2/\sum_{t=1}^nX_t^2 = \hat\phi(1 - X_n^2/\sum_{t=1}^nX_t^2)$. Since $X_n^2$ has a fixed distribution and $\sum_tX_t^2/n \to \gamma(0)$, $\sqrt n\hat\phi X_n^2/\sum_tX_t^2 = \hat\phi\cdot(X_n^2/\sqrt n)/(\sum_tX_t^2/n) \to 0$ in probability, and the converging-together lemma gives the same limit. $\square$
Gaussian noise is used only for the denominator, through the variance computation in the stationarity article. For i.i.d. noise with finite variance the denominator converges as well, by the ergodic theorem (Durrett, 2019, Theorem 6.2.1), since an i.i.d. sequence is ergodic and a stationary sequence obtained from an ergodic one by a fixed measurable function is ergodic (Durrett, 2019, Example 6.1.5 and Exercise 6.1.5); we cite that route rather than prove it. Hannan (1973) built a general asymptotic theory for linear time-series models in which, in the words of his abstract, "various estimation procedures are shown to be equivalent", together with a central limit theorem for regression on general regressors. We cite that theory, at the level of the abstract, for models beyond the AR(1), and do not prove it.
How well the three estimators do in short series. The limit says nothing about bias for small $n$ or for $\phi$ near 1. The script simulated stationary Gaussian AR(1) series with $\sigma^2 = 1$, 1,000 series for each of 30 seeds, and estimated $\phi$ by Yule–Walker, by least squares with an intercept, and by maximum likelihood with the sample mean removed. The table gives the median over the seeds of the average error $\hat\phi - \phi$, and the coverage of the Wald interval $\hat\phi \pm 1.96\sqrt{(1 - \hat\phi^2)/n}$ built on the least-squares estimate. A length of 261 is that of Iowa's training data.
| $n$ | $\phi$ | Bias, Yule–Walker | Bias, least squares | Bias, maximum likelihood | Wald coverage |
|---|---|---|---|---|---|
| 52 | 0.50 | −0.056 | −0.047 | −0.046 | 0.933 |
| 52 | 0.90 | −0.101 | −0.080 | −0.075 | 0.878 |
| 52 | 0.98 | −0.125 | −0.093 | −0.086 | 0.776 |
| 261 | 0.50 | −0.012 | −0.010 | −0.010 | 0.946 |
| 261 | 0.90 | −0.018 | −0.015 | −0.014 | 0.937 |
| 261 | 0.98 | −0.023 | −0.017 | −0.016 | 0.881 |
All three estimators are biased toward zero, more so near the unit root and in short series, and in these simulations Yule–Walker is the most biased. Its denominator includes one more squared term than least squares, and its sample autocovariance carries the downward bias that Autocorrelation and Stationarity (Proposition 2) computed exactly. At $\phi = 0.98$ with 52 observations the average Yule–Walker estimate is about 0.86, and the Wald interval covers 78% of the time instead of 95%. Near the unit root the normal limit of Theorem 9 is a poor guide, since its variance $1 - \phi^2$ tends to 0 while the bias does not. At $n = 261$ and $\phi = 0.9$ the asymptotic standard deviation is $\sqrt{0.19/261} = 0.027$, and the bias of about −0.015 is about half of it. These are results for Gaussian AR(1) series of these lengths.
Forecasting
For a causal AR(1) with mean 0, the forecast and its error follow from iterating the equation. Since $X_{n+h} = \phi^hX_n + \sum_{j=0}^{h-1}\phi^jZ_{n+h-j}$ and the sum is uncorrelated with $X_1, \dots, X_n$ (it uses only shocks after time $n$), Theorem 6 gives $$P_nX_{n+h} = \phi^hX_n, \qquad E(X_{n+h} - P_nX_{n+h})^2 = \sigma^2\sum_{j=0}^{h-1}\phi^{2j} = \sigma^2\frac{1 - \phi^{2h}}{1 - \phi^2}.$$ The forecast decays geometrically to the mean, and the error variance grows to $\gamma(0) = \sigma^2/(1 - \phi^2)$, the variance of the series itself: far ahead, the past says nothing. With Gaussian noise the 95% prediction interval is $\phi^hX_n \pm 1.96\,\sigma\sqrt{(1 - \phi^{2h})/(1 - \phi^2)}$. For a general causal ARMA model, the same computation with $X_{n+h} = \sum_{j\ge0}\psi_jZ_{n+h-j}$ shows that predicting from the infinite past leaves the error $\sum_{j=0}^{h-1}\psi_jZ_{n+h-j}$ with variance $\sigma^2\sum_{j<h}\psi_j^2$. From a finite past the exact forecast is the Gaussian conditional mean given $X_1, \dots, X_n$, which is what the script computes.
Forecasting a regression with ARMA errors. Iowa's model is $y_t = \mathbf x_t^{\mathsf T}\boldsymbol\beta + e_t$, with $\mathbf x_t$ the trend and harmonic terms and $e_t$ an ARMA process with covariance matrix $\Gamma$. Given the ARMA parameters, $\boldsymbol\beta$ is estimated by generalized least squares, $\hat{\boldsymbol\beta} = (X^{\mathsf T}\Gamma^{-1}X)^{-1}X^{\mathsf T}\Gamma^{-1}\mathbf y$, and a future value $y_f$ is forecast by $$\hat y_f = \mathbf x_f^{\mathsf T}\hat{\boldsymbol\beta} + \mathbf c^{\mathsf T}\Gamma^{-1}(\mathbf y - X\hat{\boldsymbol\beta}), \qquad \mathbf c = \operatorname{Cov}(\mathbf e, e_f).$$ Proposition 1 (the forecast error). With the ARMA parameters known and Gaussian errors, $y_f - \hat y_f$ is normal with mean 0 and variance $$\operatorname{Var}(e_f) - \mathbf c^{\mathsf T}\Gamma^{-1}\mathbf c + \mathbf d^{\mathsf T}(X^{\mathsf T}\Gamma^{-1}X)^{-1}\mathbf d, \qquad \mathbf d = \mathbf x_f - X^{\mathsf T}\Gamma^{-1}\mathbf c.$$
Proof. Write $\mathbf A = \Gamma^{-1}\mathbf c$, so that $\mathbf A^{\mathsf T}\mathbf e$ is the best linear predictor of $e_f$ from $\mathbf e$ (Theorem 6, in matrix form). Substituting $\mathbf y = X\boldsymbol\beta + \mathbf e$, the forecast error is $$y_f - \hat y_f = (e_f - \mathbf A^{\mathsf T}\mathbf e) - (\mathbf x_f - X^{\mathsf T}\mathbf A)^{\mathsf T}(\hat{\boldsymbol\beta} - \boldsymbol\beta).$$ The first term has variance $\operatorname{Var}(e_f) - \mathbf c^{\mathsf T}\Gamma^{-1}\mathbf c$ and is uncorrelated with every linear function of $\mathbf e$. The second is a linear function of $\mathbf e$, because $\hat{\boldsymbol\beta} - \boldsymbol\beta = (X^{\mathsf T}\Gamma^{-1}X)^{-1}X^{\mathsf T}\Gamma^{-1}\mathbf e$, with covariance matrix $(X^{\mathsf T}\Gamma^{-1}X)^{-1}$. So the variances add, and the error, a linear function of jointly normal variables, is normal with mean 0. $\square$
In practice the ARMA parameters are estimated, and the formula is used with the estimates plugged in. Hyndman and Athanasopoulos (2021, §9.8) note that such intervals "tend to be too narrow" because "only the variation in the errors has been accounted for", not the variation in the parameter estimates and in the model order. The next simulation measures that for the AR(1).
How much do estimated parameters cost? The script simulated Gaussian AR(1) series with $\phi = 0.9$, $\sigma^2 = 1$ and $n = 261$, fitted each by maximum likelihood, and checked whether the plug-in 95% interval covered the value $h$ steps ahead (1,000 series for each of 30 seeds). The second column is $\sqrt{(1 - \phi^{2h})/(1 - \phi^2)}$, the true forecast standard deviation in units of $\sigma$. The last two columns repeat the fit with one ingredient known: the mean (0), with $\phi$ and $\sigma^2$ estimated; or $\phi$, with the mean and $\sigma^2$ estimated.
| Horizon $h$ | True forecast SD | Coverage, true parameters | Coverage, all estimated | IQR over seeds | Mean known | $\phi$ known |
|---|---|---|---|---|---|---|
| 1 | 1.00 | 0.949 | 0.948 | 0.943 to 0.953 | 0.948 | 0.948 |
| 4 | 1.73 | 0.949 | 0.937 | 0.933 to 0.944 | 0.942 | 0.945 |
| 13 | 2.22 | 0.950 | 0.929 | 0.921 to 0.934 | 0.942 | 0.944 |
| 52 | 2.29 | 0.949 | 0.924 | 0.917 to 0.927 | 0.940 | 0.940 |
One step ahead the parameter uncertainty costs almost nothing. At longer horizons the plug-in intervals cover 92% to 94% of the time. The last two columns separate two sources of the loss. With the mean known and $\phi$ and $\sigma^2$ estimated, the coverage is 0.940 at $h = 52$; with $\phi$ known and the mean and $\sigma^2$ estimated, it is also 0.940, against 0.924 with everything estimated. Each source alone costs about a point at long horizons, and together they cost between two and three points, more than the sum of the two. The sources interact: estimating the mean also roughly doubles the downward bias of $\hat\phi$, which shrinks the estimated variance of the series, $\hat\sigma^2/(1 - \hat\phi^2)$ (the limit of the forecast variance). In this simulation the maximum likelihood estimate had an average bias of −0.0064 with the mean known and −0.0142 with it estimated (medians over the 30 seeds).
Choosing a model
Information criteria. Hyndman and Athanasopoulos (2021, §9.6) choose $p$ and $q$ by the corrected Akaike criterion, $$\mathrm{AIC} = -2\ln L + 2K, \qquad \mathrm{AICc} = \mathrm{AIC} + \frac{2K(K + 1)}{T - K - 1},$$ where $L$ is the maximized likelihood, $T$ the length of the series and $K$ the number of estimated parameters including $\sigma^2$ (their formula writes $K = p + q + k + 1$ with $k$ for a constant; with regressors we count all their coefficients). Akaike's Information Criterion, Derived derives the penalty $2K$ as an estimate of the expected Kullback–Leibler divergence and the AICc correction exactly for the normal linear model (its Theorem 1), and shows (its Theorem 3) that AIC can choose too large a model. Hyndman and Athanasopoulos also warn that the criteria "tend not to be good guides to selecting the appropriate order of differencing", since the likelihoods of differenced and undifferenced data are not comparable; our models do not difference.
Residual diagnostics. If the model is right, the standardized one-step prediction errors, the innovations $(x_t - P_{t-1}x_t)/\sqrt{v_{t-1}}$, are approximately independent with variance 1. Their correlogram and a Ljung–Box test check this. For residuals of a fitted ARMA($p$, $q$) model the reference is the chi-squared distribution with $H - p - q$ degrees of freedom (Hyndman and Athanasopoulos, 2021, §9.7), in line with the tests of fit that Box and Pierce (1970) devised to allow for the fitted coefficients; Autocorrelation and Stationarity proves the uncorrected limit for noise and cites the correction. The script computes the exact innovations from the Cholesky factor of the fitted covariance matrix.
Parameter redundancy. A model whose autoregressive and moving average polynomials nearly share a root is nearly a model of lower order with one coefficient pair added. The near-cancelling coefficients are poorly determined and should not be interpreted one by one; Iowa's ARMA(1, 1) below is an example.
Iowa: fitting 2014 to 2018
The weekly all-cause deaths of 2014 to 2018 (National Center for Health Statistics, 2021), 261 weeks (2014 has 53), are the training data, and the 52 weeks of 2019 are kept for testing. The mean is modeled as in Autocorrelation and Stationarity: an intercept, a linear trend in years, $(t - 156)/P$, and three pairs of annual harmonics with period $P = 365.25/7$ weeks, eight coefficients in all, with $t = 0$ in the first week of 2014.
What the least-squares residuals look like. Fitted by ordinary least squares on the training weeks, the least-squares residuals have standard deviation 29.9 (divisor $n$). Their autocorrelations at lags 1 to 5 are 0.18, 0.15, 0.14, −0.05 and 0.09, and the partial autocorrelations of the residuals are 0.18, 0.12, 0.10, −0.11 and 0.09, against the guide band $\pm1.96/\sqrt{261} = \pm0.121$. Neither cuts off sharply. The autocorrelations decay slowly from a small start, which suggests a mixed model rather than a pure autoregression or moving average of low order.
The candidate models. The script fitted regression with ARMA($p$, $q$) errors for $p = 0, \dots, 3$ and $q = 0, 1, 2$, by exact Gaussian maximum likelihood, the eight regression coefficients profiled out by generalized least squares. Each model was maximized from 15 starting points (three fixed, twelve random), and a second run with eight new random starts (and the same three fixed ones) was required not to improve the maximum by more than 0.01 in $-2\ln L$. The search was restricted to coefficients whose autoregressive and moving average roots all have modulus at least 1.01, a margin from the unit circle. The margin is a convention, not a result: the forecast package for R uses the same cutoff in the function myarima that its automatic model selection auto.arima calls, setting the criterion of any fitted model with an autoregressive or moving average root of modulus below $1 + 10^{-2}$ to infinity (Hyndman et al., 2026, file R/newarima2.R; it discards such fits, where we restrict the search). None of the fits reported below lies on the margin: the smallest root modulus among them is 1.096. Without the margin, ARMA(2, 2) reaches $-2\ln L$ = 2,492.94 (against 2,496.07 with it) at a complex pair of autoregressive roots of modulus 1.007 and a pair of moving average roots of modulus 1.025 that nearly cancel them: a boundary solution in which a near-unit-root cycle is almost removed again by the moving average part. Its AICc would be 0.68 below that of ARMA(3, 1). ARMA(3, 2) also moves to the boundary without the margin, to $-2\ln L$ = 2,492.06 with a moving average root of modulus 1.0003, where its AICc is 0.67 above that of ARMA(3, 1). We regard these as artefacts of the boundary, not better models, and report the fits with the margin. The choice matters for the trend's standard error, as the paragraph on the trend below shows, but not for the forecasts. Differences in AICc from the smallest, 2,521.1 for ARMA(3, 1):
| $p$ | $q = 0$ | $q = 1$ | $q = 2$ |
|---|---|---|---|
| 0 | 11.48 | 6.75 | 4.73 |
| 1 | 5.09 | 1.41 | 3.21 |
| 2 | 3.14 | 3.14 | 2.45 |
| 3 | 2.59 | 0.00 | 1.98 |
ARMA(3, 1) has the smallest AICc and ARMA(1, 1) is 1.41 above it. Independent errors ($p = q = 0$) are 11.48 above, the clearest conclusion of the table.
The fitted coefficients.
- ARMA(1, 1): $\hat\phi_1 = 0.858$, $\hat\theta_1 = -0.735$, $\hat\sigma = 29.1$. The autoregressive root is $1/0.858 = 1.17$ and the moving average root $1/0.735 = 1.36$, close together on the positive axis, a near cancellation. What survives is a small autocorrelation that decays slowly: the fitted model's autocorrelations are 0.164, 0.140 and 0.120 at lags 1 to 3, against 0.18, 0.15 and 0.14 for the least-squares residuals.
- ARMA(3, 1): $\hat\phi = (-0.610, 0.229, 0.204)$, $\hat\theta_1 = 0.778$, $\hat\sigma = 28.7$. Its autoregressive polynomial has a root at 1.873 and a complex pair at $-1.499 \pm 0.610i$ (modulus 1.62), and the moving average root is at −1.286. These do not cancel: a real root cannot nearly cancel a complex pair. The pair adds a damped oscillation of period 2.3 weeks ($2\pi$ divided by the argument of the pair), shrinking by the factor $1/1.62$ per lag, to a slow decay from the real root; together they reproduce the dip of the residuals' correlogram at lag 4: its autocorrelations at lags 1 to 5 are 0.18, 0.16, 0.15, −0.02 and 0.08. With 8 of the twelve candidate models within 3.3 of the smallest AICc, the data do not determine these coefficients well, and they should not be interpreted one by one.
- AR(1): $\hat\phi_1 = 0.180$, $\hat\sigma = 29.4$; independent errors: $\hat\sigma = 29.9$.
Residual checks. Ljung–Box statistics of the standardized innovations at 10 lags, with $10 - p - q$ degrees of freedom: independent errors 30.7 ($p = 0.0007$), AR(1) 17.2 ($p = 0.046$), ARMA(1, 1) 10.2 ($p = 0.249$), ARMA(3, 1) 6.3 ($p = 0.394$). Independent errors and the AR(1) leave dependence behind; the two mixed models do not, by this test.
The trend. The trend coefficient is the change in expected weekly deaths per year. Under ARMA(3, 1) errors it is 6.09 deaths per week per year with standard error 1.87 (6.01 under ARMA(1, 1) errors); under independent errors it is 6.35 with standard error 1.30. Allowing for the positive autocorrelation widens the standard error: to 1.56 under AR(1) errors (+20%), 1.87 under ARMA(3, 1) errors (+44%) and 2.28 under ARMA(1, 1) errors (+75%). This is the effect that Theorem 4 of the stationarity article quantifies for a mean, and its size depends on the error model, which the data leave uncertain. Both estimates say that weekly deaths in Iowa rose over 2014 to 2018 after allowing for the season. The direction of the change in the standard error depends on the margin of the candidate-models paragraph. Without it, the boundary ARMA(2, 2) would give a trend of 7.14 with standard error 0.92, narrower than under independent errors: its nearly non-stationary wave of period about two years trades off against the linear trend. The widening above therefore holds for the models with roots away from the unit circle. The forecasts do not depend on the margin in the same way: the boundary ARMA(2, 2) forecasts 2019 with root mean squared error 28.0 a year ahead and 27.8 one week ahead, close to the values of the models in the tables below.
Iowa: forecasting 2019
Each fitted model forecasts all 52 weeks of 2019 from the end of 2018, with 80% and 95% intervals from Proposition 1, the estimated ARMA parameters plugged in. Under ARMA(3, 1) errors the forecast standard deviations run from 29.3 one week ahead to 31.3, nearly flat after the first few weeks: the dependence is weak, and the uncertainty is mostly that of a single week plus that of the extrapolated trend. For comparison:
- Independent errors: ordinary least squares with the usual prediction interval, $\hat y \pm t\,s\sqrt{1 + \mathbf x_f^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf x_f}$, $s = 30.3$ with $261 - 8$ degrees of freedom.
- Seasonal naive: each week of 2019 is forecast by the same week of 2018. Hyndman and Athanasopoulos (2021, §5.2) define the method by $\hat y_{T+h|T} = y_{T+h-m(k+1)}$, with $m$ the seasonal period and $k$ the integer part of $(h - 1)/m$, and (§5.5) give its $h$-step standard deviation as $\hat\sigma\sqrt{k + 1}$, which is $\hat\sigma$ for every week of 2019. Here $\hat\sigma = 44.9$, from the 209 differences $y_t - y_{t-52}$ in the training data.
| Method | RMSE | MAE | Inside 80% interval | Inside 95% interval |
|---|---|---|---|---|
| ARMA(3, 1) errors | 28.7 | 23.6 | 45 of 52 | 51 of 52 |
| ARMA(1, 1) errors | 28.4 | 23.3 | 46 of 52 | 51 of 52 |
| independent errors | 29.2 | 24.1 | 45 of 52 | 51 of 52 |
| seasonal naive | 38.6 | 32.2 | 46 of 52 | 52 of 52 |
The regression models forecast 2019 about a quarter better than the seasonal naive method, by root mean squared error. The seasonal naive forecast copies 2018's weekly noise into 2019, so its errors have the variance of a difference of two weeks, and it carries 2018's level, a mean of 579.1 deaths a week against 2019's 587.5. Its average error was 8.4 deaths, against 2.9 for ARMA(3, 1). The ARMA errors improve on independent errors only slightly when forecasting a whole year ahead, because a forecast of $e_t$ more than a few weeks ahead returns to 0. All the intervals covered about as often as their nominal levels say (80% of 52 is 41.6). The 52 weeks of one year are not independent trials, so these counts are a check of the intervals in one year, not an estimate of their coverage.
One week ahead. The dependence earns its keep at short horizons. Keeping the parameters and regression coefficients fixed at their 2014 to 2018 estimates, the script forecast each week of 2019 from all the weeks before it:
| Method | RMSE | MAE | Inside 80% interval | Inside 95% interval |
|---|---|---|---|---|
| ARMA(3, 1) errors | 27.9 | 21.9 | 44 of 52 | 50 of 52 |
| ARMA(1, 1) errors | 27.3 | 21.3 | 45 of 52 | 50 of 52 |
| independent errors | 29.2 | 24.1 | 45 of 52 | 50 of 52 |
With independent errors the past weeks carry no information beyond the fitted mean, so the one-week-ahead forecasts are the same as the year-ahead ones. The ARMA(1, 1) cuts the root mean squared error from 29.2 to 27.3 deaths. The gain is modest because the remainder's autocorrelation is small, 0.18 at lag 1: a model can only predict the part of next week's deviation that this week's deviation carries. The interval counts for the independent-errors model differ from the previous table because here they use the maximum likelihood variance with the coefficient uncertainty of Proposition 1, not the least-squares prediction interval.
What is proved here, what is cited, what the experiment shows
Proved here. Convergence and composition of absolutely summable filters (Lemma 1); the stationary solutions of the AR(1) equation, their uniqueness and the absence of one at $\lvert\phi\rvert = 1$ (Theorem 1); the autocovariances of moving averages and causal linear processes, the MA cutoff, and the equal autocovariances of MA(1) models with $\theta$ and $1/\theta$ (Theorem 2); causality of an ARMA model if and only if $\phi(z) \ne 0$ on the closed unit disk (Theorem 3); invertibility when $\theta(z) \ne 0$ there, and the converse for causal models (Theorem 4); geometric decay of ARMA autocovariances; the Yule–Walker equations (Theorem 5); the best linear predictor and its normal equations (Theorem 6); the Durbin–Levinson recursion (Theorem 7); the cutoff of the partial autocorrelations of an AR($p$) (Theorem 8); the asymptotic normality of the least-squares and Yule–Walker estimates of a Gaussian AR(1) (Theorem 9); the AR(1) forecast and its error variance; the forecast error of a regression with ARMA errors (Proposition 1).
Cited. The general asymptotic theory of estimation for linear time-series models (Hannan, 1973, at the level of his abstract); the ergodic theorem and the ergodicity of functions of i.i.d. sequences (Durrett, 2019); the origins of the Yule–Walker equations and the Durbin–Levinson recursion (Yule, 1927; Walker, 1931, at the level of their abstracts; Levinson, 1946; Durbin, 1960, at the level of their titles); the origin of the residual correction (Box and Pierce, 1970, at the level of their abstract); the unit's main text (Brockwell and Davis, 2016), cited for its chapter topics only; the 1.01 root margin as a convention of the forecast package (Hyndman et al., 2026); AICc for ARMA models, the residual degrees of freedom, the seasonal naive method and its standard deviation, and the limits of plug-in intervals (Hyndman and Athanasopoulos, 2021); the derivation of AIC and AICc (Akaike's Information Criterion, Derived).
What the experiments show. For the AR(2) example, $n$ times the variance of the sample partial autocorrelations beyond its order was near 1 (0.98 at lag 3, 0.97 at lag 4, $n = 261$). For Gaussian AR(1) series, all three estimators of $\phi$ were biased toward zero, Yule–Walker most, and Wald intervals covered 78% at $\phi = 0.98$ with 52 observations; with 261 observations the coverage was 88% to 95%. Plug-in forecast intervals for $\phi = 0.9$ covered 95% one step ahead and about 92% fifty-two steps ahead. On Iowa's weekly deaths, AICc, over models whose roots are at least 1.01 in modulus, preferred regression with ARMA(3, 1) errors, ARMA(1, 1) was close, and independent errors were clearly worse; the mixed models passed the Ljung–Box check; forecasting 2019 from 2018, the regression models had root mean squared errors of 28 to 29 deaths against 38.6 for the seasonal naive method, and the ARMA errors reduced the one-week-ahead error from 29.2 to 27.3. These are observations of these simulations and of one year of one state's data.
Worked problems
The four problems are posed here, on small exact cases and on the article's fitted models; the article's script checks each number.
Problem 1 (forecasting an AR(1) by hand). A series follows $X_t = 0.6X_{t-1} + Z_t$ with mean 0 and $\sigma^2 = 1$, and the last observation is $X_n = 5$. Give the forecasts and 95% prediction intervals for $h = 1, 2, 3$, and the limit of the forecast variance.
Solution. The forecasts are $0.6 \times 5 = 3$, $0.36 \times 5 = 1.8$ and $0.216 \times 5 = 1.08$, by $P_nX_{n+h} = \phi^hX_n$. The error variances are $(1 - 0.6^{2h})/(1 - 0.36)$: 1, $1 + 0.36 = 1.36$ and $1 + 0.36 + 0.1296 = 1.4896$, with standard deviations 1.00, 1.166 and 1.220. The intervals are $3 \pm 1.96$, $1.8 \pm 2.29$ and $1.08 \pm 2.39$. As $h$ grows the forecast tends to the mean 0 and the variance to $\gamma(0) = 1/0.64 = 1.5625$. The third forecast, 1.08, is already within one standard deviation of the mean: an AR(1) with $\phi = 0.6$ forgets its starting point within a few steps.
Problem 2 (choosing the invertible moving average). An MA(1) process has $\rho(1) = 0.4$. Find the two values of $\theta$ that give it, and say which one is invertible.
Solution. $\theta = 0.5$ (invertible) and $\theta = 2$. From $\theta/(1 + \theta^2) = 0.4$, $0.4\theta^2 - \theta + 0.4 = 0$, so $\theta = (1 \pm \sqrt{1 - 0.64})/0.8 = (1 \pm 0.6)/0.8$, giving 0.5 and 2. Their product is 1, as the section on moving averages predicts: $\theta$ and $1/\theta$ give the same autocorrelations. By Theorem 4, invertibility needs the root of $1 + \theta z$, which is $-1/\theta$, outside the unit circle, so $\lvert\theta\rvert < 1$: $\theta = 0.5$, with $Z_t = \sum_j(-0.5)^jX_{t-j}$. The value 0.4 is inside the bound $\lvert\rho(1)\rvert \le \frac12$ of Autocorrelation and Stationarity (its Worked problem 2), which is why real solutions exist.
Problem 3 (the Durbin–Levinson recursion for an AR(2)). For the AR(2) process $X_t = 0.5X_{t-1} + 0.3X_{t-2} + Z_t$, run the recursion for $n = 1, 2, 3$ using the autocorrelations, and confirm Theorem 8.
Solution. $\alpha(1) = 0.7143$, $\alpha(2) = 0.3$, $\alpha(3) = 0$. The Yule–Walker example gave $\rho(1) = 5/7 = 0.7143$, $\rho(2) = 23/35 = 0.6571$ and $\rho(3) = 0.5429$. The recursion can be run on autocorrelations, since dividing every $\gamma$ by $\gamma(0)$ divides each $v_n$ by $\gamma(0)$ and leaves the $\phi_{nj}$ unchanged. Step 1: $\phi_{11} = \rho(1) = 0.7143$, and the relative prediction variance is $1 - \rho(1)^2 = 0.4898$. Step 2: $\phi_{22} = (\rho(2) - \phi_{11}\rho(1))/(1 - \rho(1)^2) = (0.6571 - 0.5102)/0.4898 = 0.1469/0.4898 = 0.3$, exactly $\phi_2$, and $\phi_{21} = \phi_{11} - \phi_{22}\phi_{11} = 0.7143 \times 0.7 = 0.5$, exactly $\phi_1$. Step 3: the numerator of $\phi_{33}$ is $\rho(3) - 0.5\rho(2) - 0.3\rho(1)$, which is 0 by the Yule–Walker equation at $h = 3$. So $\alpha(3) = 0$, as Theorem 8 says. In units of $\gamma(0) = 2.2436$ the prediction variances are $v_1 = 2.2436 \times 0.4898 = 1.0989$ and $v_2 = 1.0989 \times (1 - 0.09) = 1$, the noise variance: once two past values are known, nothing more can be predicted.
Problem 4 (comparing two fitted models by AICc). The ARMA(3, 1) and ARMA(1, 1) models for Iowa have $-2\ln L$ = 2,493.62 and 2,499.45, with $K$ = 13 and 11 parameters (eight regression coefficients, the ARMA coefficients and $\sigma^2$), on $T = 261$ weeks. Compute AIC and AICc for both.
Solution. ARMA(3, 1): AIC $= 2{,}493.62 + 26 = 2{,}519.62$, and the correction is $2 \times 13 \times 14/(261 - 14) = 364/247 = 1.47$, so AICc $= 2{,}521.09$. ARMA(1, 1): AIC $= 2{,}499.45 + 22 = 2{,}521.45$, the correction $2 \times 11 \times 12/249 = 1.06$, and AICc $= 2{,}522.51$. The difference is 1.41 (1.42 from the rounded values shown). A difference that small says the two are about equally supported as approximations, as Akaike's Information Criterion, Derived explains; on the 2019 forecasts the simpler ARMA(1, 1) did slightly better.
Summary
- An AR(1) equation has a unique stationary solution, causal, when $\lvert\phi\rvert < 1$; a noncausal one when $\lvert\phi\rvert > 1$; none when $\lvert\phi\rvert = 1$.
- An ARMA model is causal exactly when the autoregressive polynomial has no root in the closed unit disk, and invertible when the moving average polynomial has none; MA(1) models with $\theta$ and $1/\theta$ are indistinguishable, and invertibility chooses $\lvert\theta\rvert < 1$.
- The autocorrelations of an MA($q$) cut off after $q$, the partial autocorrelations of an AR($p$) after $p$; ARMA autocovariances decay geometrically and satisfy the Yule–Walker equations.
- The best linear predictor solves the normal equations; the Durbin–Levinson recursion updates it in linear time and yields the partial autocorrelations.
- The least-squares and Yule–Walker estimates of an AR(1) are asymptotically $N(\phi, (1 - \phi^2)/n)$; in short series and near the unit root all estimators are biased toward zero, Yule–Walker most.
- Plug-in prediction intervals ignore parameter uncertainty, which matters little one step ahead and more at long horizons.
- For Iowa's weekly deaths, regression on a trend and an annual cycle with ARMA errors forecast 2019 better than the seasonal naive method (root mean squared error 28.4 to 28.7 against 38.6), and ARMA errors improved one-week-ahead forecasts modestly.
What we have learned
ARMA models turn the correlogram into a few parameters with a meaning: how much of this week's deviation carries into the next, and for how long. Causality and invertibility, conditions on polynomial roots, decide which parameter values describe a process built from past shocks and recoverable from past data. Prediction is projection, and its errors give honest intervals once the model is right, slightly too narrow once its parameters are estimated. For Iowa, most of the predictable variation in weekly deaths is the season and the trend; the ARMA errors add a little at short range. A different kind of dependence, in which the series switches between regimes such as epidemic and ordinary weeks, is modeled by a hidden Markov chain, the subject of Hidden Markov Models.
References
- Box, G. E. P., and Pierce, D. A. (1970). Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American Statistical Association, 65(332), 1509–1526. https://doi.org/10.1080/01621459.1970.10481180 record · 2 pages
- Brockwell, P. J., and Davis, R. A. (2016). Introduction to Time Series and Forecasting (3rd ed.). Springer Texts in Statistics. Springer. https://doi.org/10.1007/978-3-319-29854-2 record · 3 pages
- Durbin, J. (1960). The fitting of time-series models. Review of the International Statistical Institute, 28(3), 233 (first page as Crossref lists it). https://doi.org/10.2307/1401322 record
- Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press. Theorem 6.2.1 (the ergodic theorem), Example 6.1.5 and Exercise 6.1.5; numbers of the author's version of January 11, 2019. record · 15 pages
- Hannan, E. J. (1973). The asymptotic theory of linear time-series models. Journal of Applied Probability, 10(1), 130–145. https://doi.org/10.1017/S0021900200042145 record · 2 pages
- Hyndman, R., Athanasopoulos, G., Bergmeir, C., Caceres, G., Chhay, L., Kuroptev, K., Mücke, M., O'Hara-Wild, M., Petropoulos, F., Razbash, S., Wang, E., and Yasmeen, F. (2026). forecast: Forecasting Functions for Time Series and Linear Models. R package version 9.0.2, CRAN, published 18 March 2026. Source file
R/newarima2.R, functionmyarima, read through the CRAN mirror on GitHub (https://github.com/cran/forecast). record - Hyndman, R. J., and Athanasopoulos, G. (2021). Forecasting: Principles and Practice (3rd ed.). OTexts, Melbourne. https://otexts.com/fpp3/ (accessed November 2026) record · 2 pages
- Levinson, N. (1946). The Wiener (root mean square) error criterion in filter design and prediction. Journal of Mathematics and Physics, 25(1–4), 261–278 (as Crossref lists it; OpenAlex gives the journal's later name, Studies in Applied Mathematics). https://doi.org/10.1002/sapm1946251261 record
- National Center for Health Statistics (2021). Weekly Counts of Deaths by State and Select Causes, 2014–2019. Centers for Disease Control and Prevention, data.cdc.gov dataset 3yf8-kanr. https://data.cdc.gov/d/3yf8-kanr (accessed November 2026) record · 16 pages
- Walker, G. T. (1931). On periodicity in series of related terms. Proceedings of the Royal Society of London, Series A, 131(818), 518–532. https://doi.org/10.1098/rspa.1931.0069 record
- Yule, G. U. (1927). On a method of investigating periodicities disturbed series, with special reference to Wolfer's sunspot numbers (title as Crossref lists it). Philosophical Transactions of the Royal Society of London, Series A, 226, 267–298. https://doi.org/10.1098/rsta.1927.0007 record