Time Series Notes (4) - Model specification

Model-building strategy

Box-Jenkins method for model-building strategy

graph TD
  A([Start with a time series realization])
  --> B[Identify a preliminary time series model]
  --> C[Estimation of the model parameters]
  --> D{Is the model adequate?}
  D --> |No| B
  D --> |Yes| E([Stop])

Three steps of building models

1. Model specification. The model is tentative at this moment and it is from the observed part of the time series. The model also should be chosen under the principle of parsimony.
2. Model fitting.
3. Model diagnostics. If inadequacies found, another model ought be considered until an acceptable model is found.

Autocorrelation function (ACF)

For a $\text{MA}(q)$ process, it has the cut-off property, i.e. ${\rho }_q\ne 0$ but ${\rho }_k=0$ for $k>q$.

For a $\text{AR}(p)$ or $\text{ARMA}(p,q)$ process, it decays to zero exponentially, but does not enjoy the cut-off property.

Sample autocorrelation function

For a sequence of observations $Z_t,\dots ,Z_n$ the sample autocorrelation function (sample ACF) is defined as follows.

$$r_k=\frac{\widehat{{\gamma }_k}}{\widehat{{\gamma }_0}},k\ge 0$$

where $\widehat{{\gamma }_k}$ is the $k$-th sample autocovariance:

$$\widehat{{\gamma }_k}=\frac{1}{n}\sum _{t=k+1}^n(Z_t-\overline{Z})(Z_{t-k}-\overline{Z}),\overline{Z}=\frac{1}{n}\sum _{t=1}^n{Z}_t$$

Bartlett’s approximation

The 95% approximate confidence interval of $r_k$ hence will be ($r_k$ denotes sampled ${\rho }_k$ here)

$$\pm \frac{1.96}{\sqrt{n}}\sqrt{1+2r_1^2+\cdots +2r_{k-1}^2}$$

The above confidence interval acts as the accepting region for the test with $H_0:{\rho }_k=0$.

In real applications, the original condition of the Bartlett’s approximation is usually ignored. The squared part are usually ignored. And the value of 1.96, the corresponding 95% percentile of the standard normality, is replaced by 2.0 in R.

Partial autocorrelation function (PACF)

Definition of partial autocorrelation

The partial autocorrelation (PACF) at lag $k$, denoted by ${\varphi }_{kk}$, is defined as the solution to the system of equations:

$$\left(\begin{array}{cccc}1& {\rho }_1& \cdots & {\rho }_{k-1}\\ {\rho }_1& 1& \cdots & {\rho }_{k-2}\\ ⋮& ⋮& \ddots & ⋮\\ {\rho }_{k-1}& {\rho }_{k-2}& \cdots & 1\end{array}\right)\left(\begin{array}{c}{\varphi }_{k1}\\ {\varphi }_{k2}\\ ⋮\\ {\varphi }_{kk}\end{array}\right)=\left(\begin{array}{c}{\rho }_1\\ {\rho }_2\\ ⋮\\ {\rho }_k\end{array}\right)$$

Or, in other words, the equations are equivalent:

$$\left(\begin{array}{c}{\varphi }_{k1}\\ {\varphi }_{k2}\\ ⋮\\ {\varphi }_{kk}\end{array}\right)={\left(\begin{array}{cccc}1& {\rho }_1& \cdots & {\rho }_{k-1}\\ {\rho }_1& 1& \cdots & {\rho }_{k-2}\\ ⋮& ⋮& \ddots & ⋮\\ {\rho }_{k-1}& {\rho }_{k-2}& \cdots & 1\end{array}\right)}^{-1}\left(\begin{array}{c}{\rho }_1\\ {\rho }_2\\ ⋮\\ {\rho }_k\end{array}\right)$$

where $Z_t$ is a stationary time series with autocorrelation functions ${\rho }_m,m=1,2,3,\dots$

In particular:

1. $k=1$: ${\varphi }_{11}={\rho }_1$
2. $k=2$: we have

$$\left(\begin{array}{c}{\varphi }_{21}\\ {\varphi }_{22}\end{array}\right)={\left(\begin{array}{cc}1& {\rho }_1\\ {\rho }_1& 1\end{array}\right)}^{-1}\left(\begin{array}{c}{\rho }_1\\ {\rho }_2\end{array}\right)$$

Thus,

$${\varphi }_{22}=\frac{{\rho }_2-{\rho }_1}{1-{\rho }_1^2}$$

The PACF is to measure the direct impact from $Z_{t-k}$ at $Z_t$, with removing the middle terms from $Z_{t-1},\dots ,Z_{t-k+1}$. So for $\text{AR}(p)$ process, the PACF has a cut-off: ${\varphi }_{kk}=0$ for $k>p$. This property mirrors the cut-off property of the ACF of an $\text{MA}(q)$ process: ${\rho }_k=0$ for $k>q$.

For $\text{MA}(1)$ process $Z_t=a_t+\theta a_{t-1}$, we have

$${\varphi }_{kk}=\frac{(-1{)}^{k+1}{\theta }^k}{1+{\theta }^2+{\theta }^4+\cdots +{\theta }^{2k}},\text{ for }k\ge 1$$

1. Notice that the partial autocorrelations of an $\text{MA}(1)$ model are never zero. Except the special case of ${\theta }^2=1$. They decay exponentially to zero, rather like the autocorrelation for an $\text{AR}(1)$ series.
2. The behaviors of PACF for ARMA processes is quite similar to that for MA processes.

Sample PACF

For an observed time series, an obvious method is to estimate the $\rho$’s by $r$’s and then to solve the Yule-Walker equations to get estimates of ${\varphi }_{kk}$, i.e.,

$$\left(\begin{array}{c}{\hat{\varphi }}_{k1}\\ {\hat{\varphi }}_{k2}\\ ⋮\\ {\hat{\varphi }}_{kk}\end{array}\right)={\left(\begin{array}{cccc}1& r_1& \cdots & r_{k-1}\\ r_1& 1& \cdots & r_{k-2}\\ ⋮& ⋮& \ddots & ⋮\\ r_{k-1}& r_{k-2}& \cdots & 1\end{array}\right)}^{-1}\left(\begin{array}{c}{r}_1\\ r_2\\ ⋮\\ r_k\end{array}\right)$$

Inverse autocorrelation function (IACF) and the sample IACF

1. The sample IACF plays much the same role in ARIMA modeling as the sample PACF, but it generally indicates subset and seasonal autoregressive models better than the sample PACF.
2. Additionally, the sample IACF can be useful for detecting over-differencing.
3. The IACF is defined in the frequency domain. It is not often discussed in textbooks.

Extended autocorrelation function (EACF) and the sample EACF

1. It can be used to identify the values of $p+d$ and $q$ for an $\text{ARIMA}(p,d,q)$.
2. It is not a very popular method.
3. There are other similar methods in the literature such as the corner method, the smallest canonical correlation (SCAN) method, etc..

Nonstationarity

How to determine the value of $d$ in $\text{ARIMA}(p,d,q)$ models

1. To determine the order of $d$ is equivalent to determine whether or not a sequence of values are from a stationary time series.
2. The definition of the sample ACF implicitly assumes stationarity.
3. However, for a nonstationary series, the sample ACF typically decays slowly as the lags increase.
4. If the sample ACF decays slowly, we may say the sequence is nonstationary. Otherwise, it is stationary.

Overdifferencing

If a time series $Z_t$ is stationary, then the differenced sequence $\mathrm{\nabla }{Z}_t$ is still stationary. Thus, we sometimes tend to overdifference the sequence.

To avoid overdifferencing, we should look carefully at each difference in succession and always keep in mind the principle of parsimony – models should be as simple as possible.

Other specification methods

Akaike information criterion (AIC)

This criterion is to select the model that minimizes

$$\text{AIC}(k)=-2\log (\text{maximum likelihood})+2k$$

where $k$ is the number of parameters. For an $\text{ARMA}(p,q)$ model, $k=p+q+1$ when the model contains an intercept, and $p+q$ otherwise.

Bayesian information criterion (BIC)

This criterion is to select the model that minimizes

$$\text{BIC}(k)=-2\log (\text{maximum likelihood})+k\log (n)$$

The AIC is more popular in real applications although the BIC has a well justified theoretical basis.