$$ \newcommand{\initvar}{v_0^2} \newcommand{\cd}{\overset{d}{\to}} \newcommand{\var}{\text{var}} \newcommand{\cov}{\text{cov}} \newcommand{\argmax}[1]{\underset{#1}{\arg\max}} \newcommand{\argmin}[1]{\underset{#1}{\arg\min}} \newcommand{\iid}{\overset{\textrm{iid}}{\sim}} \newcommand{\indep}{\overset{\textrm{indep}}{\sim}} \newcommand{\tr}{{\scriptscriptstyle\mathsf{T}}} \newcommand{\com}{,\,} \newcommand{\given}{\,|\,} \newcommand{\acal}{\mathcal{A}} \newcommand{\bcal}{\mathcal{B}} \newcommand{\ccal}{\mathcal{C}} \newcommand{\dcal}{\mathcal{D}} \newcommand{\ecal}{\mathcal{E}} \newcommand{\fcal}{\mathcal{F}} \newcommand{\gcal}{\mathcal{G}} \newcommand{\hcal}{\mathcal{H}} \newcommand{\ical}{\mathcal{I}} \newcommand{\jcal}{\mathcal{J}} \newcommand{\kcal}{\mathcal{K}} \newcommand{\lcal}{\mathcal{L}} \newcommand{\mcal}{\mathcal{M}} \newcommand{\ncal}{\mathcal{N}} \newcommand{\ocal}{\mathcal{O}} \newcommand{\pcal}{\mathcal{P}} \newcommand{\qcal}{\mathcal{Q}} \newcommand{\rcal}{\mathcal{R}} \newcommand{\scal}{\mathcal{S}} \newcommand{\tcal}{\mathcal{T}} \newcommand{\ucal}{\mathcal{U}} \newcommand{\vcal}{\mathcal{V}} \newcommand{\wcal}{\mathcal{W}} \newcommand{\xcal}{\mathcal{X}} \newcommand{\ycal}{\mathcal{Y}} \newcommand{\zcal}{\mathcal{Z}} \newcommand{\EE}{\mathbb{E}} \newcommand{\NN}{\mathbb{N}} \newcommand{\PP}{\mathbb{P}} \newcommand{\QQ}{\mathbb{Q}} \newcommand{\RR}{\mathbb{R}} \newcommand{\ZZ}{\mathbb{Z}} \newcommand{\BA}{\mathbf{A}} \newcommand{\BB}{\mathbf{B}} \newcommand{\BC}{\mathbf{C}} \newcommand{\BD}{\mathbf{D}} \newcommand{\BE}{\mathbf{E}} \newcommand{\BF}{\mathbf{F}} \newcommand{\BG}{\mathbf{G}} \newcommand{\BH}{\mathbf{H}} \newcommand{\BI}{\mathbf{I}} \newcommand{\BJ}{\mathbf{J}} \newcommand{\BK}{\mathbf{K}} \newcommand{\BL}{\mathbf{L}} \newcommand{\BM}{\mathbf{M}} \newcommand{\BN}{\mathbf{N}} \newcommand{\BO}{\mathbf{O}} \newcommand{\BP}{\mathbf{P}} \newcommand{\BQ}{\mathbf{Q}} \newcommand{\BR}{\mathbf{R}} \newcommand{\BS}{\mathbf{S}} \newcommand{\BT}{\mathbf{T}} \newcommand{\BU}{\mathbf{U}} \newcommand{\BV}{\mathbf{V}} \newcommand{\BW}{\mathbf{W}} \newcommand{\BX}{\mathbf{X}} \newcommand{\BY}{\mathbf{Y}} \newcommand{\BZ}{\mathbf{Z}} \newcommand{\Ba}{\mathbf{a}} \newcommand{\Bb}{\mathbf{b}} \newcommand{\Bc}{\mathbf{c}} \newcommand{\Bd}{\mathbf{d}} \newcommand{\Be}{\mathbf{e}} \newcommand{\Bf}{\mathbf{f}} \newcommand{\Bg}{\mathbf{g}} \newcommand{\Bh}{\mathbf{h}} \newcommand{\Bi}{\mathbf{i}} \newcommand{\Bj}{\mathbf{j}} \newcommand{\Bk}{\mathbf{k}} \newcommand{\Bl}{\mathbf{l}} \newcommand{\Bm}{\mathbf{m}} \newcommand{\Bn}{\mathbf{n}} \newcommand{\Bo}{\mathbf{o}} \newcommand{\Bp}{\mathbf{p}} \newcommand{\Bq}{\mathbf{q}} \newcommand{\Br}{\mathbf{r}} \newcommand{\Bs}{\mathbf{s}} \newcommand{\Bt}{\mathbf{t}} \newcommand{\Bu}{\mathbf{u}} \newcommand{\Bv}{\mathbf{v}} \newcommand{\Bw}{\mathbf{w}} \newcommand{\Bx}{\mathbf{x}} \newcommand{\By}{\mathbf{y}} \newcommand{\Bz}{\mathbf{z}} \newcommand{\Balpha}{\boldsymbol{\alpha}} \newcommand{\Bmu}{\boldsymbol{\mu}} \newcommand{\Btheta}{\boldsymbol{\theta}} \newcommand{\Beta}{\boldsymbol{\eta}} \newcommand{\Bepsilon}{\boldsymbol{\varepsilon}} \newcommand{\Bbeta}{\boldsymbol{\beta}} \newcommand{\BLambda}{\boldsymbol{\Lambda}} \newcommand{\BSigma}{\boldsymbol{\Sigma}} \newcommand{\BOmega}{\boldsymbol{\Omega}} \newcommand{\BPsi}{\boldsymbol{\Psi}} \newcommand{\BGamma}{\boldsymbol{\Gamma}} \newcommand{\BPhi}{\boldsymbol{\Phi}} \newcommand{\Bphi}{\boldsymbol{\phi}} \newcommand{\BUpsilon}{\boldsymbol{\Upsilon}} \newcommand{\BXi}{\boldsymbol{\Xi}} \newcommand{\Bpi}{\boldsymbol{\pi}} \newcommand{\Bzero}{\mathbf{0}} \newcommand{\Bone}{\mathbf{1}} \newcommand{\N}{\text{N}} \newcommand{\dd}{\text{d}} $$

Autoregressive models

Ed Iversen and John Zito

Duke University
STA 440 Spring 2026

Introduction

Where we’re going

  • We’ll introduce the most basic time series models;
    • autoregression (AR);
    • dynamic linear model (DLM);
  • You may or may not use them;
  • The basic models will provide a sandbox for illustrating probabilistic prediction and evaluation;
    • You will definitely use this!

Time series

  • A time series is a set of measurements collected over time;

  • We model these data as a sequence of dependent random variables:

\[ \mathbf{y}_{0:T} = \{\mathbf{y}_0,\,\mathbf{y}_1,\,\mathbf{y}_2,\,...,\,\mathbf{y}_T\}. \]

  • A time series model is “just” their joint probability distribution:

\[ p(\mathbf{y}_{0:T}) = p(\mathbf{y}_0)\prod_{t=1}^Tp(\mathbf{y}_t\,|\,\mathbf{y}_{0:t-1}). \]

Stay grounded.

Like much wisdom, that last bullet is simultaneously vacuous and profound. It tells you everything and it tells you nothing all at once. But don’t let this basic fact get lost in the sea of details.

Notation to get used to

  • I will not use uppercase \(Y_t\) versus lowercase \(y_t\) to distinguish random variables and fixed realizations. It’s all just \(y_t\), and context makes clear how it functions;
  • A vector \(\mathbf{y}\in\mathbb{R}^n\) is always an \(n\times 1\) column. The corresponding row vector is \(\By^\tr\);
  • For integers \(i<j\), you will see this shorthand all the time:

\[ y_{i:j} = \{y_i\com y_{i+1}\com y_{i+2}\com...\com y_{j-2}\com y_{j-1}\com y_{j}\} . \]

  • The symbol “\(p\)” will be aggressively abused and overloaded to represent any probability distribution, sometimes several in the same line:

\[ p(\mathbf{y}_{0:T}) = p(\mathbf{y}_0)\prod_{t=1}^Tp(\mathbf{y}_t\,|\,\mathbf{y}_{0:t-1}). \]

Starting small

The simplest non-trivial time series model

The autoregression of order 1, or AR(1):

\[ \begin{aligned} y_t &= \beta_0 + \beta_1 y_{t-1} + \varepsilon_t, \quad \varepsilon_t\iid\text{N}(0\com\sigma^2) . \end{aligned} \]

That’s the recursive form. It implies a joint distribution written marginal-conditional style:

\[ \begin{aligned} p(y_{1:T}\mid y_0) &= \prod_{t=1}^T p(y_t\given y_{0:t-1}) \\ &= \prod_{t=1}^T \underbrace{p(y_t\given y_{t-1})}_{\N(\beta_0+\beta_1y_{t-1}\com\sigma^2)} \end{aligned} \]

The implied joint distribution

Because the model is linear and Gaussian, the implied joint distribution across time is just a big ol’ multivariate normal:

\[ \begin{bmatrix} y_1 & y_2 & \cdots & y_T \end{bmatrix}^\tr\mid y_0 \sim\text{N}_{T}\left(\Bmu\com \BSigma\right). \]

Just like the linear mixed model is “just” a big multivariate normal with a particular block covariance structure (dependent within and independent between groups).

Don’t worry about this!

The mean and covariance are given by:

\[ \begin{aligned} E(y_t\mid y_0) &= \beta_0\sum\limits_{i=0}^{t-1}\beta_1^i + \beta_1^ty_0 \\ \var(y_t\mid y_0) &= \sigma^2 \sum\limits_{i=0}^{t-1}\beta_1^{2i} \\ \cov(y_t\com y_s\mid y_0) &= \begin{cases} \beta_1^{s-t}\var(y_t) & t\leq s\\ \beta_1^{t-s}\var(y_s) & s < t. \end{cases} . \end{aligned} \]

What does this look like?

#| '!! shinylive warning !!': |
#|   shinylive does not work in self-contained HTML documents.
#|   Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 700

library(shiny)

simulate_ar_1 <- function(T, b0, b1, s, m0, s0){
  y <- numeric(T)
  y[1] <- rnorm(1, m0, s0)
  for(t in 2:T){
    y[t] <- b0 + b1 * y[t - 1] + rnorm(1, 0, s)
  }
  return(y)
}

ar_1_mean <- function(t, b0, b1, m0){
  if(t == 0){
    return(m0)
  }else{
    return(b0 * sum(b1 ^ (0:(t-1))) + m0 * (b1^t)) 
  }
}

ar_1_var <- function(t, b1, s, s0){
  if(t == 0){
    return(s0^2)
  }else{
    return((s0^2) * (b1^(2*t)) + (s^2) * sum(b1 ^ (2*(0:(t-1)))))
  }
}

ar_1_sd <- function(t, b1, s, s0){
  sqrt(ar_1_var(t, b1, s, s0))
}

# Define UI for application that draws a histogram
ui <- fluidPage(
  
  # Application title
  titlePanel("Marginal distributions and sample paths of a Gaussian AR(1)"),
  
  # Sidebar with a slider input for number of bins 
  sidebarLayout(
    sidebarPanel(
      sliderInput("b0",
                  "β₀",
                  min = -5,
                  max = 5,
                  value = 0,
                  step = 0.1),
      sliderInput("b1",
                  "β₁",
                  min = -2,
                  max = 2,
                  value = 0,
                  step = 0.1),
      sliderInput("s",
                  "σ",
                  min = 0,
                  max = 2,
                  value = 1, 
                  step = 0.1),
      sliderInput("m0",
                  "y₀",
                  min = -5,
                  max = 5,
                  value = 0,
                  step = 0.1),
      sliderInput("T",
                  "T",
                  min = 20,
                  max = 200,
                  step = 20,
                  value = 100),
      actionButton("redo", "New sample path"),
    ),
    
    # Show a plot of the generated distribution
    mainPanel(
      plotOutput("distPlot", height = "600px")
    )
  )
)

# Define server logic required to draw a histogram
server <- function(input, output) {
  
  output$distPlot <- renderPlot({
    input$redo
    b0 <- input$b0
    b1 <- input$b1
    redo <- input$redo
    T <- input$T
    s <- input$s
    m0 <- input$m0
    s0 = 0
    
    range = 0:T
    alpha = c(0.01, seq(0.1, 0.9, by = 0.1))
    
    middle <- sapply(range, ar_1_mean, b0, b1, m0)
    sds <- sapply(range, ar_1_sd, b1, s, s0)
    
    
    plot(range, middle, type = "l",
         xaxt = "n", 
         yaxt = "n",
         xlab = "t",
         ylab = expression(y[t]),
         ylim = c(-20, 20), bty = "n",
         col = "white")
    
    for(a in alpha){
      
      U = qnorm(1 - a / 2, mean = middle, sd = sds)
      L = qnorm(a / 2, mean = middle, sd = sds)
      
      polygon(
        c(range, rev(range)),
        c(U, rev(L)),
        col = rgb(1, 0, 0, 0.15),
        border = NA
      )
    }
    
    inc = 20
    axis(1, pos = 0, at = seq(0, max(range), by = inc), 
         labels = c(NA, seq(inc, max(range), by = inc)))
    axis(2, pos = 0)
    
    lines(range, simulate_ar_1(max(range) + 1, b0, b1, s, m0, s0), col = "black", lwd = 2)
  })
}

# Run the application 
shinyApp(ui = ui, server = server)

Observations

  • \(|\beta_1|<1\): stable;
  • \(|\beta_1|>1\): explosive;
  • \(\beta_1<0\): oscillation;
  • \(|\beta_1|=1\): random walk (w/ drift if \(\beta_0\neq0\)).

What do we mean by stable?

  • A joint distribution is (strictly) stationary if it is “shift invariant”:

\[ \{y_{t_1}\com y_{t_2}\com ...\com y_{t_n}\}\overset{d}{=}\{y_{t_1+h}\com y_{t_2+h}\com ...\com y_{t_n+h}\}. \]

  • The joint distribution is the same anywhere you choose to zoom in;

  • The Gaussian AR(1) with \(|\beta_1|<1\) has this property.

Stationary AR(1)

If \(-1<\beta_1<1\) and \(y_0=\beta_0/(1-\beta_1)\), then the AR(1) is strictly stationary with the following:

\[ \begin{aligned} E(y_t) &= \frac{\beta_0}{1-\beta_1} \\ \var(y_t) &= \frac{\sigma^2}{1-\beta_1^2} \\ \cov(y_t\com y_s) &= \beta_1^{|t-s|}\var(y_t) = \beta_1^{|t-s|}\frac{\sigma^2}{1-\beta_1^2}. \end{aligned} \]

The common marginal shared by all \(y_t\) is called the stationary distribution:

\[ y_t\sim\text{N}\left(\frac{\beta_0}{1-\beta_1}\com \frac{\sigma^2}{1-\beta_1^2}\right). \]

So “did: dependent but identically distributed.”

Autocovariance of a stationary process

For a stationary process, the covariance kernel satisfies

\[ \cov(y_t\com y_{s})=\cov(y_{t+h}\com y_{s+h})\quad \forall (t\com s\com h). \]

So you can define something called the autocovariance function:

\[ \gamma(h)=\cov(y_{t+h}\com y_{t}). \]

For the AR(1), this is

\[ \begin{aligned} \gamma(0)&=\sigma^2/(1-\beta_1^2) \\ \gamma(h)&=\beta_1^h\gamma(0). \end{aligned} \]

What does the autocov function look like?

#| '!! shinylive warning !!': |
#|   shinylive does not work in self-contained HTML documents.
#|   Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 700

library(shiny)

# Define UI for application that draws a histogram
ui <- fluidPage(
  
  # Application title
  titlePanel("Theoretical autocovariance of a stationary AR(1)"),
  
  # Sidebar with a slider input for number of bins 
  sidebarLayout(
    sidebarPanel(
      sliderInput("b1",
                  "β₁",
                  min = -0.99,
                  max = 0.99,
                  value = 0,
                  step = 0.01),
      sliderInput("s",
                  "σ",
                  min = 0,
                  max = 2,
                  value = 1, 
                  step = 0.1),
      sliderInput("H",
                  "H",
                  min = 0,
                  max = 50,
                  step = 1,
                  value = 20)
    ),
    
    # Show a plot of the generated distribution
    mainPanel(
      plotOutput("distPlot", height = "500px")
    )
  )
)

# Define server logic required to draw a histogram
server <- function(input, output) {
  
  output$distPlot <- renderPlot({
    beta_1 <- input$b1
    max_lag <- input$H
    sigma2 <- input$s^2

    # Compute theoretical autocovariances
    lags <- 0:max_lag
    gamma_h <- sigma2 / (1 - beta_1^2) * beta_1^lags
    
    # Plot
    plot(lags, gamma_h, type="h", lwd=2,
         xlab="Lag h", ylab="γ(h)",
         main="")
    points(lags, gamma_h, pch=19, col="blue")
    abline(h = 0, lty = 2, col = "darkgrey")
    
    # Add legend using Unicode
    legend_text <- paste0("β1 = ", beta_1, ", σ² = ", sigma2)
    legend("topright", legend=legend_text, bty="n")

  })
}

# Run the application 
shinyApp(ui = ui, server = server)

Observations

The autocovariance of a stationary AR(1) is:

\[ \begin{aligned} \gamma(0)&=\sigma^2/(1-\beta_1^2) \\ \gamma(h)&=\beta_1^h\gamma(0). \end{aligned} \]

  • \(\gamma(h)\to0\) as \(h\to\infty\);
  • The larger \(|\beta_1|\), the stronger the serial dependence, the slower the decay;
  • If \(-1<\beta_1<0\), you get oscillation because \(\beta_1^h\) is positive for even \(h\) and negative for odd \(h\).

There’s nothing special about one lag

The autoregression of order \(p\), or AR(\(p\)):

\[ \begin{aligned} y_t &= \beta_0 + \beta_1 y_{t-1} + \beta_2 y_{t-2} + \cdots + \beta_p y_{t-p} + \varepsilon_t, \quad \varepsilon_t\iid\text{N}(0\com\sigma^2) \end{aligned} \]

This again implies a joint distribution written marginal-conditional style:

\[ \begin{aligned} p(y_{1:T}\mid y_0\com y_{-1}\com ...\com y_{1-p}) &= \prod_{t=1}^Tp(y_t\mid y_{1-p:t-1})\\ \\ &= \prod_{t=1}^Tp(y_t\mid y_{t-p:t-1}). \end{aligned} \]

As before, the joint distribution is just a big ol’ multivariate normal, but the mean and covariance are harder to characterize.

Likelihood-based inference

Let’s improve the notation

The autoregression of order p, or AR(p):

\[ \begin{aligned} y_t &= \beta_0 + \sum\limits_{\ell=1}^p \beta_\ell y_{t-\ell} + \varepsilon_t, \quad \varepsilon_t\iid\text{N}(0\com\sigma^2). \end{aligned} \]

This implies a joint distribution governed by a finite set of static parameters \(\Btheta = \begin{bmatrix}\beta_0&\beta_1&\cdots &\beta_p&\sigma^2\end{bmatrix}^\tr\):

\[ \begin{aligned} p(y_{1:T}\given y_{1-p:0}\com\Btheta) &= \prod_{t=1}^T p(y_t\given y_{t-p:t-1}\com\Btheta). \end{aligned} \]

Viewed as a function of \(\Btheta\), that’s a (conditional) likelihood!

Maximum likelihood estimation

Treating the observed data \(y_{1-p:T}\) as fixed, we want:

\[ \hat{\Btheta}_T=\argmax{\Btheta}\,p(y_{1:T}\given y_{1-p:0}\com \Btheta). \]

To do this, it helps to view the AR(p) is “just” a multiple linear regression where \(\Bx_t\) encodes the lagged values of \(y_t\):

\[ \begin{aligned} y_t&=\Bx_t^\tr\Bbeta+\varepsilon_t,&&\varepsilon_t\iid\N(0\com\sigma^2) \\ \Bx_t&=\begin{bmatrix} 1 & y_{t-1} & y_{t-2} & \cdots & y_{t-p} \end{bmatrix}^\tr\\ \Bbeta&=\begin{bmatrix} \beta_0 & \beta_1 & \beta_2 & \cdots & \beta_p \end{bmatrix}^\tr. \end{aligned} \]

Stack ’em up

Construct the response vector and design matrix:

\[ \begin{aligned} \underbrace{\By_T}_{T\times 1} &= \begin{bmatrix}y_1&y_2 & \cdots & y_T\end{bmatrix}^\tr \\ \underbrace{\BX_T}_{T\times (p+1)} &= \begin{bmatrix} 1 & y_0 & y_{-1} & \cdots & y_{1-p} \\ 1 & y_1 & y_0 & \cdots & y_{2-p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & y_{T-1} & y_{T-2} & \cdots & y_{T-p} \end{bmatrix} = \begin{bmatrix} \Bx_1^\tr\\ \Bx_2^\tr\\ \vdots\\ \Bx_T^\tr \end{bmatrix} . \end{aligned} \]

OLS for the AR(p)

  • Turns out, the maximum (conditional) likelihood estimator in the AR(p) is the same as the ordinary least squares estimator:

\[ \begin{aligned} \hat{\Bbeta}_T &= (\BX_T^\tr\BX_T)^{-1}\BX_T^\tr\By_T \\ \hat{\sigma^2_T} &= ||\By_T-\BX_T\hat{\Bbeta}_T||_2^2 / T. \end{aligned} \]

  • It doesn’t matter that there’s time series dependence. Once the data are observed and fixed, the mechanics of computing the estimate are identical to iid multiple regression;
  • The analogy breaks down after that: different sampling distributions, intervals, etc. No exact procedures. Need bootstrap, etc.

Bayes is just like iid regression too

Bayesian model with a conjugate prior:

\[ \begin{aligned} \sigma^2 &\sim \text{IG}(a_0\com b_0) \\ \Bbeta\given \sigma^2 &\sim \text{N}_{p+1}(\Bm_0\com\sigma^2\BH^{-1}_0) \\ y_t \given \Bx_t \com \Bbeta\com\sigma^2 &\sim \text{N} \left( \Bx_t^\tr\Bbeta\com\sigma^2 \right). \end{aligned} \]

The posterior is available in closed-form:

\[ \begin{aligned} \sigma^2\given y_{0:T} &\sim \text{IG}(a_T\com b_T) \\ \Bbeta\given \sigma^2\com y_{0:T} &\sim \text{N}_{p+2}(\Bm_T\com\sigma^2\BH^{-1}_T) \\ \\ \BH_T &= \BX_T^\tr\BX_T+\BH_0 \\ \Bm_T &= \BH_T^{-1}(\BX_T^\tr\By_T+\BH_0\Bm_0) \\ a_T &= a_0 + T/2 \\ b_T &= b_0 + (\By_T^\tr\By_T+\Bm_0^\tr\BH_0\Bm_0-\Bm_T^\tr\BH_T\Bm_T)/2. \end{aligned} \]

Hyperparameter tuning

Lag order selection

The lag order \(p\) is a hyperparameter that needs to be selected. \(p\) determines the number of predictors in the (auto)regression model, and so selecting the lag order is analogous to variable selection in iid regression.

Rules of thumb

Pick \(p\) = number of observations in one natural cycle of the data:

Data frequency Typical lag length (p)
Hourly 24
Daily 7
Monthly 12
Quarterly 4
Annual 1

Fine as a quick-and-dirty solution. Guidance on this will be domain-specific.

Minimize some criterion

Take:

\[ \hat{p}=\argmin{p=1\com2\com...\com\bar{p}}\,\text{C}(p;\,\By_T\com\BX_T). \]

Common choices:

  • Akaike information criterion (AIC);
  • Bayesian information criterion (BIC);
  • Cross-validation error;

CV assesses out-of-sample performance and is highly customizable, but can be more labor-intensive to implement.

Recall: penalized iid regression

Penalized estimator for the linear model:

\[ \hat{\Bbeta}_{\lambda} = \underset{\Bbeta}{\arg\min}\left\{||\By-\BX\Bbeta||_2^2+\lambda\mathcal{P}(\Bbeta)\right\}. \] The penalty provides shrinkage or regularization, and the tuning parameter \(\lambda>0\) governs how much:

  • Ridge regression: \(\mathcal{P}(\Bbeta)=||\Bbeta||_2^2\);
  • LASSO: \(\mathcal{P}(\Bbeta)=||\Bbeta||_1\);
  • etc.

How do you tune \(\lambda\)?

Recall: LOO-CV for iid regression

Leave-one-out cross-validation:

  • For each model configuration, train on all data except observation \(i\), and then test on held-out \((\Bx_i\com y_i)\);
  • Average prediction error over all training/test splits, and pick the best model:

\[ \hat{\lambda}=\argmin{\lambda\geq0}\,\sum\limits_{i=1}^n\left(y_i-\Bx_i^\tr\hat{\Bbeta}_\lambda^{(-i)}\right)^2 . \]

  • Optimizes out-of-sample point prediction accuracy;
  • Fine for iid, but with serially dependent time series data, it’s not appropriate to randomly rip observations out of the middle and fit a model to what’s left before and after. Gotta respect the time-ordering.

Time series cross-validation

Leave-future-out cross-validation (LFO-CV):

\[ \begin{aligned} &{\color{blue}\bullet\,\text{training}}\quad{\color{red}\bullet\,\text{test}}\\ &\begin{matrix} \color{blue}{y_1} & \color{lightgray}{y_2} & \color{lightgray}{y_3} & \color{red}{y_4} & \color{lightgray}{y_5} & \color{lightgray}{y_6} & \color{lightgray}{y_7} & \color{lightgray}{y_8} & \color{lightgray}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_1^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{lightgray}{y_3} & \color{lightgray}{y_4} & \color{red}{y_5} & \color{lightgray}{y_6} & \color{lightgray}{y_7} & \color{lightgray}{y_8} & \color{lightgray}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_2^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{blue}{y_3} & \color{lightgray}{y_4} & \color{lightgray}{y_5} & \color{red}{y_6} & \color{lightgray}{y_7} & \color{lightgray}{y_8} & \color{lightgray}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_3^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{blue}{y_3} & \color{blue}{y_4} & \color{lightgray}{y_5} & \color{lightgray}{y_6} & \color{red}{y_7} & \color{lightgray}{y_8} & \color{lightgray}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_4^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{blue}{y_3} & \color{blue}{y_4} & \color{blue}{y_5} & \color{lightgray}{y_6} & \color{lightgray}{y_7} & \color{red}{y_8} & \color{lightgray}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_5^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{blue}{y_3} & \color{blue}{y_4} & \color{blue}{y_5} & \color{blue}{y_6} & \color{lightgray}{y_7} & \color{lightgray}{y_8} & \color{red}{y_9} & \color{lightgray}{y_{10}}& \cdots & \longrightarrow & \text{error}_6^{(p)} \\ \color{blue}{y_1} & \color{blue}{y_2} & \color{blue}{y_3} & \color{blue}{y_4} & \color{blue}{y_5} & \color{blue}{y_6} & \color{blue}{y_7} & \color{lightgray}{y_8} & \color{lightgray}{y_9} & \color{red}{y_{10}}& \cdots & \longrightarrow & \text{error}_7^{(p)}\\ &&&&&\vdots&&&&&&&\vdots\\ \end{matrix} \end{aligned} \]

Then pick:

\[ \hat{p}=\argmin{p=1\com2\com...\com\bar{p}}\,\sum\limits_{t=1}^T\text{error}_t^{(p)} \]

Cross-validation

Virtues:

  • out-of-sample: we care most about how methods perform on data we haven’t seen yet. CV mimics that directly;
  • customizable: whatever prediction task you care about (8-step-ahead point prediction, interval prediction, density, etc), simply plug-in the error metric that matches;
  • method-agnostic: once we have the predictions, it doesn’t matter where they came from. CV provides an apples-to-apples comparison across wildly differing procedures;

Vices:

High computational overhead to actually implement. Ideally, you fully re-train the entire procedure on every possible split. This can be very costly. So then what? Parallelize over splits? Only partially re-train? Randomly sample a subset of the splits and accept a noisier estimate of the CV error?

Sacrifices will have to be made somewhere.