Like much wisdom, that last bullet is simultaneously vacuous and profound. It tells you everything and it tells you nothing all at once. But don’t let this basic fact get lost in the sea of details.
Notation to get used to
I will not use uppercase \(Y_t\) versus lowercase \(y_t\) to distinguish random variables and fixed realizations. It’s all just \(y_t\), and context makes clear how it functions;
A vector \(\mathbf{y}\in\mathbb{R}^n\) is always an \(n\times 1\) column. The corresponding row vector is \(\By^\tr\);
For integers \(i<j\), you will see this shorthand all the time:
Just like the linear mixed model is “just” a big multivariate normal with a particular block covariance structure (dependent within and independent between groups).
#| '!! shinylive warning !!': |#| shinylive does not work in self-contained HTML documents.#| Please set `embed-resources: false` in your metadata.#| standalone: true#| viewerHeight: 700library(shiny)# Define UI for application that draws a histogramui <- fluidPage( # Application title titlePanel("Theoretical autocovariance of a stationary AR(1)"), # Sidebar with a slider input for number of bins sidebarLayout( sidebarPanel( sliderInput("b1", "β₁", min = -0.99, max = 0.99, value = 0, step = 0.01), sliderInput("s", "σ", min = 0, max = 2, value = 1, step = 0.1), sliderInput("H", "H", min = 0, max = 50, step = 1, value = 20) ), # Show a plot of the generated distribution mainPanel( plotOutput("distPlot", height = "500px") ) ))# Define server logic required to draw a histogramserver <- function(input, output) { output$distPlot <- renderPlot({ beta_1 <- input$b1 max_lag <- input$H sigma2 <- input$s^2 # Compute theoretical autocovariances lags <- 0:max_lag gamma_h <- sigma2 / (1 - beta_1^2) * beta_1^lags # Plot plot(lags, gamma_h, type="h", lwd=2, xlab="Lag h", ylab="γ(h)", main="") points(lags, gamma_h, pch=19, col="blue") abline(h = 0, lty = 2, col = "darkgrey") # Add legend using Unicode legend_text <- paste0("β1 = ", beta_1, ", σ² = ", sigma2) legend("topright", legend=legend_text, bty="n") })}# Run the application shinyApp(ui = ui, server = server)
This implies a joint distribution governed by a finite set of static parameters \(\Btheta = \begin{bmatrix}\beta_0&\beta_1&\cdots &\beta_p&\sigma^2\end{bmatrix}^\tr\):
It doesn’t matter that there’s time series dependence. Once the data are observed and fixed, the mechanics of computing the estimate are identical to iid multiple regression;
The analogy breaks down after that: different sampling distributions, intervals, etc. No exact procedures. Need bootstrap, etc.
The lag order \(p\) is a hyperparameter that needs to be selected. \(p\) determines the number of predictors in the (auto)regression model, and so selecting the lag order is analogous to variable selection in iid regression.
Rules of thumb
Pick \(p\) = number of observations in one natural cycle of the data:
Data frequency
Typical lag length (p)
Hourly
24
Daily
7
Monthly
12
Quarterly
4
Annual
1
Fine as a quick-and-dirty solution. Guidance on this will be domain-specific.
CV assesses out-of-sample performance and is highly customizable, but can be more labor-intensive to implement.
Recall: penalized iid regression
Penalized estimator for the linear model:
\[
\hat{\Bbeta}_{\lambda}
=
\underset{\Bbeta}{\arg\min}\left\{||\By-\BX\Bbeta||_2^2+\lambda\mathcal{P}(\Bbeta)\right\}.
\] The penalty provides shrinkage or regularization, and the tuning parameter \(\lambda>0\) governs how much:
Optimizes out-of-sample point prediction accuracy;
Fine for iid, but with serially dependent time series data, it’s not appropriate to randomly rip observations out of the middle and fit a model to what’s left before and after. Gotta respect the time-ordering.
out-of-sample: we care most about how methods perform on data we haven’t seen yet. CV mimics that directly;
customizable: whatever prediction task you care about (8-step-ahead point prediction, interval prediction, density, etc), simply plug-in the error metric that matches;
method-agnostic: once we have the predictions, it doesn’t matter where they came from. CV provides an apples-to-apples comparison across wildly differing procedures;
Vices:
High computational overhead to actually implement. Ideally, you fully re-train the entire procedure on every possible split. This can be very costly. So then what? Parallelize over splits? Only partially re-train? Randomly sample a subset of the splits and accept a noisier estimate of the CV error?