Understanding and modeling time-dependent data is crucial in many fields. This document outlines the foundational concepts of time series analysis, starting with the building blocks of random variables and stochastic processes, and progressing to the widely used AR, MA, and ARMA models. These tools allow us to characterize, predict, and analyze data that evolves randomly over time.

Random Variables and Stochastic Processes

A random variable is a function that assigns a real number to each outcome in a sample space of a random experiment. It's a mathematical representation of a quantity whose value is subject to variations due to chance. Random variables are typically classified as discrete or continuous.

A stochastic process is a collection of random variables indexed by time, denoted as {X_t | t ∈ T}. Each random variable X_t represents the state of a system at time t. Stochastic processes are fundamental for modeling systems that evolve randomly over time.

Probability Distributions

A Probability Density Function (PDF) describes the likelihood of a continuous random variable taking on a value within a specific range. For a continuous random variable X, the probability that X lies between a and b is the integral of its PDF, f(x), from a to b.

The Gaussian (or Normal) distribution is a symmetric, bell-shaped curve defined by its mean (μ) and variance (σ²). It is widely used because many natural phenomena approximate this distribution.

Stationarity in Time Series

For a stochastic process to be analyzed with standard models, it often needs to be stationary, meaning its statistical properties do not change over time.

Strictly Stationary Process

A time series {X_t} is strictly stationary if its joint probability distribution remains unchanged under shifts in time. That is, for any set of time indices t₁, ..., t_m and any time shift τ, the joint distribution of (X_t₁+τ, ..., X_tₘ+τ) is the same as that of (X_t₁, ..., X_tₘ).

Wide-Sense Stationary (WSS) Process

A wide-sense stationary (WSS) process is a type of stochastic process whose key statistical properties do not change over time. It is also called covariance stationary.

For a process {X_t} to be WSS, it must satisfy the following three conditions:

1. Constant Mean: The expected value stays the same at all times.
   E[X_t] = μ
2. Constant Variance: The variance is constant and finite.
   Var(X_t) = σ² < ∞
3. Time-Invariant Covariance: The covariance between values at two different times depends only on the time difference (lag τ), not the actual time.
   Cov(X_t, X_t+τ) = γ(τ)

White Noise: A Special Case of WSS

White noise is the simplest example of a WSS process. It is a sequence of random variables {ε_t} with the following properties:

• Zero Mean: E[ε_t] = 0
• Constant Variance: Var(ε_t) = σ²
• No Correlation Over Time: Cov(ε_t, ε_s) = 0 for all t ≠ s

These conditions make white noise not only WSS but also an important building block for more complex models like AR, MA, and ARMA. In fact, the error terms (ε_t) in those models are usually assumed to be white noise.

If the ε_t values also follow a Gaussian distribution, it is called Gaussian white noise. Many time series models describe how this fundamental white noise process is transformed by a system.

LTI Systems and Their Application to Time Series

Linear Time-Invariant (LTI) systems are a primary tool for analyzing and modeling time series. A stationary time series can often be viewed as the output of an LTI system whose input is white noise.

LTI Systems and Convolution

An LTI system is a system that is both linear (obeys the superposition principle) and time-invariant (its behavior doesn't change over time).

The output y(t) of an LTI system is determined by the convolution of the input signal x(t) with the system's unique impulse response h(t). Convolution is a mathematical operation expressed as:

Output of LTI Systems with WSS Inputs

When a WSS process, such as a time series, is passed through a stable LTI system:

• The output process is also WSS.
• If the input process is Gaussian, the output process will also be Gaussian. The linear transformation preserves the nature of the distribution.

Linear Time Series Models: AR, MA, and ARMA

The concepts of stationary processes and LTI systems lead to a powerful set of time series models. The Wold Decomposition Theorem provides the theoretical justification, stating that any WSS time series can be represented as the sum of a predictable (deterministic) component and a stochastic component. This stochastic part can be modeled as the output of an LTI system fed by white noise, leading to the three main classes of linear models.

Autoregressive (AR) Model

An Autoregressive (AR) model is a statistical model used in time series analysis where the current value of a series is predicted based on a linear combination of its own past values. It is denoted as AR(p), where 'p' represents the order, indicating the number of preceding values used in the prediction. The core idea is that the future is dependent on the past.

An AR model is conceptually similar to a linear regression where the variable is regressed against its own past values. For the model to be stationary, meaning its statistical properties do not change over time, there are constraints on the model's parameters.

Moving Average (MA) Model

A Moving Average (MA) model describes the current value of a time series as a linear combination of the current and past white noise error terms. It is denoted as MA(q), where 'q' is the order and indicates how many past error terms are included. This model is useful for capturing short-run shocks where the effects dissipate after 'q' periods. Unlike AR models, finite MA models are always stationary.

The MA model is essentially a finite impulse response (FIR) filter applied to white noise.

Autoregressive Moving Average (ARMA) Model

An Autoregressive Moving Average (ARMA) model combines both AR and MA components to describe a stationary time series. It models the current value as a function of both past values of the series and past error terms. An ARMA(p,q) model has an autoregressive part of order 'p' and a moving average part of order 'q'. This combination allows for a more parsimonious representation of a time series compared to a pure AR or MA model.

Z-Transform Analysis of Time Series Models

The z-transform is a mathematical tool that converts a discrete-time signal (like a time series) into a complex frequency-domain representation. It is the discrete-time equivalent of the Laplace transform and is instrumental in analyzing the properties of time series models.

Z-Transform Representation

Using the backshift operator B, where BX_t = X_t-1, the ARMA(p,q) model can be written in polynomial form:

Let Φ(B) and Θ(B) be the polynomials in the backshift operator. Replacing B with z^-1 gives the z-transform representation:

where X(z) and E(z) are the z-transforms of the time series and the error term, respectively.

Transfer Function

The transfer function, H(z), of an ARMA model describes the relationship between the input (error term) and the output (time series) in the z-domain. It is defined as the ratio of the MA polynomial to the AR polynomial:

• For a pure AR(p) model, the transfer function is H(z) = 1 / Φ(z^-1), which is an all-pole function.
• For a pure MA(q) model, the transfer function is H(z) = Θ(z^-1), which is an all-zero function.
• An ARMA(p,q) model has a pole-zero transfer function.

Stability and Invertibility Conditions

The stability of an ARMA model is determined by the roots of the autoregressive polynomial, Φ(z). For a model to be stable (and thus stationary), all the roots of Φ(z) must lie outside the unit circle in the z-plane. This is equivalent to the poles of the transfer function H(z) lying inside the unit circle when expressed in terms of z.

The invertibility of an ARMA model is determined by the roots of the moving average polynomial, Θ(z). For the model to be invertible, all the roots of Θ(z) must lie outside the unit circle. Invertibility ensures that the model can be represented as a pure autoregressive process of infinite order.

Applications

AR, MA, and ARMA models are widely used for modeling and forecasting time series in numerous domains, including:

• Economics and finance (e.g., stock prices, inflation)
• Weather and climate prediction
• Signal processing and control systems
• Retail and business sales forecasting
• Healthcare analytics and environmental monitoring

These models are supported by a rich body of linear system theory and form the basis for many advanced time series methods.