Random variable

A random variable is a mathematical formalization of a quantity or object which depends on random events.

 

Random process

A random process is a mathematical object that usually depends on a sequence defined as a sequence of random variables, where the index of the sequence has the interpretation of time. 

 

 

3. Definition of Stationary and Wide Sense Stationary Process

A stochastic process {…, X_t-1, X_t, X_t+1, X_t+2, …} consisting of random variables indexed by time index t is a time series.

The stochastic behavior of {X_t} is determined by specifying the probability density or mass functions (pdf’s).

p(x_{t1, ­} x_t2, x_t3, …, x_tm)

for all finite collections of time indexes

{(t₁, t₂, …, t_m), m < ∞}

i.e., all finite-dimensional distributions of {X_t}

 

A time series {X_t} is strictly stationary if

p(t₁ + τ, t₂ + τ, …, t_m + τ) = p(t₁, t₂, …, t_m),

∀τ, ∀m, ∀(t₁, t₂, …, t_m).

Where p(t₁ + τ, t₂ + τ, …, t_m + τ) represents the cumulative distribution function of the unconditional (i.e., with no reference to any particular starting value) joint distribution (parameters such as mean and variance) of {X_t} is said to be strictly stationary or strict-sense stationary if τ doesn’t affect the function p. So, p is not a function of time.

 

A time series {X_t} is called covariance stationary if

E(X_t) = μ

Var(X_t) = σ²_x

Cov(X_t, X_t+τ) = γ(τ)

** (All constant over time t)

 

 

Wide Sense Stationary Process

A random process is called weak-sense stationary or wide-sense stationary (WSS) if its mean function and its correlation function do not change by shifts in time.

µ_x(t) = µ_x

Rxx(t1, t2) = Rxx(t1 + α, t2 + α)   for every α

                   = Rxx(t1 - t2, 0)

 

 

3. a. The output of a Wide Sense Stationary When the System is LTI

The output of filter is Y(t) = h(t) * X(t), where h(t) is the impulse response of a filter. Let the output mean be µ_y.

µ_y = E[Y(t)] = E[h(t) * X(t)]                                           

[‘E’ denotes the expected value]

µ_y = E[]

µ_y = ]

 

Now, if X(t) is WSS, a shift in time does not affect its mean, i.e.,

E[X(t)] = E[X(t - )] = µ_x

E[Y(t)] =  µ_x

            = µ_x

 

∵  = H(0)

 

µ_{y =} µ_x . H(0)

 

Since the H(0) will be some constant, we conclude that the output will also have the same nature as the input signal. 

So, the output of a wide sense stationary when the system is LTI is zero mean and WSS signal.

 

 

3. b. When the system is Linear and the input is Gaussian

An important property of the Gaussian random process is that their Probability density function is completely determined by their mean and covariance, i.e.,

f_x(X) =  exp()

Where  is mean

And σ is standard deviation

 

The Fourier analysis states that any Linear system in the frequency spectrum can be represented as the sum of sinusoids, i.e.,

H(ω) = ω₀)

Now, if the input X(t) is Gaussian, then the output y(t) in the frequency spectrum will be :

Y(ω) = X(ω).H(ω)

 

For a sinusoid input, the output will be varied in Magnitude and phase, but nature will remain the same according to the input.

 

So, the output will be Gaussian.

 

The output phase and magnitude will be varied, but nature will remain the same as that of the input. So, if the input is Gaussian, the output will also be Gaussian.

 

 

3. c. Time Series Analysis - AR, MA, and ARMA Models

 

Wold Representation Theorem

Any zero-mean covariance stationary time series {X_t}can be decomposed as

X_t = V_t  +  S_t

Where { V_t } is a linearly deterministic process, i.e., a linear combination of past values of  V_t with constant coefficients. It is analogical to autoregressive (AR) model.

S_t = _i Ꜫ_t-i  is an infinite moving average process of error terms, where

θ₀ = 1

_i )²  <  ∞

Ꜫ_t is linearly unpredictable white noise, i.e.,

E(Ꜫ_t) = 0,  E(Ꜫ_t²) = σ²,  E(Ꜫ_t Ꜫ_s) = 0  ∀_t,  ∀_{s ≠ t}

And Ꜫ_t is uncorrelated with V_t

E(Ꜫ_{t ­}V_s) = 0  ∀_{t, s}

 

The output of a linear system characterized by a rational system function of the form

    … (i)

The corresponding difference equation is

…. (ii)

 

Where w[n] is the input sequence to the system and the observation data, x[n], represents the output sequence.

In power spectrum estimation, the input sequence is not observable. However, if the observed data are characterized as a stationary random process, then the input sequence is also assumed to be a stationary random process. In such a case the power density spectrum of the data is

   … (iii)

Where Γ_ww(f) is the power density spectrum of the input sequence and H(f) is the frequency response of the model.

Since our objective is to estimate the power density spectrum Γ_xx(f), it is convenient to assume that the input sequence w[n] is a zero mean white noise sequence with autocorrelation.

Γ_ww(m) = σ²_w δ(m)    … (iv)

 

 

Where σ²_w  is variance (i.e., σ²_w = |w(n)|²). Then the power density spectrum of the observed data is simply

   … (v)

In the model-based approach the spectrum estimation procedure consists of two steps. Given data sequence x[n], 0 ≤ n ≤ N-1, we estimate the parameters {a_k} and {b_k} of the model. Then from this estimates, we compute the power spectrum estimate according to equation (v)

 

Random process x[n] generated by pole-zero model as shown in equation (i) is called an autoregressive moving average (ARMA) process of order (p,q) and it is usually denoted as ARMA (p,q). If q = 0 and b₀ = 1, the resulting system model has a system function H(z) = 1 / A(z) and its output x[n] is called an autoregressive (AR) process of order p. This is denoted as AR(p). The third possible model is obtained by setting obtained by setting A(z) = 1, so that H(z) = B(z). Its output x[n] is called a moving average (MA) process of order q and denoted as MA(q).

Of these three linear models the AR model is by far the most widely used. The reasons are twofold. First, the AR model is suitable for representing spectra with narrow peaks (resonances). Second, the AR model results in very simple linear equations for the AR parameters. On the other hand, the MA model, as a general rule, requires many more coefficients to represent a narrow spectrum. Consequently, it is rarely used by itself as a model for spectrum estimation. By combining poles and zeros, the ARMA model provides a more efficient representation from the viewpoint of the number of model parameters, of the spectrum of a random process. The decomposition theorem due to Wold representation asserts that any ARMA or MA process can be represented uniquely by an AR model of possibly infinite order. In view of this theorem, the issue of model selection reduces to selecting the model that requires the smallest number of model parameters that are easy to compute. Usually the choice in practice is the AR model.

 

 

 

Autoregressive (AR) Model

The notation AR(p) indicates an autoregressive model of order p. The AR(p) model is defined as

X_t = _i X_t-i  + Ꜫ_t

Where φ₁, …, φ_p are parameters of the model, and Ꜫ_t is white noise. This can be equivalently written using the backshift operator B as

X_t = _i Bⁱ X_t  + Ꜫ_t

 

so that, moving the summation term to the left side and using polynomial notation, we have

ϕ[B] X_t  = Ꜫ_t

 

An autoregressive model is simply a linear regression of the current value of the series against one or more prior values of the series. The value of p is called the order of the AR model. AR models can be analyzed with one of various methods; including standard linear least squares techniques. They also have a straightforward interpretation.

 

An autoregressive model can thus be viewed as the output of an all-pole infinite impulse response filter whose input is white noise.

Some parameter constraints are necessary for the model to remain weak-sense stationary. For example, processes in the AR(1) model with |φ₁| ≥ 1 are not stationary. More generally, for an AR(p) model to be weak-sense stationary, the roots of the polynomial Φ(z) : = 1 - _i zⁱ   must lie outside the unit circle i.e., each (complex) root z_i must satisfy  | z_i | > 1

 

 

Moving-average (MA) model

The notation MA(q) refers to the moving average model of order q:

X_t = µ + Ꜫ_t + θ₁Ꜫ_t-1 + … + θ_qꜪ_t-q   =  µ + _i Ꜫ_t-i + Ꜫ_t

 

Where µ is the mean of the series, the θ₁, θ₂, …, θ_q are parameters of the model and the Ꜫ_t, Ꜫ_t-1, …, Ꜫ_t-q  are white noise error terms. The value of q is called the order of the MA model. This can be equivalently written in terms of the backshift operator B as

X_t = µ + (1 + θ₁B + … + θ_qB^q)Ꜫ_t   

 

Thus a moving-average model is conceptually a linear regression of the current value of series against current and previous (observed) white noise error terms or random shocks. The random shocks at each point are assumed to be mutually independent and to come from the same distribution, typically a normal distribution, with location at zero and constant scale.

 

 

Autoregressive-moving-average (ARMA) Model

In the statistical analysis of time series, autoregressive-moving-average (ARMA) models provide a parsimonious description of a (weakly) stationary stochastic process in terms of two polynomials, one for autoregression (AR) and the second for moving-average (MA).

For a given time series of data X_t, the ARMA model is a tool for understanding and, perhaps, predicting future values in this series. The AR part involves regressing the variable on its own lagged (i.e., past) values. The MA part involves modeling the error term as a linear combination of error terms occurring contemporaneously and at various times in the past.

The model is usually referred to as the ARMA(p,q) model where p is the order of the AR part and q is the order of the MA part.

 

The notation ARMA(p, q) refers to the model with p autoregressive terms and q moving-average terms. This model contains the AR(p) and MA(q) models,

X_t = Ꜫ_t  +  _i X_t-i   +  _i Ꜫ_t-i

 

This method was useful for low-order polynomials (of degree three or less). The ARMA model is essentially an infinite impulse response filter applied to white noise.

 

 

Specifications in terms of lag operator

The model can be specified in terms of the lag operator L. In these terms then the AR(p) model is given by

Ꜫ_t  =  (_i Lⁱ ) X_t = φ(L) X_t

 

Where φ represents the polynomial

φ(L) = _i Lⁱ

 

The MA(q) model is given by

X_t  =  ( 1  +   _i Lⁱ  ) Ꜫ_t  =  θ(L) Ꜫ_t,

 

Where θ represents the polynomial

θ(L) = 1  +  _i Lⁱ  

 

Finally, the combined ARMA(p,q) model is given by

 (_i Lⁱ ) X_t   =   ( 1  +   _i Lⁱ  ) Ꜫ_t

 

Or, more concisely,

φ(L)X_t   =   θ(L) Ꜫ_t

 

or,

 X_t  =  Ꜫ_t

 

 

Applications

The time-series models are used to analyze and predict the data. A linear time series is modeled by linear difference equations involving the time series and the white noise or the innovation process. Such ARMA(p, q) models can be analyzed using the linear system theory.

Moreover, it is used for weather forecasting, climate forecasting, economic forecasting, healthcare forecasting engineering forecasting, finance forecasting, retail forecasting, business forecasting, environmental studies forecasting, social studies forecasting, and more.