Tools

Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) factorizes a matrix into orthogonal vectors and singular values, providing insight into its rank, range, and numerical stability.

SVD is a powerful matrix decomposition technique that expresses any matrix \( A \in \mathbb{R}^{m \times n} \) as:

\( A = U \Sigma V^T \)
  • \( U \): Orthogonal matrix (size \( m \times m \))
  • \( \Sigma \): Diagonal matrix with singular values
  • \( V^T \): Transpose of orthogonal matrix \( V \)

Step 1: Define the matrix

We are using the matrix:

\( A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \)

Step 2: Compute \( A^T A \) and \( A A^T \)

\( A^T A = \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 10 & 14 \\ 14 & 20 \end{bmatrix} \)
\( A A^T = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} = \begin{bmatrix} 5 & 11 \\ 11 & 25 \end{bmatrix} \)

Step 3: Find eigenvalues of \( A^T A \)

We solve the characteristic equation:

\( \text{det}(A^T A - \lambda I) = 0 \)

For \( A^T A = \begin{bmatrix} 10 & 14 \\ 14 & 20 \end{bmatrix} \), the eigenvalues are:

  • λ₁ ≈ 29.866
  • λ₂ ≈ 0.134

Step 4: Singular Values

Take square roots of eigenvalues to get singular values:

  • σ₁ = \( \sqrt{29.866} \approx 5.466 \)
  • σ₂ = \( \sqrt{0.134} \approx 0.366 \)

Step 5: Eigenvectors of \( A^T A \) and \( A A^T \)

Let \( v_1 \), \( v_2 \) be eigenvectors of \( A^T A \). These form the columns of \( V \).

To get \( u_1, u_2 \):

\( u_i = \frac{A v_i}{\sigma_i} \)

These form the columns of \( U \).

Step 6: Final Decomposition

Putting everything together:

\( U \approx \begin{bmatrix} 0.404 & -0.915 \\ 0.915 & 0.404 \end{bmatrix} \quad \Sigma = \begin{bmatrix} 5.466 & 0 \\ 0 & 0.366 \end{bmatrix} \quad V^T \approx \begin{bmatrix} 0.576 & 0.817 \\ -0.817 & 0.576 \end{bmatrix} \)