Understanding SVD interactively and visually
Welcome to an interactive look at singular value decomposition! This article may differ from others you have read: when you double-click most of the animations below, they become interactive, so you can build intuition by exploring the ideas yourself.
Let's try it. A matrix represents a stretch and a rotation of the spatial grid. More precisely, it sends the standard basis to the positions given by its columns. The animation below demonstrates this. Double-click it and drag the two basis vectors yourself to see how the space transforms.
To make this more concrete, let's look at how one linear transformation affects different shapes. Double-click the animation to see each shape change under the same transformation.
Intuitively, the circle becomes an ellipse, while the previously neat triangle and square become less regular. The circle turning into an ellipse tells us that the transformation stretches space along two perpendicular directions. The square, whose sides were aligned with the x and y axes, no longer stays regular—so the two stretching directions are no longer the original basis directions.
Can you find a cross whose arms are exactly the major and minor axes of the ellipse? Try dragging the point on the unit circle in the interaction below. When you find such a direction, a "passed" message appears. The two sliders in the bottom-right let you scale the shapes along the two directions you found. Try several transformations and see what you notice.
Scaling along the major and minor axes makes the circle, square, and triangle all return to their original shapes at the same time. So even though this linear transformation makes the equilateral triangle and square look irregular, from another point of view it is just an ordinary stretch along two perpendicular directions.
In other words, every linear transformation consists of stretching along some orthogonal directions and then rotating. In the animation below, double-click to choose any linear transformation you like, then press Space or the next-page button in the bottom-right to see how it is built from a stretch along one direction followed by a rotation.
This process—"find a direction, stretch, then rotate"—is exactly singular value decomposition. We can decompose any matrix into this form. How do we express these operations in linear algebra, and how do we find the rotation angle and the stretch factors?
First, let's review how matrices represent stretching and rotation. A diagonal matrix scales along the basis directions. As usual, double-click to drag the numbers and see how the ball changes.
A rotation is a matrix $[v_1, v_2]$ where both vectors $v_1$ and $v_2$ lie on the unit circle and remain orthogonal. In the demo below you can see the effect of a rotation matrix and of its transpose. Double-click to drag the rotation yourself.
You may have noticed that the transpose rotates in the opposite direction, which means the transpose of a unit orthogonal matrix is its inverse. This is easy to prove: the dot product of the two columns is 0, and each column has length 1.
Now we have everything we need to express "find a direction, stretch, then rotate into the right orientation" in linear algebra. Since the stretch above happens along the coordinate axes, we first rotate by the negative of the target angle to align the stretching axis with a coordinate axis, apply the stretch, and then rotate.
Let's rewrite what we saw in the previous animations in the language of linear algebra.
This is the form of singular value decomposition you see in textbooks. The singular values are the coefficients of the stretching step in the middle.
The next question is: how do we find the three factors? It turns out that $A^T A$ gives us an eigendecomposition. Eigendecompose $A^T A$: its eigenvectors are $V$, and the square roots of its eigenvalues are the numbers on the diagonal of $\Sigma$, called the singular values.
So what exactly is a singular value? Watch the animation below, double-click and drag freely, and see the intuitive connection between the linear transformation and its singular values.
If you look closely, you can see that a singular value is the factor by which the circle's diameter stretches along the ellipse's major or minor axis—that is, how much the transformation stretches the directions of greatest change.
If we expand the SVD, we find that it is a sum of rank-$1$ matrices, each expressing the change this linear transformation makes in one direction.
The next example shows the difference between keeping only the first two terms and keeping all three when the last singular value is very small. Double-click to choose which terms to keep, and drag the view to see how the linear transformation changes.
You can see that dropping the last term leaves the transformed shape almost unchanged, because although the matrix is full rank, that layer is almost flattened—two of its columns are nearly linearly dependent. Real-world matrices are rarely exactly rank-deficient; they are usually only nearly so. By thresholding the singular values, we can estimate the effective rank of a matrix. Since removing directions with little influence changes almost nothing, the same idea is used for compression and feature extraction.
This article was made with Wanim. Did you enjoy this kind of lesson?