Figure 1
Let S be a nontrivial subspace of a vector space V and assume that v is a vector in V that does not lie in S. Then the vector v can be uniquely written as a sum, v ‖ S + v ⊥ S , where v ‖ S is parallel to S and v ⊥ S is orthogonal to S; see Figure .
The vector v ‖ S , which actually lies in S, is called the projection of v onto S, also denoted proj S v. If v 1, v 2, …, v r form an orthogonal basis for S, then the projection of v onto S is the sum of the projections of v onto the individual basis vectors, a fact that depends critically on the basis vectors being orthogonal:
Figure shows geometrically why this formula is true in the case of a 2‐dimensional subspace S in R 3.
Figure 2
Example 1: Let S be the 2‐dimensional subspace of R 3 spanned by the orthogonal vectors v 1 = (1, 2, 1) and v 2 = (1, −1, 1). Write the vector v = (−2, 2, 2) as the sum of a vector in S and a vector orthogonal to S.
From (*), the projection of v onto S is the vector
Therefore, v = v ‖ S where v ‖ S = (0, 2, 0) and
That v ⊥ S = (−2, 0, 2) truly is orthogonal to S is proved by noting that it is orthogonal to both v 1 and v 2:
In summary, then, the unique representation of the vector v as the sum of a vector in S and a vector orthogonal to S reads as follows:
See Figure .
Figure 3
Example 2: Let S be a subspace of a Euclidean vector space V. The collection of all vectors in V that are orthogonal to every vector in S is called the orthogonal complement of S:
( S ⊥ is read “S perp.”) Show that S ⊥ is also a subspace of V.
Proof. First, note that S ⊥ is nonempty, since 0 ∈ S ⊥. In order to prove that S ⊥ is a subspace, closure under vector addition and scalar multiplication must be established. Let v 1 and v 2 be vectors in S ⊥; since v 1 · s = v 2 · s = 0 for every vector s in S,
proving that v 1 + v 2 ∈ S ⊥. Therefore, S ⊥ is closed under vector addition. Finally, if k is a scalar, then for any v in S ⊥, ( k v) · s = k( v · s) = k(0) = 0 for every vector s in S, which shows that S ⊥ is also closed under scalar multiplication. This completes the proof.
Example 3: Find the orthogonal complement of the x−y plane in R 3.
At first glance, it might seem that the x−z plane is the orthogonal complement of the x−y plane, just as a wall is perpendicular to the floor. However, not every vector in the x−z plane is orthogonal to every vector in the x−y plane: for example, the vector v = (1, 0, 1) in the x−z plane is not orthogonal to the vector w = (1, 1, 0) in the x−y plane, since v · w = 1 ≠ 0. See Figure . The vectors that are orthogonal to every vector in the x−y plane are only those along the z axis; this is the orthogonal complement in R 3 of the x−y plane. In fact, it can be shown that if S is a k‐dimensional subspace of R n , then dim S ⊥ = n − k; thus, dim S + dim S ⊥ = n, the dimension of the entire space. Since the x−y plane is a 2‐dimensional subspace of R 3, its orthogonal complement in R 3 must have dimension 3 − 2 = 1. This result would remove the x−z plane, which is 2‐dimensional, from consideration as the orthogonal complement of the x−y plane.
Figure 4
Example 4: Let P be the subspace of R 3 specified by the equation 2 x + y = 2 z = 0. Find the distance between P and the point q = (3, 2, 1).
The subspace P is clearly a plane in R 3, and q is a point that does not lie in P. From Figure , it is clear that the distance from q to P is the length of the component of q orthogonal to P.
Figure 5
One way to find the orthogonal component q ⊥ P is to find an orthogonal basis for P, use these vectors to project the vector q onto P, and then form the difference q − proj P q to obtain q ⊥ P . A simpler method here is to project q onto a vector that is known to be orthogonal to P. Since the coefficients of x, y, and z in the equation of the plane provide the components of a normal vector to P, n = (2, 1, −2) is orthogonal to P. Now, since
the distance between P and the point q is 2.
The Gram‐Schmidt orthogonalization algorithm. The advantage of an orthonormal basis is clear. The components of a vector relative to an orthonormal basis are very easy to determine: A simple dot product calculation is all that is required. The question is, how do you obtain such a basis? In particular, if B is a basis for a vector space V, how can you transform B into an orthonormal basis for V? The process of projecting a vector v onto a subspace S—then forming the difference v − proj S v to obtain a vector, v ⊥ S , orthogonal to S—is the key to the algorithm.
Example 5: Transform the basis B = { v 1 = (4, 2), v 2 = (1, 2)} for R 2 into an orthonormal one.
The first step is to keep v 1; it will be normalized later. The second step is to project v 2 onto the subspace spanned by v 1 and then form the difference v 2 − proj v1 v 2 = v ⊥1 Since
the vector component of v 2 orthogonal to v 1 is
as illustrated in Figure .
Figure 6
The vectors v 1 and v ⊥1 are now normalized:
Thus, the basis B = { v 1 = (4, 2), v 2 = (1, 2)} is transformed into the orthonormal basis
shown in Figure .
Figure 7
The preceding example illustrates the Gram‐Schmidt orthogonalization algorithm for a basis B consisting of two vectors. It is important to understand that this process not only produces an orthogonal basis B′ for the space, but also preserves the subspaces. That is, the subspace spanned by the first vector in B′ is the same as the subspace spanned by the first vector in B′ and the space spanned by the two vectors in B′ is the same as the subspace spanned by the two vectors in B.
In general, the Gram‐Schmidt orthogonalization algorithm, which transforms a basis, B = { v 1, v 2,…, v r }, for a vector space V into an orthogonal basis, B′ { w 1, w 2,…, w r }, for V—while preserving the subspaces along the way—proceeds as follows:
Step 1. Set w 1 equal to v 1
Step 2. Project v 2 onto S 1, the space spanned by w 1; then, form the difference v 2 − proj S 1 v 2 This is w 2.
Step 3. Project v 3 onto S 2, the space spanned by w 1 and w 2; then, form the difference v 3 − proj S 2 v 3. This is w 3.
Step i. Project v i onto S i −1, the space spanned by w 1, …, w i−1 ; then, form the difference v i − proj S i−1 v i . This is w i .
This process continues until Step r, when w r is formed, and the orthogonal basis is complete. If an orthonormal basis is desired, normalize each of the vectors w i .
Example 6: Let H be the 3‐dimensional subspace of R 4 with basis
Find an orthogonal basis for H and then—by normalizing these vectors—an orthonormal basis for H. What are the components of the vector x = (1, 1, −1, 1) relative to this orthonormal basis? What happens if you attempt to find the componets of the vector y = (1, 1, 1, 1) relative to the orthonormal basis?
The first step is to set w 1 equal to v 1. The second step is to project v 2 onto the subspace spanned by w 1 and then form the difference v 2− proj W1 v 2 = W 2. Since
the vector component of v 2 orthogonal to w 1 is
Now, for the last step: Project v 3 onto the subspace S 2 spanned by w 1 and w 2 (which is the same as the subspace spanned by v 1 and v 2) and form the difference v 3− proj S 2 v 3 to give the vector, w 3, orthogonal to this subspace. Since
and
and { w 1, w 2} is an orthogonal basis for S 2, the projection of v 3 onto S 2 is
This gives
Therefore, the Gram‐Schmidt process produces from B the following orthogonal basis for H:
You may verify that these vectors are indeed orthogonal by checking that w 1 · w 2 = w 1 · w 3 = w 2 · w 3 = 0 and that the subspaces are preserved along the way:
An orthonormal basis for H is obtained by normalizing the vectors w 1, w 2, and w 3:
Relative to the orthonormal basis B′′ = { ŵ 1, ŵ 2, ŵ 3}, the vector x = (1, 1, −1, 1) has components
These calculations imply that
a result that is easily verified.
If the components of y = (1, 1, 1, 1) relative to this basis are desired, you might proceed exactly as above, finding
These calculations seem to imply that
The problem, however, is that this equation is not true, as the following calculation shows:
What went wrong? The problem is that the vector y is not in H, so no linear combination of the vectors in any basis for H can give y. The linear combination
gives only the projection of y onto H.
Example 7: If the rows of a matrix form an orthonormal basis for R n , then the matrix is said to be orthogonal. (The term orthonormal would have been better, but the terminology is now too well established.) If A is an orthogonal matrix, show that A −1 = A T.
Let B = { vˆ 1, vˆ 2, …, vˆ n } be an orthonormal basis for R n and consider the matrix A whose rows are these basis vectors:
The matrix A T has these basis vectors as its columns:
Since the vectors vˆ 1, vˆ 2, …, vˆ n are orthonormal,
Now, because the ( i, j) entry of the product AA T is the dot product of row i in A and column j in A T,
Thus, A −1 = A T. [In fact, the statement A −1 = A T is sometimes taken as the definition of an orthogonal matrix (from which it is then shown that the rows of A form an orthonormal basis for R n ).]
An additional fact now follows easily. Assume that A is orthogonal, so A −1 = A T. Taking the inverse of both sides of this equation gives
which implies that A T is orthogonal (because its transpose equals its inverse). The conclusion
means that if the rows of a matrix form an orthonormal basis for R n , then so do the columns.