Linear Transformations

Most multivariate statistical methods are built on the foundation of linear transformations. A linear transformation is a weighted combination of scores where each scores is first multiplied by a constant and then the products are summed. In its most general form, a linear transformation appears as follows:

X_i' = w₀ + w₁X_1i + w₂X_2i + … + w_kX_ki

where K is the number of different scores for each subject and X' is the linear combination of all the scores for a given individual.

A linear transformation combines a number of scores into a single score. A linear transformation is useful in that it is cognitively simpler to deal with a single number, the transformed score, than it is to deal with many numbers individually. For example, suppose a statistics teacher had records of absences (X₁) and number of missed homework assignments (X₂) during a semester for six students (N=6). 

 The teacher wishes to combine these separate measures into a single measure of student tardiness. The teacher could just add the two numbers together, with implied weights of one for each variable. This solution is rejected, however, as more weight would be given to absences than missed assignments because absences have greater variability. The solution, the teacher decides, is to take the sum of one-half of the absences and twice the missed homework assignments. This would result in a linear transformation of the following form:

X_i' = w₀ + w₁X_1i + w₂X_2i

where: w₀ = 0, w₁ = .5, and w₂ = 2 giving

X_i' = .5X_1i + 2X_2i

Application of this transformation to the first subject's scores would result in the following:

X_i' = .5X_1i + 2X_2i

X_i' = .5*10 + 2*2 = 5 + 4 = 9

The following table results when the linear transformation is applied to all scores for each of the six students:

 As can be seen, student number 4 has the largest measure of tardiness with a score of 19.

The Mean and Variance of the Transformed Scores

The pairs of data may be represented as points on a scatter plot. The linear transformation, defined by the equation X_i' = w₀ + w₁X_1i + w₂X_2I, may be represented as a line on the scatter plot. For example, the six pairs of example data appear as six points on the following scatter plot.

The linear transformation, X_i' = .5X_1i + 2X_2i, appears as a straight line on this scatter plot. Plotting two points that fall on the line and then using a straight edge to connect the points draws the line. For example, when X₁ = 0 then X₂ = 0 because

0 = .5X_1i + 2X_2i

0 = .5*0 + 2X_2i

0 = 2X_2i

0 = X_2i

In a similar manner, if X₁ = 4, then X₂ = -1

0 = .5X_1i + 2X_2i

0 = .5*4 + 2X_2i

0 = 2 + 2X_2i

-2 = 2X_2i

-1 = X_2i

Solving for X_2i in terms of X_1i

0 = .5X_1i + 2X_2i

2X_2i = -.5X_1i

X_2i = -.5X_1i/2 = -.25X_1i

In general where

X_i' = w₀ + w₁X_1i + w₂X_2i

X_2i = - (w₀ / w₂) - (w₁/w₂) X_1i

This line is drawn on the example scatter plot below.

A line is drawn perpendicular to the line that defines the transformation. In the example below, the perpendicular line is shown as the green line on the graph. The points on the graph are then projected onto the perpendicular line by drawing a lines parallel to the one that defined the transformation. On the example below, the red numbers written next to the line (0, 6, 9, 10, 14, 19) indicate the resulting transformations. Note the relative spacing between the numbers is similar to the difference between the numbers. For example, the distance between the points 0 and 6 is twice that between 6 and 9.

In general, if X_2i = - (w₀ / w₂) - (w₁/w₂) X_1I then a perpendicular results when X_2i = (w₂/w₁) X_1i. In this case, the perpendicular line is defined by

X_2i = (w₂/w₁) X_1i = (2/.25) X_1i = 4X_1i

when X_1i = -1, X_2i = -4

and when X_1i = 1, X_2i = 4

A linear transformation is called a mean centered transformation if the mean is subtracted from the scores before the linear transformation is done. Mean centering basically allows a cleaner view of the data. The following table presents the results of mean centering the example data and applying the transformation X_i' = .5X_1i + 2X_2i.

Score	X1	X1 - 1	X2	X2 - 2	X'
1	10	2	2	-.833	-.667
2	12	4	4	1.167	4.333
3	8	0	3	.167	.333
4	14	6	6	3.167	9.333
5	0	-8	0	-2.833	-9.667
6	4	-4	2	-.833	-3.667
Mean	8	0	2.833	0	0
s.d.	5.215	5.125	2.041	2.041	6.531
Variance	27.2	27.2	4.167	4.167	42.650

VISUALIZING NORMALIZED LINEAR TRANSFORMATIONS

The two transformations presented above may be visualized in a manner similar to that described earlier. Since the transformations have been normalized, the "scales" on the transformed axes (the blue and green lines) are identical to the scales of the original axes. Conceptually, the axes are "rotated" and the points are "projected" onto the new axes.

It appears the variance of X' might be increased if the axes were rotated clockwise even further than the present transformation. At some point the variance would begin to grow smaller again. Obtaining transformation weights that optimize variance is the problem that the next section addresses.

EIGENVALUES AND EIGENVECTORS

It was proved earlier that the total variability is unchanged when normalized transformations are done on mean centered data. It was also demonstrated that the distribution of variability changed, that is, X' had greater variance than X''. Mathematically, the question can be asked, "can a transformation be found such that one variable has a maximal amount of variance and the other has a minimal amount of variance?" Obviously the answer is "Yes" or I wouldn't have asked the question. Optimizing linear transformations such that transformed variables contain a maximal amount of variability is the fundamental problem addressed by eigenvalues and eigenvectors.

 Eigenvalues are the variances of the transformations when an optimal (maximal variance) linear transformation has been found. Eigenvectors are the transformation weights of optimal linear transformations.

Mathematical procedures are available to compute eigenvalues and eigenvectors and will be presented shortly. Before these methods are presented, however, a manual method using an interactive computer exercise will be discussed.

USING THE TRANSFORMATION PROGRAM TO FIND APPROXIMATE EIGENVALUES AND EIGENVECTORS

The transformation program has been modified by reducing the data pairs to six and rescaling the axes. After clicking on the "Enter Own Data" button, the first step is to enter the mean centered data. After entering the data, click on the "Compute Own Data" button. The means and variances of the data will appear in the appropriately labeled boxes.

In addition, the following scatter plots, controls, and text boxes will appear. Note that the variances of the transformed variables (X^*₁ and X^*₂) are the same as the original variables (X₁ and X₂). The weights are set at values w₁=1 and w₂=0 so that the transformed axes are identical to the original axes.

The weights can be changed two different ways. Clicking on the large area of the scroll bar causes a fairly large change in the transformation weights.

Clicking on the triangles on either end causes a small change in the transformation weights.

In either case, the variances of the transformed scores are recomputed and displayed. The points on the scatter plot on the left remain unchanged, but the axes are rotated to display the lines defined by the transformations. The scatter plot on the right displays the plot of the transformed scores.

The goal is to adjust the axes so that the variance of one of the transformed variables is maximized and the other is minimized. This can be accomplished by first changing the weights with fairly large steps. The variance will continue increasing until a certain point has been reached. At this point begin using smaller steps. Continue until the variance begins to decrease. Because of what I believe is rounding error, the program sometimes behaves badly at this level. Be sure to continue in both directions for a number of small steps before deciding that a maximum and minimum variance has been found.

Note that the program automatically normalizes the transformation weights and the sum of the variances remains a constant, no matter what weights are used.

In the example data, the adjustments to the weights were continued until the values in the display were found. Note that the axes pass through the points in the direction that most students intuitively believe is the position of the regression line (it isn't). In this case the eigenvalues would be 30.67 and .69. The two pairs of eigenvectors would be (.936, .352) and (.352, -.936). The transformations corresponding to the above values are presented below.

USING SPSS/WIN FACTOR ANALYSIS TO FIND EIGENVALUES AND EIGENVECTORS

 It should come as no surprise to the student that mathematical procedures have been developed to find exact eigenvalues and eigenvectors of both this relatively simple case of two variables and far more complicated situations involving linear combinations of many variables. The procedures involve matrix algebra and are beyond the scope of this text. The interested reader will find a much more complete and mathematical treatment in Johnson and Wickren, 1996.

Eigenvalues and eigenvectors can be found using the Factor Analysis package of SPSS/WIN. Starting with the raw data as variables in a data matrix, the next step is to click on "Statistics", "Data Reduction", and "Factor" in that order. The display should appear as follows:

The program will then display the choices associated with the Factor Analysis package. Select the variables that are to be included in the analysis and click them to the right-hand box. At this point some of the default values associated with the "Extraction" button will need to be modified, so clicking on this button gives the following choices:

Checking the "Covariance matrix" will result in the analysis of raw data rather than standardized scores. In addition, the computer will be told that 2 "factors" will be extracted, rather than allowing the computer to automatically decide how many "factors" will be extracted. Be sure that the "Principal components" is the selected method for factor extraction. Click on "Continue" and the main factor analysis selections should reappear. Click the "Scores" button to modify the output to print tables that will allow the computation of the eigenvectors.

Click on the "Display factor score coefficient matrix" option and then click on "Continue." Back in the main factor analysis display, click on the "OK" button to run the program.

The eigenvalues appear in an output table labeled "Total Variance Explained." Note that the values of 30.676 and .690 closely correspond to what was found by manually rotating the axes.

The eigenvectors do not appear directly on the table. They may be computed by normalizing the "Raw Components" in the following "Component Matrix" table.

While not exact, these values are within rounding error of the values found using the manual approximation procedure. The student may verify that the "Raw Components" for "2" correspond to the second normalized eigenvector.

APPLICATIONS OF LINEAR TRANSFORMATIONS

Linear transformations are used to simplify the data. In general, if the same amount of information (in this case variance) can be explained by fewer variables, the interpretation will generally be simpler.

Linear transformations are the cornerstone of multivariate statistics. In multiple regression linear transformations are used to find weights that allow many independent variables to predict a single dependent variable. In canonical correlation, both the dependent and independent variables have two or more variables and the goal of the analysis is to find a linear combination of the independent variables which best predicts a linear combination of the dependent variables.

Analysis of Variance is a special case of multiple regression where the independent variable is "dummy coded." Multivariate Analysis of Variance is a special case of canonical correlation where the independent variable(s) are also "dummy coded."

Factor analysis is similarly a linear transformation of many variables. The goal in factor analysis is a description of the variables, rather than prediction of a variable or set of variables. In factor analysis, a combination of weights is selected (extracted), usually with some goal, such as maximizing the variance of the transformed score or maximizing the correlation of the transformed score with all the scores that produce it. In factor analysis, a second combination of weights is then selected which meets the goal of the analysis. This process could continue until the number of transformed variables equals the number of original variable, but usually does not because after a few meaningful transformations, the rest do not make much sense and are discarded. The goal of factor analysis is to explain a set of variables with a few transformed variables