6 David W. Stockburger Multiple Regression with Many Predictor Variables 04/09/2001 mlt07.xml
a statistical procedure used to predict a single dependent variable from one or more independent variables. The procedure uses a linear transformation of the independent variables to predict the dependent variable. The linear transformation is one that minimizes the sum of the squared differences between the observed and predicted values of the dependent variable. the variables used to predict the dependent variable. the variable that is predicted. In multiple regression there may be many independent variables constant terms dependent variables optimal solutions David Stockburger 03/24/2001 Linear regression with many predictor variables

The purpose of multiple regression is to predict a single variable from one or more independent variables. Multiple regression with many predictor variables is an extension of linear regression with two predictor variables. A linear transformation of the X variables is done so that the sum of squared deviations of the observed and predicted Y is a minimum. The computations are more complex, however, because the interrelationships among all the variables must be taken into account in the weights assigned to the variables. The interpretation of the results of a multiple regression analysis is also more complex for much the same reason.

The prediction of Y is accomplished by the following equation:

Y'i = b0 + b1X1i + b2X2i + ... + bkXki

In multiple regression, the unstandardized regression weights minimize the sum of squared residuals. maximize the error. revolve around the value of the mean of Y. when squared they must sum to one. David Stockburger 03/24/2001 Linear regression with many predictor variables

The "b" values are called regression weights and are computed in a way that minimizes the sum of squared deviations

The sum of squared deviations of the observed and predicted Y values. Images/mlt0638.gif 86 45 The sum of squared deviations of the observed and predicted Y values. The sum of squared deviations of the observed and predicted Y values.

see independent variable.

in the same manner as in simple linear regression. In this case there are K independent or predictor variables rather than two and K + 1 regression weights must be estimated, one for each of the K predictor variable and one for the constant (b0) term.

In multiple regression, if there are K independent variables and N observations, how many parameters must be estimated? K + 1 K N - K - 2 N - K - 1 David Stockburger 03/24/2001 Linear regression with many predictor variables

EXAMPLE DATA

The data used to illustrate the inner workings of multiple regression will be generated from the "Example Student." The data are presented below:

Life Satisfaction Simulated Data
SubjectAgeGenderMarriedIncomeCHealthCChildCLifeSatCSESSmokeSpiritFinishLifeSat7Income7
116000380171713012226
228100380162113912015
316116521394003014288
423106510223106014873
51801752025380320 14
6300125432533613903338
7190119550284105113345
819100522175203512116
9340029602205602312664
1016100530212702903719
1125103390183416114056
1216111420312915813570
131600430152813913271
14160118541343804003744
1516100520203802713525
1632112654139370304738
1719000460172503612639
1817111055248530430426
19240017520163605413875
202611 57139 4103214267

h22.txt Life satisfaction data file in text format. h22.sav Life satisfaction data file in SPSS SAV format

The major interest of this study is the prediction of life satisfaction seven years after college from the variables that can be measured while the student is in college. These data are available both as a text file and as an SPSS data file.

a value that lies outside the range of the other values in the sample.

After doing a univariate analysis to check for outliers, the first step in analysis of data such as this is to explore the relationship borders. The minimum border of the relationships will be the bivariate correlations of all possible predictor variables with the dependent measures, LifeSat7 and Income7. The maximum border will be a linear regression model with all possible predictor variables in the regression model.

The Correlation Matrix

The largest multiple R predicting a dependent variable from a single independent variable can be found using a correlation matrix of the dependent variable with all independent variables. using a table of partial correlations of the dependent variable with all possible independent variables. using the step-down regression procedure. using the coefficients table predicting the dependent variable simultaneously from all independent variables. David Stockburger 03/24/2001 Linear regression with many predictor variables

The correlation matrix is given below for all possible predictor variables and the two dependent measures, LifeSat7 and Income7.

The SPSS output of a correlation matrix of eleven possible independent variables with the two dependent variables is illustrated. This table has three eleven by two tables shown, with each table having all independent variables listed as rows and the two dependent variables listed as columns. The first table shows correlation coefficients, the second significance levels, and the third sample sizes. images/mlt0768.gif 316 700 Correlation matrix of independent variables with dependent variables. Correlation matrix of independent variables with dependent variables.

The best and only significant ( =.05) predictor of life satisfaction seven years after college was life satisfaction in college with a correlation coefficient of .494. Other relatively high correlation coefficients included: Married (.454), Health in College (.386), Gender (.350 with females showing a generally higher level of life satisfaction), and Smoking (-.349 with Non-smokers showing a generally higher level of life satisfaction).

Income seven years after college was best predicted by knowing whether the student finished the college program or not (.499). Other variables that predicted income included the measure of spirituality (.340) and income in college (.282).

The matrix of correlations of all predictor variables is presented below.

The SPSS output of a correlation matrix of all independent variables is illustrated. In this case there are eleven possible independent variables. This table has three eleven by eleven tables shown, with each table having all variables listed as both columns and rows. The first table shows correlation coefficients, the second significance levels, and the third sample sizes. images/mlt0769.gif 916 706 Correlation matrix of all independent variables. Correlation matrix of all independent variables.

The Full Model

The largest unadjusted multiple R predicting a dependent variable from a sample of data can be found using the summary table with all independent variables in the model. a correlation matrix of the dependent variable with all independent variables. a table of partial correlations of the dependent variable with all possible independent variables. the step-up regression procedure. David Stockburger 03/24/2001 Linear regression with many predictor variables the regression model that contains all the independent variables.

The other boundary in multiple regression is called the full model, or model with all possible predictor variables included. To construct the full model, all predictor variables are included in the first block and the "Method" remains on the default value of "Enter." The three tables of output for life satisfaction seven years after college are presented below.

The multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables is shown. The value for both the multiple R is .976 and the R squared change is .953. The significance level for the R squared change is .094. Images/mlt0770.gif 739 181 Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables. Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables.

a multiple correlation coefficient squared that has been adjusted for both the number of independent variables in the model and the number of observations. The difference between the unadjusted and adjusted multiple R squared will be greatest when the number of independent variables and the number of observations are almost equal. there are a relatively large number of observations. collinearity exits between the independent variables. the unadjusted multiple R is less than .30. David Stockburger 03/24/2001 Linear regression with many predictor variables

Note that the unadjusted multiple R for this data is .976, but that the adjusted multiple R is .779. This rather large change is due to the fact that a relatively small number of observations are being predicted with a relatively large number of variables. the unadjusted value of R2 means that all subsets of predictor variables will have a value of multiple R that is smaller than .976. Note also that these variables in combination do not significantly (Sig. F Change = .094) predict life satisfaction seven years after college.

The middle table ANOVA doesn't provide much information in addition to the R2 change in the previous table. Note that the "Sig. F Change" in the preceding table is the same as the "Sig." value in the "ANOVA" table. This table was more useful in previous incarnation of multiple regression analysis (see Draper and Smith, 1981).

The multiple regression ANOVA table from the SPSS output predicting life satisfaction seven years after college with eleven independent variables is presented. The significance value for the F statistic in this table is .094. images/mlt0771.gif 534 189 Multiple regression ANOVA table predicting life satisfaction seven years after college with eleven independent variables. Multiple regression ANOVA table predicting life satisfaction seven years after college with eleven independent variables.

The multiple regression coefficients table from the SPSS output predicting life satisfaction seven years after college with eleven independent variables is presented. None of the coefficients are statistically significant at the .05 level. images/mlt0772.gif 527 366 Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables. Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables.

The full model is not statistically significant (F = 5.493, df = 11,3, sig.= .094), even though life satisfaction in college was statistically significant (p<.05) by itself. The value for this table had a total degrees of freedom of 14 because four observation had missing data and were not included in the analysis. The other degree of freedom corresponds to the intercept (constant) of the regression line. The method of handling missing data is called "listwise" because all data for a particular observation are not included if a single variable is missing.

The "Sig." column on the "Coefficients" table presents the statistical significance of that variable given all the other variables have been entered into the model. Note that no variables are statistically significant in this table. The variable "Married" comes close (Sig. = .055), but close doesn't count in significance testing.

The value in the significance column on the coefficients table in multiple regression is the exact significance level of that variable given all the other variables have been entered into the model. statistical significance of the full model relative to the partial model. exact significance level of the full model. probability of the coefficient given that there were real effects. David Stockburger 03/24/2001 Linear regression with many predictor variables

Previously it was found that the correlation between being married and life satisfaction seven years after college was relative high and positive (.454), meaning that individuals who were married in college were generally more satisfied with life seven years later. The regression weight for this same variable in the full model was negative (-20.542), meaning that over twenty points would be subtracted from an individual's predicted life satisfaction score seven years after college if they were married in college! Such are the nuances of multiple regression.

Partial output for the full model predicting the other dependent measure, income seven years after college, is presented below.

The multiple regression summary table predicting income seven years after college with eleven independent variables is shown. The value for the multiple R is .905 and the R squared change is .820. The significance level for the R squared change is .333. images/mlt0773.gif 739 181 Multiple regression summary table predicting income seven years after college with eleven independent variables. Multiple regression summary table predicting income seven years after college with eleven independent variables.

The multiple regression coefficients table from the SPSS output predicting income seven years after college with eleven independent variables is presented. None of the coefficients are statistically significant at the .05 level. images/mlt0774.gif 527 366 Multiple regression coefficients table predicting income seven years after college with eleven independent variables. Multiple regression coefficients table predicting income seven years after college with eleven independent variables.

The results are similar to the prediction on life satisfaction, with an unadjusted multiple R of .905, giving an upper limit to the combined predictive power of all the predictor variables.

Fitting Sequential Models

a statistical procedure that uses multiple regression to examine how adding independent variables in stages affects the prediction equation.

After the boundaries of the regression analysis have been established, the area between the extremes may be examined to get an idea of the interaction between the independent variables with respect to prediction. There are different schools of thought about how this should be accomplished. One school, hierarchical regression, argues that theory should drive the statistical model and that the decision of what and when terms enter the regression model should be determined by theoretical concerns. A second school of thought, stepwise regression, argues that the data can speak for themselves and allows the procedure to select predictor variables to enter the regression equation.

a sequential regression model procedure that enters the independent variables in a predetermined sequence. Entering blocks of independent variables in an order determined by the statistician is done using hierarchical regression. step-up regression. step-down regression. step-around regression. David Stockburger 03/24/2001 Linear regression with many predictor variables A mantra that could be associated with the author of the text would be Let the theory drive the data analysis. Let the data analysis drive the theory. Hypothesis testing of model assumptions is a critical first step. Start with the complex model and work toward the simple model. David Stockburger 03/24/2001 Linear regression with many predictor variables

Hierarchical Regression

Hierarchical regression adds terms to the regression model in stages. At each stage, an additional term or terms are added to the model and the change in R2 is calculated. An hypothesis test is done to test whether the change in R2 is significantly different from zero.

Using the example data, suppose a researcher wishes to examine the prediction of life satisfaction seven years after college in several stages. In the first stage, he/she enters demographic variables that the individual has little or no control over, age, gender, and socio-economic status of parents. In the second block variables are entered that the individual has at least some control, such as smoking, having children, being married, etc. The third block consists of the two attitudinal variables, life satisfaction and spirituality. This is accomplished in SPSS by entering the independent variables in blocks. Be sure the R2 change box is selected as a "Statistics" option.

The first table is a table of what variables were entered or removed at the different stages. The second table is summary of the results of the different models.

The multiple regression summary table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables sequentially entered as three blocks of independent variables (SES, Gender, and Age), (Finish, Married, Smoke, ChildC, HealthC, and IncomeC), and (SpiritC and LifeSatC) is shown. The values for the multiple Rs are .319, .900, and .976 and the corresponding R squared change values are .102, .708, and .143. The significance levels for the R squared change (sig. F change) are .746, .117, and .124. images/mlt0775.gif 739 259 Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks. Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks.

The largest change in R2 was from model 1 to model 2, with an R2 change of .708 from .102 to .810. This value was not significant, however, as were R2 changes associated with either of the other two models. Then final model has the same multiple R as the full model presented in an earlier section.

The third table presents the ANOVA significance table for the three models. The fourth table contains the regression weights and significance levels for each model. As before, the "Sig." column is an hypothesis test of the significance of that variable, given all the other variables at that stage have been entered into the model.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables sequentially entered as three blocks of independent variables (SES, Gender, and Age), (Finish, Married, Smoke, ChildC, HealthC, and IncomeC), and (SpiritC and LifeSatC) is shown. As before, noe of the coefficients, other than the constant term, are statistically significant. images/mlt0776.gif 527 621 Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks. Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks.

Note how the values of the regression weights and significance levels change as a function of when they have been entered into the model and what other variables are present.

The fifth table presents information about variables not in the regression equation at any particular stage, called excluded variables.

As additional independent variables are added or subtracted from a multiple regression model, which of the following values will remain constant for a given independent variable? the significance level of the variable in the coefficients table the standardized regression coefficient the unstandardized regression coefficient none of these values will remain constant David Stockburger 03/24/2001 Linear regression with many predictor variables

The multiple regression excluded variables table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables sequentially entered as three blocks of independent variables (SES, Gender, and Age), (Finish, Married, Smoke, ChildC, HealthC, and IncomeC), and (SpiritC and LifeSatC) is shown. images/mlt0777.gif 534 357 Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks. Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables sequentially fitted in three blocks.

the correlation between that variable and the residual of the previous model. the degree to which an independent variable cannot be predicted by other independent variables. the degree to which an independent variable can be predicted by other independent variables.

The value of "Beta In" is the size of the standardized regression weight if that variable had been entered into the model by itself in the next stage. The "Sig." column is the R2 change significance level that the variable would enter the regression equation. In this case, it can be seen that individually both INCOMEC and SPIRITC would significantly enter the regression model in the second stage. The "Partial Correlation" is the correlation between that variable and the residual of the previous model. The higher the partial correlation, the greater the change in R2 if that variable were entered into the equation by itself at the next stage.

As described in the help files of SPSS, the "Collinearity Statistics Tolerance" is "calculated as 1 minus R squared for an independent variable when it is predicted by the other independent variables already included in the analysis." This statistic may be interpreted as "A variable with very low tolerance contributes little information to a model, and can cause computational problems." (SPSS v. 10 help files.) In this case LIFESATC has a low Collinearity Statistics Tolerance (7.835E-02 or .07835) in model 2 and might cause problems if entered into the model at that point. Problems in collinearity were discussed in an earlier chapter in this text.

Collinearity occurs when some of the independent variables are highly correlated. the multiple R is less than .30. an independent variable is highly correlated with the dependent variable. the residuals are not normally distributed. David Stockburger 03/24/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables.
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables. The analysis could be best described as
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
hierarchical. step-up. step-down. non-linear. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables. The best predictor, given all the other variables were already entered in the model would be
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
AGE@REF VIQ subabuse speced David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables. The variable that least predicts given all the other variables are entered into the model would be
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
speced AGE@REF VIQ subabuse David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables. The conclusion reached with respect to the three IQ measures could best be stated as
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
in combination they significantly increased the predictive power of the model. in combination they increased the predictive power of the model, but not significantly. any increase in predictive power was mainly due to VIQ and not FSIQ or PMIQ. individually all three measures significantly increased the predicative power of the model given that the other IQ measures were already in the model. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output from a study predicting number of offences from various predictor variables. The conclusion reached with respect to AGE@REF measure could best be stated as
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress1.gif 659 614 Figure for test
the variable became a better predictor as more variables were entered into the model. the variable was statistically significant by itself. the larger the size of this variable, the fewer the number of offences. this variable would be eliminated first in a step-down regression procedure. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package.
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. The analysis could best be described as mregress2.gif 554 689 Figure for test
hierarchical. step-up. step-down. non-linear. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package. With respect to the Months since hire variable
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress2.gif 554 689 Figure for test
the fewer the number of months, the less the current salary, given the other variables in the model. the greater the number of months, the less the current salary, given the other variables in the model. the direction of the relationship of this variable to the current salary changed as a function of the other variables in the equation. no conclusions about this data can be reached based on the information given. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package. In making a 95% confidence interval for a given score using all the included variables, the range of the confidence interval would be approximately
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress2.gif 554 689 Figure for test
$19098 $4872 $9436 $2593 David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package. Conclusions with respect to gender and minority status might include
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress2.gif 554 689 Figure for test
everything else being equal, both females and minorities were paid significantly less. everything else being equal, females were paid less, but not significantly less, and minorities were paid significantly less. everything else being equal, females were paid significantly less while minorities were paid significantly more. everything else being equal, both females and minorities were paid significantly more. David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package. Which of the following variables has results that go against common sense?
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress2.gif 554 689 Figure for test
Previous experience Months since hire Beginning salary Minority classification David Stockburger 04/12/2001 Linear regression with many predictor variables The following figure presents example SPSS output predicting current salary using a regression analysis of only the clerical workers in the Employees.sav file included with the SPSS package. Everything else being equal, the best predictor of current salary would be
Example SPSS output consisting of a model summary and coefficients table used for testing purposes. mregress2.gif 554 689 Figure for test
Beginning salary Previous experience Minority classification Months since hire David Stockburger 04/12/2001 Linear regression with many predictor variables

Step-up Regression

A full model in multiple regression will likely be "less significant" than a partial model when multicollinearity is present when the additional variables only slightly increase predictive power when the additional variables are uncorrelated with the predictor variables already in the model it is mathematically impossible David Stockburger 03/05/2001 In a step-up regression, the next variable to be entered into the regression equation will have the largest value of "Beta in" in the "Excluded Variables" table. will have the smallest value of "Beta in" in the "Excluded Variables" table. will have the largest value of "Partial Correlation" in the "Excluded Variables" table. will have the largest value of "Sig." in the "Coefficients" table. David Stockburger 03/05/2001 The step-up and step-down procedures in multiple regression will always converge on the same solution will always provide an "optimal" solution using the least-squares criterion require the assumption of multicollinearity be satisfied for the uncorrelated predictor variables are viewed as the mindless application of statistical procedures to multivariate data by some statisticians David Stockburger 03/05/2001 In a step-down regression procedure, the next variable to be removed will have the highest significance level for that coefficient. highest standardized coefficient. highest unstandardized coefficient. lowest unstandardized coefficient. David Stockburger 03/24/2001 Linear regression with many predictor variables a multiple regression modeling procedure that sequentially adds variables to the regression equation based on how much additional predictive power the variable adds to the prediction equation.

At any stage, rather than entering all the variables as a block, step-up regression enters the variables one at a time, the order of entry determined by the variable that causes the greatest R2 increase, given the variables already entered into the model. To do a step-up regression using SPSS, enter all the variables in the first block and select "Method" as "Forward."

The results of the step-up regression can be better understood if the correlation coefficients are recomputed between life satisfaction seven years after college and all the predictor variables, using the "Listwise" option for missing data.

The SPSS output of a correlation matrix using the listwise deletion of missing data of eleven possible independent variables with life satisfaction seven years after college is illustrated. This table has two eleven by one tables shown, with each table having all independent variables listed as rows and the dependent variable listed as columns. The first table shows correlation coefficients and the second significance levels. All correlations are based on a sample size of 15 scores, those being the scores with complete data on all variables. images/mlt0778.gif 244 521 Correlation matrix using the listwise deletion of missing data of eleven possible independent variables with life satisfaction seven years after college. Correlation matrix using the listwise deletion of missing data of eleven possible independent variables with life satisfaction seven years after college.

Note that the correlation coefficients have changed from the original table and that the highest correlation is with SPIRITC with a value of .587. The SPIRITC variable, then, would enter the step-up regression in the first step. The partial correlation of all the remaining variables and the residual of the first stage model would then be computed. The variable with the largest partial correlation would be entered into the regression at the next step, given that it was statistically significant. The criteria for entering variables into the regression model may be optionally adjusted.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-up option is shown. The table shows two models, the first with the variable SpiritC and the second with the variables SpiritC and Finish. The multiple R for the first model is .587 and a similar value for the second is .743. Corresponding values for R squared change are .344 and .208 with significance levels of .022 and .036. images/mlt0779.gif 739 205 Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure. Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure.

The "Model Summary" table shows that two variables, SPIRITC and FINISH, are entered into the prediction model with a multiple R of .743. The SPIRITC variable was entered first (it had the largest correlation with life satisfaction) and FINISH was entered next.

The "Coefficients" table is presented next.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-up option is shown. The table shows two models, the first with the variable SpiritC and the second with the variables SpiritC and Finish. The unstandardized regression coefficients for the second model are 20.307 for the constant, .449 for SpiritC, and -8.374 for Finish. Corresponding significance levels are .004, .005, and .036. images/mlt0780.gif 527 239 Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure. Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure.

The final table presents information about variables not in the regression equation.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-up option is shown. The table shows two models, the first with the variable SpiritC and the second with the variables SpiritC and Finish. The first table shows all variables except SpiritC and the second shows all variables except SpiritC and Finish. In the first table, both Finish (.036) and HealthC (.044) are statistically significant, while in the second, none of the variables are statistically significant. images/mlt0781.gif 534 541 Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure. Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables using a step-up procedure.

At the conclusion of the first model, both FINISH and HEALTHC would significantly (p<.05) enter the regression equation at the next step. Since FINISH had a higher partial correlation (-.653) than HEALTHC (.544) it was entered into the equation at the next step. When FINISH was entered into the equation in model 2, HEALTHC would no longer significantly enter the regression model.

Step-down Regression

The step-up and step-down regression procedures will seldom result in similar regression models. will usually result in a regression model with the same variables. will usually result in a regression model with the same number of variables. will always result in a regression model with the same variables. David Stockburger 03/24/2001 Linear regression with many predictor variables The next variable to be eliminated in a step-down regression procedure will have the smallest standardized coefficient. will have the smallest significance level. will have the largest unstandardized coefficient. will have the largest standardized coefficient. David Stockburger 03/24/2001 Linear regression with many predictor variables a multiple regression modeling procedure that sequentially subtracts variables from the regression equation based on how little additional predictive power the variable adds to the current prediction equation.

By starting with a full model and eliminating variables that do not significantly enter the regression equation, a partial model may be found. This can be accomplished in SPSS by selecting a "Method" of "Backward" in the linear regression procedure. As can be seen below, the results of this analysis differ greatly from the use of the Forward Method.

The Model Summary table is presented below.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-down option is shown. Starting with all eleven independent variables, five models are shown, sequentially dropping HealthC, Smoke, IncomeC, and Gender. The multiple R drops from a value of .989 to a value of .981. None of the R squared change values are statistically significant. images/mlt0782.gif 739 353 Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure. Multiple regression summary table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure.

As the table above illustrates, this method starts with the full model with an R2 of .978. The variable of HEALTC is eliminated at the first step because it has the lowest partial correlation of any variable given that all the other predictor variables are entered into the regression equation. The next variables eliminated, in order, were SMOKE, INCOMEC, and GENDER, resulting in a model with eight predictor variables and a multiple R of .981. Note that all variables in Model 5 were significant in the following table.

The multiple regression coefficients table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-down option is shown. Starting with all eleven independent variables, five models are shown, sequentially dropping HealthC, Smoke, IncomeC, and Gender. In the final model, all coefficients are statistically significant except the constant term (.837) and Age (.058). images/mlt0783.gif 527 1151 Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure. Multiple regression coefficients table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure.

As before, the table of excluded values gives information about variables not in the regression equation at any point in time.

The multiple regression excluded variables table of the SPSS output predicting life satisfaction seven years after college with eleven independent variables entered using the step-down option is shown. Starting with all eleven independent variables, five models are shown, sequentially dropping HealthC, Smoke, IncomeC, and Gender. None of the excluded variables are statistically significant. images/mlt0784.gif 534 445 Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure. Multiple regression excluded variables table predicting life satisfaction seven years after college with eleven independent variables using a step-down procedure.

Note that none of these variables were significant in the final model.

Caveats and Options

Stepwise procedures allow the data to drive the theory. Some statisticians (I would have to include myself among them) object to the mindless application of statistical procedures to multivariate data.

There is no guarantee that the Forward and Backward procedures would agree on the same model if the options were set to different values so that the same number of variables were entered into the model. At some point a variable may no longer contribute to the regression model because of other variables in the model, even if it did contribute at an earlier point in time. For that reason SPSS provides methods of "STEPWISE" and "REMOVE" which test at each stage to see if a variable still belongs in the model. These methods could be considered a combination of Forward and Backward methods. Using them still does not guarantee that the methods will converge on a single regression model.

Cross-validation

Cross-validation will almost always result in substantial shrinkage of the multiple R is easy to apply in real-life applications is related to mean-validation, except crosses are used instead of means is viewed as the mindless application of statistical procedures to multivariate data by some statisticians David Stockburger 03/05/2001 Using a similar multiple regression analysis on two separate samples of data will result in two different regression models. will result in different unstandardized regression coefficients, but identical standardized regression coefficients. will result in identical values for the multiple R. will result in both standardized and unstandardized coefficients with similar absolute values, but different signs. David Stockburger 03/24/2001 Linear regression with many predictor variables Cross-validation requires two separate samples of data. can be done using the Probability Calculator. is a statistical technique to insure the significance of the analysis. tests whether the dependent measure is reliable. David Stockburger 03/24/2001 Linear regression with many predictor variables the loss of predictive power in a model when a sample other than the sample used to create the model is used. a statistical procedure that checks the accuracy of a model by using the model created by a given sample to predict a different sample.

The manner is which regression weights are computed guarantee that they will provide an optimal fit with respect to the least square criterion for the existing set of data. If a statistician wishes to predict a different set of data, the regression weights are no longer optimal. There will be substantial shrinkage in the value of R2 if the weights estimated on one set of data are used on a second set of data. The amount of shrinkage can be estimated using a cross-validation procedure.

In cross-validation, regression weights are estimated using one set of data and are tested on a second set of data. If the regression weights estimated on the first set of data predict the second set of data, the weights are said to be cross-validated.

Suppose an industrial/organization psychologist wished to predict job success using four different test scores. The psychologist could collect the four test scores from a randomly selected group of job applicants. After hiring all the selected group of job applicants, regardless of their scores on the tests, a measure of success on the job is taken. Success on the job is now predicted from the four test scores using a multiple regression procedure. Stepwise procedures may be used to eliminate tests that are predicting similar variance in job success. In any case, the psychologist is now ready to predict job success from the test scores for a new set of job applicants.

Not so fast! Careful application of multiple regression methods require that the regression weights be cross-validated on a different set of job applicants. Another random sample of job applicants is taken. Each applicant is given the test battery and then hired, again regardless of what scores they made on the tests. After some time on the job a measure of job success is taken. Job success is then predicted by using the regression weights found using the first set of job applicants. If the new data is successfully predicted using old regression weights, the regression procedure is said to be cross-validated. It is expected that the accuracy of prediction will not be as good for the second set of data. This is because the regression procedure is subject to variances in data from sample to sample, called "error". The greater the error in the regression procedure, the greater the shrinkage of the value of R2.

The above procedure is an idealized method of the use of multiple regression. In many real life applications of the procedure, random samples of job applicants are not feasible. There may be considerable pressure from administration to select on the basis of the test battery for the first sample, let alone the second sample needed for cross-validation. In either case the multiple regression procedure is compromised. In most cases application of regression procedures to a selected rather than a random sample will result in poorer predictions. All this must be kept in mind when evaluating research on prediction models.

Summary

Multiple regression provides a powerful method to analyze multivariate data. Considerable caution, however, must be observed when interpreting the results of a multiple regression analysis. Personal recommendations include a theory that drives the selection of variables and cross-validation of the results of the analysis.