Linear Models in Statistics

"Rencher...offers a textbook for a one-semester advanced undergraduate or beginning graduate course.... He includes more material than can actually squeeze into one semester...a good idea in statistics."

--SciTech Book News, Vol. 24, No. 4, December 2000

"An excellent book. Highly recommended. Upper-division undergraduate and graduate students; professionals."

--Choice, Vol. 38, No. 7, March 2001

"I would recommend the book to anyone as a reference book for the topics covered.... The book should also be a strong candidate for any M.S. course in linear models because of the numerous exercises with solutions and clear writing style."

--Technometrics, Vol. 42, No. 4, May 2001

Submitted by the Publisher, September, 2001

Linear models (models that are linear in the parameters) are used in regression, analysis of variance, analysis of covariance, and extensions of these such as logistic regression and generalized linear models. This book concentrates on development of the basic theory for the first three of these topics, with a brief introduction to other models in Chapter 17. Applications are illustrated by examples and problems using real data. This combination of theory and applications will prepare the reader to further explore the literature and to more correctly interpret the output from a linear models computer package.

This introductory linear models book is primarily designed for a one-semester course for advanced undergraduates or MS students. It includes more material than can be covered in one semester so as to give an instructor a choice of topics and to serve as a reference book for researchers who wish to gain a better understanding of regression and analysis of variance. The book would also serve well as a text for PhD classes in which the instructor is looking for a one-semester introduction, and it would be a good supplementary text or reference for a more advanced PhD class for which the students need to review the basics on their own.

My overriding objective in the preparation of this book has been clarity of exposition. I hope that students, instructors, researchers, and practitioners will find this linear models text more comfortable than most. In the final stages of development, I asked my students for written comments as they read each day's assignment. They made many suggestions that led to improvements in readability of the book. I will be very grateful to readers who take the time to notify me of errors and of other suggestions they may have for improvements.

Another objective of the book is to tie up loose ends. There are many approaches to teaching regression, for example. Some books present estimation of regression coefficients for fixed x's only, other books use random x's, some use centered models, and others define estimated regression coefficients in terms of variances and covariances or in terms of correlations. I have tried to cover all these approaches carefully and to show how they relate to each other. I have attempted to do something similar for various approaches to analysis of variance. I believe this will make the book useful as a reference as well as a textbook. An instructor can choose the approach he or she prefers, and a student or researcher has access to other methods as well.

The book includes a large number of theoretical problems and a smaller number of applied problems using real data sets. The problems, along with the extensive set of answers in Appendix A, extend the book in two significant ways: (1) the theoretical problems and answers fill in nearly all gaps in derivations and proofs and also extend the coverage of material in the text, (2) the applied problems and answers become additional examples illustrating the theory. As an instructor, I find that having answers available for the students saves a great deal of class time and enables me to cover more material and cover it better. The answers would be especially useful to a reader who is engaging this material outside the formal classroom setting.

The mathematical prerequisites for this book are calculus and matrix algebra. The review of matrix algebra in Chapter 2 is intended to be sufficiently complete so that the reader with no previous experience can master matrix manipulation up to the level required in this book. Statistical prerequisites include some exposure to statistical theory, with coverage of topics such as distributions of random variables, expected values, moment-generating functions, and an introduction to estimation and testing hypotheses. These topics are briefly reviewed as each is introduced. One or two statistical methods courses would also be helpful, with coverage of topics such as t-tests, regression, and analysis of variance.

I have made considerable effort to maintain consistency of notation throughout the book. I have also attempted to employ standard notation as far as possible and to avoid exotic characters that cannot be readily reproduced on the chalkboard. With a few exceptions, I have refrained from the use of abbreviations and mnemonic devices. Personally, I often find these annoying in a book or journal article.

Equations are numbered sequentially throughout each chapter; for example, (3.29) indicates the 29th numbered equation in Chapter 3. Tables and figures are also numbered sequentially throughout each chapter in the form ''Table 3.8'' or ''Figure 3.2.'' On the other hand, examples and theorems are numbered sequentially within a section, for example, Theorems 2.2A and 2.2B.

The solution of most of the problems with real data sets requires the use of the computer. I have not discussed command files nor output of any particular program, because there are so many good packages available. Computations for the numerical examples and numerical problems were done with SAS. The data sets and SAS command files for all the numerical examples and problems in the text are available on the Internet; see Appendix B.

The bibliography is not intended to be an exhaustive survey of the literature. I have provided original references for some of the basic results in linear models and have also referred the reader to many up-to-date text and reference books useful for further reading. When citing references in the text, I have used the standard format involving the year of publication. For journal articles, the year alone suffices, for example, Fisher (1921). But for a specific reference in a book, I have included a page number or section, as in Hocking (1996, p. 216).

My selection of topics is intended to prepare the reader for a better understanding of applications and for further reading in topics such as mixed models and generalized linear models. Following a brief introduction in Chapter 1, Chapter 2 contains a careful review of all aspects of matrix algebra needed to read the book. Chapters 3, 4, and 5 cover properties of random vectors, matrices, and quadratic forms. Chapters 6, 7, and 8 cover simple and multiple linear regression, including estimation and testing hypotheses and consequences of misspecification of the model. Chapter 9 provides diagnostics for model validation and detection of influential observations. Chapter 10 treats multiple regression with random x's. Chapters 11, 12, and 13 cover balanced analysis of variance models using an overparameterized model, and Chapter 14 covers unbalanced models using a cell means model. Chapter 15 covers analysis of covariance models, and Chapter 16 treats random effects models and mixed models. Chapter 17 introduces additional topics such as nonlinear regression, logistic regression, loglinear models, Poisson regression, and generalized linear models. In my class for masters level students, I cover most of the material in Chapters 1 through 8 and 10 through 14.

My introduction to linear models came in classes taught by Dale Richards and Rolf Bargmann. I also learned much from the books by Graybill, Scheffé, and Rao. I am grateful to the following for reading the manuscript and making many valuable suggestions: David Turner, John Walker, Joel Reynolds, and Gale Rex Bryce. I thank the following students at BYU who helped with computations, graphics, and typing: David Fillmore, Candace Baker, Scott Curtis, Douglas Burton, David Dahl, Brenda Price, Eric Hintze, James Liechty, and Joy Willbur. The students in my Linear Models class went through the manuscript carefully and spotted many typographical errors and passages that needed additional clarification.

This book would not have been possible without the patience and support of my wife LaRue, who has helped me in more ways than she knows.

Alvin C. Rencher

Linear models made easy with this unique introduction Linear Models in Statistics discusses classical linear models from a matrix algebra perspective, making the subject easily accessible to readers encountering linear models for the first time. It provides a solid foundation from which to explore the literature and interpret correctly the output of computer packages, and brings together a number of approaches to regression and analysis of variance that more experienced practitioners will also benefit from. With an emphasis on broad coverage of essential topics, Linear Models in Statistics carefully develops the basic theory of regression and analysis of variance, illustrating it with examples from a wide range of disciplines. Other features of this remarkable work include:

• Easy-to-read proofs and clear explanations of concepts and procedures
• Special topics such as multiple regression with random x's and the effect of each variable on R2
• Advanced topics such as mixed and generalized linear models as well as logistic and nonlinear regression
• The use of real data sets in examples, with all data sets available over the Internet
• Numerous theoretical and applied problems, with answers in an appendix
• A thorough review of the requisite matrix algebra
• Graphs, charts, and tables as well as extensive reference

Matrix Algebra.
Random Vectors and Matrices.
Multivariate Normal Distribution.
Distribution of Quadratic Forms in y.
Simple Linear Regression.
Multiple Regression: Estimation.
Multiple Regression: Tests of Hypotheses and Confidence Intervals.
Multiple Regression: Model Validation and Diagnostics.
Multiple Regression: Random x's.
Analysis of Variance Models.
One-Way Analysis of Variance: Balanced Case.
Two-Way Analysis of Variance: Balanced Case.
Analysis of Variance: Unbalanced Data.
Analysis of Covariance.
Random Effects Models and Mixed Effects Models.
Additional Models.

Answers and Hints to Selected Problems.
Data Sets and SAS Files.
Bibliography.