[En] Our recent book review with Psychometrika

This is a post-peer-review, pre-copyedit version of an article published in Psychometrika. The final authenticated version is available online at: http://dx.doi.org/10.1007/s11336-020-09714-5

Book Review

A review of “A Course in Item Response Theory and Modeling with

Stata” by Raykov and Marcoulides

Dandan Chen

Jinming Zhang

University of Illinois at Urbana-Champaign (UIUC)

Raykov, T., & Marcoulides, G. A. (2018). A course in item response theory and modeling with Stata. College Station, TX: StataCorp LLC. 270 p. $54.00. ISBN: 978-1-59718-266-9.

The 1950s witnessed the establishment of the theoretical foundations of item response theory (IRT; Lord, 1952, 1953). However, IRT modeling did not come into full use until the 1970s when
technological advancements made it feasible. IRT software tools such as BILOG, PARSCALE, MULTILOG, ConQuest, IRTGEN, IRTDIF, and OPLM have been developed and applied to analyze response data. A thorough review of them can be found in Zhao and Hambleton (2009). Recently, more general statistical software packages such as SAS (e.g., Cole & Paek, 2017), SPSS (e.g., DiTrapani et al., 2018), and Stata (e.g., Yang & Zheng, 2018) have enabled functions for IRT modeling. In particular, the new features with the release of Stata 14 include IRT modeling (StataCorp LLC, 2015). Raykov and Marcoulides, professors in the field of measurement science and research methodology, provide a comprehensive presentation of IRT and its modeling practice in Stata in their new book A Course in Item Response Theory and Modeling with Stata. It excels in illuminating conceptual commonalities across different terminologies in applied statistics.

This book is unlike the majority of IRT books for several reasons. First, previous books on IRT typically do not show connections between IRT and other related statistical frameworks like factor analysis (FA), generalized linear modeling (GLM), and latent variable modeling (LVM). For example, Fundamentals of Item Response Theory by Hambleton, Swaminathan, and Rogers (1991) concentrate on the gist of IRT without covering the broader statistical frameworks that help readers connect dots and understand how IRT is a part of them. Second, many books on IRT contain unnecessary juxtaposition of IRT to CTT, which Raykov and Marcoulides intentionally avoid in their book. It is worth noting that IRT-CTT comparisons are prevalent in the psychometric literature (e.g., Albano, 2020; Hambleton & Jones, 1993). As the authors themselves elucidate, however, illustrating IRT itself is enough, and there is no need to compare it with CTT to demonstrate IRT’s value. More importantly, they hold the view that inappropriate juxtaposition can worsen compartmentalized research, which can in return do a long-term disservice to the knowledge development in IRT. Third, many existing instructional materials for software applications in IRT modeling do not include straightforward conceptual explanations about some critical components of IRT. This situation is partly attributable to the fact that many individuals capitalize on software features for IRT modeling to their advantage. Technical manuals such as Stata Item Response Reference Manual (StataCorp LLC, 2019) do not provide enough detail about the choice of methods and algorithms and the interpretation of results, which are available in Raykov and Marcoulides’s book. In sum, Raykov and Marcoulides confront these issues via a coherent coverage of IRT modeling in both the conceptual and applied aspects in this book.

In the first part of the book, from Chapter 1 to Chapter 5, Raykov and Marcoulides primarily introduce vital concepts in IRT. This part highlights the close conceptual rapports between IRT and CTT, FA, GLM, and nonlinear models. It shows the first initiative ever, in a book on IRT, to systematically elaborate on IRT in connection with various other statistical frameworks. The authors start with fundamental concepts from general statistics, such as the probit and logit analyses, cumulative distribution function (CDF) and normal ogive, and formula conversions for FA and GLM. Then, they illustrate the presence and interpretation of these concepts and their use in the context of IRT, drawing on statistical terms, frameworks, and mathematical proofs that can be understood by a broad audience.

Several key points the authors assert and demonstrate in this first part are summarized as follows. First, some misinterpretations of CTT or IRT should be demystified. For instance, that the variance term for an observed score in CTT is constant across all subjects is a misconception. Also, there is no such thing as “a CTT model” because the CTT decomposition is a tautology that cannot be falsified. In contrast, a model is a statement based on a set of assumptions that can be falsified empirically. Moreover, the three-parameter logistic (3PL) model in IRT is not strictly a logistic model despite the widespread use of the “logistic model” in the name of this model. Second, IRT is connected with other statistical frameworks. For example, the true score in CTT is of primary concern in IRT; the IRT-based models are analogous to multivariate binary logistic regression models (BLR), except that the variables in the former are latent or unobservable whereas the variables in the latter are manifest or observable. BLR, furthermore, is a special case of GLM and nonlinear FA models. Third, some technical issues in IRT can be understood in the context of more general statistical problems. The local independence condition in IRT is analogous to the conditional independence condition; the former concerns latent variables while the latter concerns observed variables. Also, the “scale indeterminacy” in IRT is part of the “factor indeterminacy” in latent variable models. Additionally, this factor indeterminacy is associated with “lack of identification,” a more general issue in applied statistics.

In the second part, from Chapter 6 to Chapter 12, Raykov and Marcoulides combine the codes and technical description of IRT modeling procedures with exemplary statistical interpretations of the output from Stata. It is a unique contribution to the existing modeling literature as it combines the most up-to-date Stata applications with detailed conceptual interpretations. A technical manual like Stata Item Response Reference Manual (StataCorps LLC, 2019) primarily presents Stata code and technical output, with only sparing theoretical content. Raykov and Marcoulides do more than that with extensive statistical interpretations via a heuristic approach for problem-solving in applied research.

Specifically, this second part starts with a brief introduction of the Stata interface and then proceeds with a presentation of specific problem-solving procedures guided by statistical questions. This question-oriented approach helps elaborate relevant technical methods, solutions, and interpretations of results in response to each question in IRT modeling. Also, these guiding questions are conceptually connected, facilitating the transition of the illustration on IRT modeling and provoking independent thinking and knowledge retrieval. The authors detail what issues should be addressed, what technical procedures to be completed, how to interpret the statistical output, and what else should be considered after we settle with a decision from one test.

One impressive feature worth noting is the highly accessible non-technical text throughout this book that stimulates readers to learn. The language is straightforward enough for any reader who has taken intro-level statistics or math courses to follow. For example, when it comes to the relationship between local independence and unidimensionality in IRT, instead of taking whatever is given like in the widespread statistical literature, Raykov and Marcoulides in this book first illustrate the difference between a sufficient condition and a necessary condition. They then demonstrate that the local independence condition is necessary but not sufficient for meeting the unidimensionality. This approach helps readers continuously build new knowledge about IRT in the commonly seen statistical terms. Specifically, when expounding how to determine whether a condition is a sufficient condition as opposed to a necessary one, Raykov and Marcoulides draw on practical examples. They discuss the case of determining whether an integer is even or odd on condition that this integer can be divided by 4. As this discussion evolves, they conclude with an important takeaway that it is more difficult to find a sufficient condition than a necessary condition to a condition of interest.

Meanwhile, aware of learning strategies, Raykov and Marcoulides explain many concepts that may be hard to digest in multiple ways. Explicitly, they reiterate concepts in different parts of the book and guide readers to recall what they have learned. For instance, after briefing the normal ogive in Chapter 2, the authors discussed it again in Chapter 5, using “[w]e recall from [C]hapter 2 that…” to help readers retrieve prior information. Also, the authors pose intriguing questions to lead readers to resolve problems with the methods they have just learned in the book. An instance is illustrating the relationship between normal ogive and logistic regression. After showing a graph of both the normal ogive and the logistic regression curves, the authors follow up with an attention-grabbing and thought-provoking question: “Is this resemblance…of the logistic function to the normal ogive perhaps reflecting a deeper relationship?” Then, they come to the point that the normal ogive is practically equivalent to a logistic curve after rescaling the horizontal axis. People who are new to the logistic curve may find this way of illustrating help them memorize and internalize the key messages and explore more in this regard in the future.

Like any book of ambitious scope, this book has its limitations. Targeting a broad audience (from new learners of IRT to senior research scientists in methodology across vast disciplines), Raykov and Marcoulides present knowledge in a way that helps readers acquire knowledge of IRT against a backdrop of the general science of applied statistics. However, not everyone would find this approach appealing. A more advanced group of readers, such as senior scientists who are already familiar with some aspects of IRT, might prefer a condensed version that can let them quickly view the newer perspectives offered in this book. Meanwhile, the reiteration of the same concepts in some situations can appear redundant or confusing. For example, the normal ogive and CDF are presented twice in different parts of Chapter 2. Similar issues exist in Chapters 3 through 5. Also, texts in some areas of this book could have been more precise or consistent with rigorous reviewing and copyediting. For instance, the authors mention in Section 3.3 (p. 35) that readers should check CTT-based models in another section of the book (p. 37-42), which, however, covered classical FA in linear and nonlinear models instead. Also, the Stata codes in Footnote 1 of Chapter 5 (p. 71) cannot successfully run in Stata because several symbols are missing. In addition, many in-text references and brackets throughout the book were not formatted in the same way (e.g., p. 8-11).

Despite its limitations, A Course in Item Response Theory and Modeling with Stata is an unconventional masterpiece. Its generous and considerable exposition of the conceptual connections across varied terminologies sets it apart from any traditional texts about IRT. It might better serve the people who have taken an introductory course about IRT than more or less advanced readers. The primary reason is that the structure of this current edition, to some extent, mixes up-to-date perspectives about IRT with the older, relatively well-known knowledge, and it presents some key concepts progressively but iteratively. This structure is most effective with people who are comparatively new to the foundational concepts in IRT and would desire to know how IRT concepts are positioned relatively to other terminologies in applied statistics in the broad sense.

References

Albano, T. (2020). Introduction to educational and psychological measurement using R. Retrieved May 12, 2020 from https://www.thetaminusb.com/intro-measurement-r/

Cole, K. M., & Paek, I. (2017). PROC IRT: A SAS Procedure for item response theory. Applied Psychological Measurement, 41(4), 311–320. https://doi.org/10.1177/0146621616685062

DiTrapani, J., Rockwood, N., & Jeon, M. (2018). IRT in SPSS using the SPIRIT macro. Applied Psychological Measurement, 42(2), 173–174. https://doi.org/10.1177/0146621617733956

Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47. https://doi.org/10.1111/j.1745-3992.1993.tb00543.x

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: SAGE Publications, Inc.

Lord, F. M. (1952). A theory of test scores. Psychometric Monographs, 7, x, 84.

Lord, F. M. (1953). An application of confidence intervals and of maximum likelihood to the estimation of an examinee’s ability. Psychometrika, 18(1), 57–76. https://doi.org/10.1007/BF02289028

StataCorp LLC. (2015). Stata 14 available now. Stata News, 30(1). Retrieved April 1, 2020, from https://www.stata.com/stata-news/news30-1/

StataCorp LLC. (2019). Stata item response reference manual: Release 16. College Station, Texas: Stata Press. https://www.stata.com/manuals/irt.pdf

Zhao, Y., & Hambleton, R. (2009). Software for IRT analyses: Descriptions and features (Center for Educational Assessment Research Report No. 652). Amherst, MA: University of Massachusetts, Center for Educational Assessment. Retrieved May 12, 2020 from https://www.umass.edu/remp/software/CEA-652.ZH-IRTSoftware.pdf