An Empirical Evaluation of the Slip Correction in the Four Parameter Logistic Models With Computerized Adaptive Testing - Yen, Yung-Chin; Ho, Rong-Guey; Laio, Wen-Wei; Chen, Li-Ju; Kuo, Ching-Chin (Applied Psychological Measurement, 2012)
In a selected response test, aberrant responses such as careless errors and lucky guesses might cause error in ability estimation because these responses do not actually reflect the knowledge that examinees possess. In a computerized adaptive test (CAT), these aberrant responses could further cause serious estimation error due to dynamic item administration. To enhance the robust performance of CAT against aberrant responses, Barton and Lord proposed the four-parameter logistic (4PL) item response theory (IRT) model. However, most studies relevant to the 4PL IRT model were conducted based on simulation experiments. This study attempts to investigate the performance of the 4PL IRT model as a slip-correction mechanism with an empirical experiment. The results showed that the 4PL IRT model could not only reduce the problematic underestimation of the examinees’ ability introduced by careless mistakes in practical situations but also improve measurement efficiency.

Improving personality facet scores with multidimensional computerized adaptive testing: An illustration with the NEO PI-R - Makransky, G.; Mortensen, E. L.; Glas, C. A. W. (Assessment, 2012)

Investigating the Effect of Item Position in Computer-Based Tests - Li, Feiming; Cohen, Allan; Shen, Linjun (Journal of Educational Measurement, 2012)
Computer-based tests (CBTs) often use random ordering of items in order to minimize item exposure and reduce the potential for answer copying. Little research has been done, however, to examine item position effects for these tests. In this study, different versions of a Rasch model and different response time models were examined and applied to data from a CBT administration of a medical licensure examination. The models specifically were used to investigate whether item position affected item difficulty and item intensity estimates. Results indicated that the position effect was negligible.

Item Overexposure in Computerized Classification Tests Using Sequential Item Selection - Huebner, A. (Practical Assessment, Research & Evaluation, 2012)
Computerized classification tests (CCTs) often use sequential item selection which administers items according to maximizing psychometric information at a cut point demarcating passing and failing scores. This paper illustrates why this method of item selection leads to the overexposure of a significant number of items, and the performances of three different methods for controlling maximum item exposure rates in CCTs are compared. Specifically, the Sympson-Hetter, restricted, and item eligibility methods are examined in two studies realistically simulating different types of CCTs and are evaluated based upon criteria including classification accuracy, the number of items exceeding the desired maximum exposure rate, and test overlap. The pros and cons of each method are discussed from a practical perspective.

Item Selection and Ability Estimation Procedures for a Mixed-Format Adaptive Test - Ho, Tsung-Han; Dodd, Barbara G. (Applied Measurement in Education, 2012)

A Mixture Rasch Model–Based Computerized Adaptive Test for Latent Class Identification - Hong Jiao,; Macready, George; Liu, Junhui; Cho, Youngmi (Applied Psychological Measurement, 2012)
This study explored a computerized adaptive test delivery algorithm for latent class identification based on the mixture Rasch model. Four item selection methods based on the Kullback–Leibler (KL) information were proposed and compared with the reversed and the adaptive KL information under simulated testing conditions. When item separation was large, all item selection methods did not differ evidently in terms of accuracy in classifying examinees into different latent classes and estimating latent ability. However, when item separation was small, two methods with class-specific ability estimates performed better than the other two methods based on a single latent ability estimate across all latent classes. The three types of KL information distributions were compared. The KL and the reversed KL information could be the same or different depending on the ability level and the item difficulty difference between latent classes. Although the KL information and the reversed KL information were different at some ability levels and item difficulty difference levels, the use of the KL, the reversed KL, or the adaptive KL information did not affect the results substantially due to the symmetric distribution of item difficulty differences between latent classes in the simulated item pools. Item pool usage and classification convergence points were examined as well.

Multistage Computerized Adaptive Testing With Uniform Item Exposure - Edwards, Michael C.; Flora, David B.; Thissen, David (Applied Measurement in Education, 2012)

Panel Design Variations in the Multistage Test Using the Mixed-Format Tests - Kim, Jiseon; Chung, Hyewon; Dodd, Barbara G.; Park, Ryoungsun (Educational and Psychological Measurement, 2012)
This study compared various panel designs of the multistage test (MST) using mixed-format tests in the context of classification testing. Simulations varied the design of the first-stage module. The first stage was constructed according to three levels of test information functions (TIFs) with three different TIF centers. Additional computerized adaptive test (CAT) conditions provided baseline comparisons. Three passing rate conditions were also included. The various MST conditions using mixed-format tests were constructed properly and performed well. When the levels of TIFs at the first stage were higher, the simulations produced a greater number of correct classifications. CAT with the randomesque-10 procedure yielded comparable results to the MST with increased levels of TIFs. Finally, all MST conditions achieved better test security results compared with CAT’s maximum information conditions.

The Problem of Bias in Person Parameter Estimation in Adaptive Testing - Doebler, Anna (Applied Psychological Measurement, 2012)
It is shown that deviations of estimated from true values of item difficulty parameters, caused for example by item calibration errors, the neglect of randomness of item difficulty parameters, testlet effects, or rule-based item generation, can lead to systematic bias in point estimation of person parameters in the context of adaptive testing. This effect occurs even when the errors of the item difficulty parameters are themselves unbiased. Analytical calculations as well as simulation studies are discussed.

On the Reliability and Validity of a Numerical Reasoning Speed Dimension Derived From Response Times Collected in Computerized Testing - Davison, Mark L.; Semmes, Robert; Huang, Lan; Close, Catherine N. (Educational and Psychological Measurement, 2012)
Data from 181 college students were used to assess whether math reasoning item response times in computerized testing can provide valid and reliable measures of a speed dimension. The alternate forms reliability of the speed dimension was .85. A two-dimensional structural equation model suggests that the speed dimension is related to the accuracy of speeded responses. Speed factor scores were significantly correlated with performance on the ACT math scale. Results suggest that the speed dimension underlying response times can be reliably measured and that the dimension is related to the accuracy of performance under the pressure of time limits.

A Stochastic Method for Balancing Item Exposure Rates in Computerized Classification Tests - Huebner, Alan; Li, Zhushan (Applied Psychological Measurement, 2012)
Computerized classification tests (CCTs) classify examinees into categories such as pass/fail, master/nonmaster, and so on. This article proposes the use of stochastic methods from sequential analysis to address item overexposure, a practical concern in operational CCTs. Item overexposure is traditionally dealt with in CCTs by the Sympson-Hetter (SH) method, but this method is unable to restrict the exposure of the most informative items to the desired level. The authors’ new method of stochastic item exposure balance (SIEB) works in conjunction with the SH method and is shown to greatly reduce the number of overexposed items in a pool and improve overall exposure balance while maintaining classification accuracy comparable with using the SH method alone. The method is demonstrated using a simulation study.

A Stochastic Method for Balancing Item Exposure Rates in Computerized Classification Tests - Huebner, Alan; Li, Zhushan (Applied Psychological Measurement, 2012)
Computerized classification tests (CCTs) classify examinees into categories such as pass/fail, master/nonmaster, and so on. This article proposes the use of stochastic methods from sequential analysis to address item overexposure, a practical concern in operational CCTs. Item overexposure is traditionally dealt with in CCTs by the Sympson-Hetter (SH) method, but this method is unable to restrict the exposure of the most informative items to the desired level. The authors’ new method of stochastic item exposure balance (SIEB) works in conjunction with the SH method and is shown to greatly reduce the number of overexposed items in a pool and improve overall exposure balance while maintaining classification accuracy comparable with using the SH method alone. The method is demonstrated using a simulation study.

Termination Criteria in Computerized Adaptive Tests: Do Variable-Length CATs Provide Efficient and Effective Measurement? - Babcock, B.; Weiss, D. J. (Journal of Computerized Adaptive Testing, 2012)

Adaptive Item Calibration and Norming: Unique Considerations of a Global Deployment - Alexander Schwall; Evan Sinar (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: CAT; common item equating; Figural Reasoning Test; item calibration; norming

Applying computerized adaptive testing to the CES-D scale: A simulation study - Smits, N.; Cuijpers, P.; van Straten, A. (Psychiatry Research, 2011)
In this paper we studied the appropriateness of developing an adaptive version of the Center of Epidemiological Studies-Depression (CES-D, Radloff, 1977) scale. Computerized Adaptive Testing (CAT) involves the computerized administration of a test in which each item is dynamically selected from a pool of items until a pre-specified measurement precision is reached. Two types of analyses were performed using the CES-D responses of a large sample of adolescents (N=1392). First, it was shown that the items met the psychometric requirements needed for CAT. Second, CATs were simulated by using the existing item responses as if they had been collected adaptively. CATs selecting only a small number of items gave results which, in terms of depression measurement and criterion validity, were only marginally different from the results of full CES-D assessment. It was concluded that CAT is a very fruitful way of improving the efficiency of the CES-D questionnaire. The discussion addresses the strengths and limitations of the application of CAT in mental health research.
Psychiatry Res. 2011 Jan 3.

Applying computerized adaptive testing to the CES-D scale: A simulation study - Smits, N.; Cuijpers, P.; van Straten, A. (Psychiatry Research, 2011)

Better Data From Better Measurements Using Computerized Adaptive Testing - Weiss, D. J. (Journal of Methods and Measurement in the Social Sciences, 2011)
The process of constructing a fixed-length conventional test frequently focuses on maximizing internal consistency reliability by selecting test items that are of average difficulty and high discrimination (a ―peaked‖ test). The effect of constructing such a test, when viewed from the perspective of item response theory, is test scores that are precise for examinees whose trait levels are near the point at which the test is peaked; as examinee trait levels deviate from the mean, the precision of their scores decreases substantially. Results of a small simulation study demonstrate that when peaked tests are ―off target‖ for an examinee, their scores are biased and have spuriously high standard deviations, reflecting substantial amounts of error. These errors can reduce the correlations of these kinds of scores with other variables and adversely affect the results of standard statistical tests. By contrast, scores from adaptive tests are essentially unbiased and have standard deviations that are much closer to true values. Basic concepts of adaptive testing are introduced and fully adaptive computerized tests (CATs) based on IRT are described. Several examples of response records from CATs are discussed to illustrate how CATs function. Some operational issues, including item exposure, content balancing, and enemy items are also briefly discussed. It is concluded that because CAT constructs a unique test for examinee, scores from CATs will be more precise and should provide better data for social science research and applications.

Building Affordable CD-CAT Systems for Schools To Address Today's Challenges In Assessment - Chang, Hua-Hua (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: affordability; CAT; cost

catR: An R Package for Computerized Adaptive Testing - Magis, D.; Raîche, G. (Applied Psychological Measurement, 2011)
Computerized adaptive testing (CAT) is an active current research field in psychometrics and educational measurement. However, there is very little software available to handle such adaptive tasks. The R package catR was developed to perform adaptive testing with as much flexibility as possible, in an attempt to provide a developmental and testing platform to the interested user. Several item-selection rules and ability estimators are implemented. The item bank can be provided by the user or randomly generated from parent distributions of item parameters. Three stopping rules are available. The output can be graphically displayed.
Keywords: computer program; computerized adaptive testing; Estimation; Item Response Theory

A Comment on Early Student Blunders on Computer-Based Adaptive Tests - Green, Bert F. (Applied Psychological Measurement, 2011)
This article refutes a recent claim that computer-based tests produce biased scores for very proficient test takers who make mistakes on one or two initial items and that the ‘‘bias’’ can be reduced by using a four-parameter IRT model. Because the same effect occurs with pattern scores on nonadaptive tests, the effect results from IRT scoring, not from adaptive testing. Because very proficient test takers rarely err on items of middle difficulty, the so-called bias is one of selective data analysis. Furthermore, the apparently large score penalty for one error on an otherwise perfect response pattern is shown to result from the relative stretching of the IRT scale at very high and very low proficiencies. The recommended use of a four-parameter IRT model is shown to have drawbacks.

A Comment on Early Student Blunders on Computer-Based Adaptive Tests - Green, B. F. (Applied Psychological Measurement, 2011)
This article refutes a recent claim that computer-based tests produce biased scores for very proficient test takers who make mistakes on one or two initial items and that they can be reduced by using a four-parameter IRT model. Because the same effect occurs with pattern scores on nonadaptive tests, the effect results from IRT scoring, not from adaptive testing. Because very proficient test takers rarely err on items of middle difficulty, the so-called bias is one of selective data analysis. Furthermore, the apparently large score penalty for one error on an otherwise perfect response pattern is shown to result from the relative stretching of the IRT scale at very high and very low proficiencies. The recommended use of a four-parameter IRT model is shown to have drawbacks.

Computer adaptive testing for small scale programs and instructional systems - Rudner, L. M.; Guo, F. (Journal of Applied Testing Technology, 2011)
This study investigates measurement decision theory (MDT) as an underlying model for computer adaptive testing when the goal is to classify examinees into one of a finite number of groups. The first analysis compares MDT with a popular item response theory model and finds little difference in terms of the percentage of correct classifications. The second analysis examines the number of examinees needed to calibrate MDT item parameters and finds accurate classifications even with calibration sample sizes as small as 100 examinees.

Computerized adaptive assessment of personality disorder: Introducing the CAT–PD project - Simms, L. J.; Goldberg, L .R.; Roberts, J. E.; Watson, D.; Welte, J.; Rotterman, J. H. (Journal of Personality Assessment, 2011)
Assessment of personality disorders (PD) has been hindered by reliance on the problematic categorical model embodied in the most recent Diagnostic and Statistical Model of Mental Disorders (DSM), lack of consensus among alternative dimensional models, and inefficient measurement methods. This article describes the rationale for and early results from a multiyear study funded by the National Institute of Mental Health that was designed to develop an integrative and comprehensive model and efficient measure of PD trait dimensions. To accomplish these goals, we are in the midst of a 5-phase project to develop and validate the model and measure. The results of Phase 1 of the project—which was focused on developing the PD traits to be assessed and the initial item pool—resulted in a candidate list of 59 PD traits and an initial item pool of 2,589 items. Data collection and structural analyses in community and patient samples will inform the ultimate structure of the measure, and computerized adaptive testing will permit efficient measurement of the resultant traits. The resultant Computerized Adaptive Test of Personality Disorder (CAT–PD) will be well positioned as a measure of the proposed DSM–5 PD traits. Implications for both applied and basic personality research are discussed.

Computerized Adaptive Testing with the Zinnes and Griggs Pairwise Preference Ideal Point Model - Stark, Stephen; Chernyshenko, Oleksandr S. (International Journal of Testing, 2011)

Computerized Classification Testing Under the Generalized Graded Unfolding Model - Wang, Wen-Chung; Liu, Chen-Wei (Educational and Psychological Measurement, 2011)
The generalized graded unfolding model (GGUM) has been recently developed to describe item responses to Likert items (agree—disagree) in attitude measurement. In this study, the authors (a) developed two item selection methods in computerized classification testing under the GGUM, the current estimate/ability confidence interval method and the cut score/sequential probability ratio test method and (b) evaluated their accuracy and efficiency in classification through simulations. The results indicated that both methods were very accurate and efficient. The more points each item had and the fewer the classification categories, the more accurate and efficient the classification would be. However, the latter method may yield a very low accuracy in dichotomous items with a short maximum test length. Thus, if it is to be used to classify examinees with dichotomous items, the maximum text length should be increased.

Computerized Classification Testing Under the One-Parameter Logistic Response Model With Ability-Based Guessing - Wang, Wen-Chung; Huang, Sheng-Yun (Educational and Psychological Measurement, 2011)
The one-parameter logistic model with ability-based guessing (1PL-AG) has been recently developed to account for effect of ability on guessing behavior in multiple-choice items. In this study, the authors developed algorithms for computerized classification testing under the 1PL-AG and conducted a series of simulations to evaluate their performances. Four item selection methods (the Fisher information, the Fisher information with a posterior distribution, the progressive method, and the adjusted progressive method) and two termination criteria (the ability confidence interval [ACI] method and the sequential probability ratio test [SPRT]) were developed. In addition, the Sympson–Hetter online method with freeze (SHOF) was implemented for item exposure control. Major results include the following: (a) when no item exposure control was made, all the four item selection methods yielded very similar correct classification rates, but the Fisher information method had the worst item bank usage and the highest item exposure rate; (b) SHOF can successfully maintain the item exposure rate at a prespecified level, without compromising substantial accuracy and efficiency in classification; (c) once SHOF was implemented, all the four methods performed almost identically; (d) ACI appeared to be slightly more efficient than SPRT; and (e) in general, a higher weight of ability in guessing led to a slightly higher accuracy and efficiency, and a lower forced classification rate.

Content range and precision of a computer adaptive test of upper extremity function for children with cerebral palsy - Montpetit, K.; Haley, S.; Bilodeau, N.; Ni, P.; Tian, F.; Gorton, G., 3rd; Mulcahey, M. J. (Physical & Occupational Therapy in Pediatrics, 2011)
This article reports on the content range and measurement precision of an upper extremity (UE) computer adaptive testing (CAT) platform of physical function in children with cerebral palsy. Upper extremity items representing skills of all abilities were administered to 305 parents. These responses were compared with two traditional standardized measures: Pediatric Outcomes Data Collection Instrument and Functional Independence Measure for Children. The UE CAT correlated strongly with the upper extremity component of these measures and had greater precision when describing individual functional ability. The UE item bank has wider range with items populating the lower end of the ability spectrum. This new UE item bank and CAT have the capability to quickly assess children of all ages and abilities with good precision and, most importantly, with items that are meaningful and appropriate for their age and level of physical function.
Montpetit, KathleenHaley, StephenBilodeau, NathalieNi, PengshengTian, FengGorton, George 3rdMulcahey, M JEnglandPhys Occup Ther Pediatr. 2011 Feb;31(1):90-102. Epub 2010 Oct 13.

Continuous Testing (an avenue for CAT research) - G. Gage Kingsbury (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Publishing an Adaptive Test Write Items Field Test Items Select an Operational Pool Publish Pool Distribute Pool Administer Tests Rinse and Repeat Problems with Publishing The static pool Restricts content Restricts psychometrics Restricts corrections Structures item exposure The publication process Creates logistic problems Slows response to new content Item filtration – An alternative to test publishing Realtime test creation Begins with entire item pool Removes items until one item is chosen to administer Research Questions Since filtering adds constraints, what is the psychometric impact? Can we use filtering to control item drift? How does filtering interact with item selection?
Keywords: CAT; item filter; item filtration

Creating a K-12 Adaptive Test: Examining the Stability of Item Parameter Estimates and Measurement Scales - Kingsbury, G. G.; Wise, S. L. (Journal of Applied Testing Technology, 2011)
Development of adaptive tests used in K-12 settings requires the creation of stable measurement scales to measure the growth of individual students from one grade to the next, and to measure change in groups from one year to the next. Accountability systems like No Child Left Behind require stable measurement scales so that accountability has meaning across time. This study examined the stability of the measurement scales used with the Measures of Academic Progress. Difficulty estimates for test questions from the reading and mathematics scales were examined over a period ranging from 7 to 22 years. Results showed high correlations between item difficulty estimates from the time at which they where originally calibrated and the current calibration. The average drift in item difficulty estimates was less than .01 standard deviations. The average impact of change in item difficulty estimates was less than the smallest reported difference on the score scale for two actual tests. The findings of the study indicate that an IRT scale can be stable enough to allow consistent measurement of student achievement.

Cross-cultural development of an item list for computer-adaptive testing of fatigue in oncological patients - Giesinger, J. M.; Petersen, M. A.; Groenvold, M.; Aaronson, N. K.; Arraras, J. I.; Conroy, T.; Gamper, E. M.; Kemmler, G.; King, M. T.; Oberguggenberger, A. S.; Velikova, G.; Young, T.; et al. (Health and Quality of Life Outcomes, 2011)
ABSTRACT: INTRODUCTION: Within an ongoing project of the EORTC Quality of Life Group, we are developing computerized adaptive test (CAT) measures for the QLQ-C30 scales. These new CAT measures are conceptualised to reflect the same constructs as the QLQ-C30 scales. Accordingly, the Fatigue-CAT is intended to capture physical and general fatigue. METHODS: The EORTC approach to CAT development comprises four phases (literature search, operationalisation, pre-testing, and field testing). Phases I-III are described in detail in this paper. A literature search for fatigue items was performed in major medical databases. After refinement through several expert panels, the remaining items were used as the basis for adapting items and/or formulating new items fitting the EORTC item style. To obtain feedback from patients with cancer, these English items were translated into Danish, French, German, and Spanish and tested in the respective countries. RESULTS: Based on the literature search a list containing 588 items was generated. After a comprehensive item selection procedure focusing on content, redundancy, item clarity and item difficulty a list of 44 fatigue items was generated. Patient interviews (n=52) resulted in 12 revisions of wording and translations. DISCUSSION: The item list developed in phases I-III will be further investigated within a field-testing phase (IV) to examine psychometric characteristics and to fit an item response theory model. The Fatigue CAT based on this item bank will provide scores that are backward-compatible to the original QLQ-C30 fatigue scale.
Health Qual Life Outcomes. 2011 Mar 29;9(1):19.

Design of a Computer-Adaptive Test to Measure English Literacy and Numeracy in the Singapore Workforce: Considerations, Benefits, and Implications - Jacobsen, J.; Ackermann, R.; Egüez, J.; Ganguli, D.; Rickard, P.; Taylor, L. (Journal of Applied Testing Technology, 2011)
A computer adaptive test CAT) is a delivery methodology that serves the larger goals of the assessment system in which it is embedded. A thorough analysis of the assessment system for which a CAT is being designed is critical to ensure that the delivery platform is appropriate and addresses all relevant complexities. As such, a CAT engine must be designed to conform to the validity and reliability of the overall system. This design takes the form of adherence to the assessment goals and objectives of the adaptive assessment system. When the assessment is adapted for use in another country, consideration must be given to any necessary revisions including content differences. This article addresses these considerations while drawing, in part, on the process followed in the development of the CAT delivery system designed to test English language workplace skills for the Singapore Workforce Development Agency. Topics include item creation and selection, calibration of the item pool, analysis and testing of the psychometric properties, and reporting and interpretation of scores. The characteristics and benefits of the CAT delivery system are detailed as well as implications for testing programs considering the use of a CAT delivery system.

Detecting DIF between Conventional and Computerized Adaptive Testing: A Monte Carlo Study - Barth B. Riley; Adam C. Carle (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
A comparison od two procedures, Modified Robust Z and 95% Credible Interval, were compared in a Monte Carlo study. Both procedures evidenced adequate control of false positive DIF results. Exception: low difficulty items (< -2.5 logits). Not significantly affected by % of DIF items. Was affected by mean trait level difference. 95% Credibility Interval evidenced slightly higher power to detect DIF, but also higher false positive rate.
Keywords: 95% Credible Interval; CAT; DIF; differential item function; modified robust Z statistic; Monte Carlo methodologies

Effects of Different Computerized Adaptive Testing Strategies of Recovery of Ability - Kalender, I. (THE GRADUATE SCHOOL OF NATURAL AND APPLIED SCIENCES OF MIDDLE EAST TECHNICAL UNIVERSITY, 2011)
The purpose of the present study is to compare ability estimations obtained from computerized adaptive testing (CAT) procedure with the paper and pencil test administration results of Student Selection Examination (SSE) science subtest considering different ability estimation methods and test termination rules. There are two phases in the present study. In the first phase, a post-hoc simulation was conducted to find out relationships between examinee ability levels estimated by CAT and paper and pencil test versions of the SSE. Maximum Likelihood Estimation and Expected A Posteriori were used as ability estimation method. Test termination rules were standard error threshold and fixed number of items. Second phase was actualized by implementing a CAT administration to a group of examinees to investigate performance of CAT administration in an environment other than simulated administration. Findings of post-hoc simulations indicated CAT could be implemented by using Expected A Posteriori estimation method with standard error threshold value of 0.30 or higher for SSE. Correlation between ability estimates obtained by CAT and real SSE was found to be 0.95. Mean of number of items given to examinees by CAT is 18.4. Correlation between live CAT and real SSE ability estimations was 0.74. Number of items used for CAT administration is approximately 50% of the items in paper and pencil SSE science subtest. Results indicated that CAT for SSE science subtest provided ability estimations with higher reliability with fewer items compared to paper and pencil format.

A framework for the development of computerized adaptive tests - Thompson, N. A.; Weiss, D. J. (Practical Assessment Research & Evaluation, 2011)
A substantial amount of research has been conducted over the past 40 years on technical aspects of computerized adaptive testing (CAT), such as item selection algorithms, item exposure controls, and termination criteria. However, there is little literature providing practical guidance on the development of a CAT. This paper seeks to collate some of the available research methodologies into a general framework for the development of any CAT assessment.

From Reliability to Validity: Expanding Adaptive Testing Practice to Find the Most Valid Score for Each Test Taker - Steven L. Wise (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
CAT is an exception to the traditional conception of validity. It is one of the few examples of individualized testing. Item difficulty is tailored to each examinee. The intent, however, is increased efficiency. Focus on reliability (reduced standard error); Equivalence with paper & pencil tests is valued; Validity is enhanced through improved reliability. How Else Might We Individualize Testing Using CAT? By addressing construct-irrelevant factors influencing individual test scores (usually in negatively biased ways). Individual Score Validity (ISV) – how free is a particular score from construct-irrelevant factors (often called construct-irrelevant variance, or CIV). An ISV-Based View of Validity Test Event -- An examinee encounters a series of items in a particular context. •All 3 elements (examinee, items, context) are potential sources of CIV. Examples: Test anxiety (examinee) Amount/difficulty of reading required (item) Test stakes (context) ISV can be affected by all 3 elements. CAT Goal: individualize testing to address CIV threats to score validity (i.e., maximize ISV). Some Research Issues: What are some innovative methods for expanding CAT that address ISV threats while preserving measurement of the target construct? How might CAT help address the ISV challenges posed by test anxiety? How should policy-makers deal with scores that have been shown to have low ISV?
Keywords: CAT; CIV; construct-irrelevant variance; Individual Score Validity; ISV; low test taking motivation; Reliability; validity

A Heuristic Of CAT Item Selection Procedure For Testlets - Yuehmei Chien; David Shin; Walter Denny Way (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: CAT; shadow test; testlets

High-throughput Health Status Measurement using CAT in the Era of Personal Genomics: Opportunities and Challenges - Eswar Krishnan (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: CAT; health applications; PROMIS

Hypothetical use of multidimensional adaptive testing for the assessment of student achievement in PISA. - Frey, A.; Seitz, N-N. (Educational and Psychological Measurement, 2011)

Impact of Item Drift on Candidate Ability Estimation - Sarah Hagge; Ada Woo; Phil Dickison (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
For large operational pools, candidate ability estimates appear robust to item drift, especially under conditions that may represent ‘normal’ amounts of drift. Even with ‘extreme’ conditions of drift (e.g., 20% of items drifting 1.00 logits), decision consistency was still high.
Keywords: item drift

Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): depression, anxiety, and anger - Pilkonis, P. A.; Choi, S. W.; Reise, S. P.; Stover, A. M.; Riley, W. T.; Cella, D. (Assessment, 2011)

Item Selection Criteria With Practical Constraints for Computerized Classification Testing - Lin, Chuan-Ju (Educational and Psychological Measurement, 2011)
This study compares four item selection criteria for a two-category computerized classification testing: (1) Fisher information (FI), (2) Kullback—Leibler information (KLI), (3) weighted log-odds ratio (WLOR), and (4) mutual information (MI), with respect to the efficiency and accuracy of classification decision using the sequential probability ratio test as well as the extent of item usage. The comparability of the four item selection criteria are examined primarily under three types of item selection conditions: (1) using only the four item selection algorithms, (2) using the four item selection algorithms and content balancing control, and (3) using the four item selection algorithms, content balancing control, and item exposure control. The comparability of the four item selection criteria is also evaluated in two types of proficiency distribution and three levels of indifference region width. The results show that the differences of the four item selection criteria are washed out as more realistic constraints are imposed. Moreover, within two-category classification testing, the use of MI does not necessarily generate greater efficiency than FI, WLOR, and KLI, although MI might seem attractive for its general form of formula in item selection.

Item Selection Methods based on Multiple Objective Approaches for Classification of Respondents into Multiple Levels - Maaike van Groen; Theo Eggen; Bernard Veldkamp (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Is it possible to develop new item selection methods which take advantage of the fact that we want to classify into multiple categories? New methods: Taking multiple points on the ability scale into account; Based on multiple objective approaches. Conclusions Sequential Classification Tests higher ATL than Adaptive Classification Tests Sequential Classification Tests slightly lower PCD than Adaptive Classification Tests Results also hold with three and four cutting points
Keywords: adaptive classification test; CAT; item selection; sequential classification test

JATT Special Issue on Adaptive Testing: Welcome and Overview - Thompson, N. A. (Journal of Applied Testing Technology, 2011)

Measuring Individual Growth With Conventional and Adaptive Tests - Weiss, D. J.; Von Minden, S. (Journal of Methods and Measurement in the Social Sciences, 2011)

Moving beyond Efficiency to Allow CAT to Provide Better Diagnostic Information - Brian D. Bontempo (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Future CATs will provide better diagnostic information to –Examinees –Regulators, Educators, Employers –Test Developers This goal will be accomplished by –Smart CATs which collect additional information during the test –Psychomagic The time is now for Reporting
Keywords: CAT; dianostic information; MIRT; Multiple unidimensional scales; psychomagic; smart CAT

National Tests in Denmark – CAT as a Pedagogic Tool - Wandall, J. (Journal of Applied Testing Technology, 2011)
Testing and test results can be used in different ways. They can be used for regulation and control, but they can also be a pedagogic tool for assessment of student proficiency in order to target teaching, improve learning and facilitate local pedagogical leadership. To serve these purposes the test has to be used for low stakes purposes, and to ensure this, the Danish National test results are made strictly confidential by law. The only test results that are made public are the overall national results. Because of the test design, test results are directly comparable, offering potential for monitoring added value and developing new ways of using test results in a pedagogical context. This article gives the background and status for the development of the Danish national tests, describes what is special about these tests (e.g., Information Technology [IT]-based, 3 tests in 1, adaptive), how the national test are carried out, and what is tested. Furthermore, it describes strategies for disseminating the results to the pupil, parents, teacher, headmaster and municipality; and how the results can be used by the teacher and headmaster.

A new adaptive testing algorithm for shortening health literacy assessments - Kandula, S.; Ancker, J.S.; Kaufman, D.R.; Currie, L.M.; Qing, Z.-T. (BMC Medical Informatics and Decision Making, 2011)
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3178473/?tool=pmcentrez

A New Stopping Rule for Computerized Adaptive Testing - Choi, Seung W.; Grady, Matthew W.; Dodd, Barbara G. (Educational and Psychological Measurement, 2011)
The goal of the current study was to introduce a new stopping rule for computerized adaptive testing (CAT). The predicted standard error reduction (PSER) stopping rule uses the predictive posterior variance to determine the reduction in standard error that would result from the administration of additional items. The performance of the PSER was compared with that of the minimum standard error stopping rule and a modified version of the minimum information stopping rule in a series of simulated adaptive tests, drawn from a number of item pools. Results indicate that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant.

Optimal Calibration Designs for Computerized Adaptive Testing - Angela Verschoor (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Optimaztion How can we exploit the advantages of Balanced Block Design while keeping the logistics manageable? Maximize number of item pairs Subject to maximum number of test booklets Subject to other constraints Homogeneous Designs: Overlap between test booklets as regular as possible Conclusions: Establish overlaps as regular as possible between all test booklets Or, at least as many test booklets as possible
Keywords: balanced block design; CAT; item calibration; optimization; Rasch

A Paradigm for Multinational Adaptive Testing - A Zara (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Impact of Issues in “Exported” Adaptive Testing Exam content issues Validity issues/construct differences Method bias DIF (item performance) Interpretation of scores “what do they mean?” Goal is construct equivalency in the new environment Research Questions How can we assure that the constructs being measured across the geographies are equivalent? Do we need to assure that the educational opportunities are “equivalent” across geographies? Do standard CAT content balancing constraints work? Computerized testing is common in the US, is there a need to re-visit the basic research on modality comparability in different geographies?
Keywords: CAT; multinational adaptive testing

Polytomous Adaptive Classification Testing: Effects of Item Pool Size, Test Termination Criterion, and Number of Cutscores - Gnambs, Timo; Batinic, Bernad (Educational and Psychological Measurement, 2011)
Computer-adaptive classification tests focus on classifying respondents in different proficiency groups (e.g., for pass/fail decisions). To date, adaptive classification testing has been dominated by research on dichotomous response formats and classifications in two groups. This article extends this line of research to polytomous classification tests for two- and three-group scenarios (e.g., inferior, mediocre, and superior proficiencies). Results of two simulation experiments with generated and real responses (N = 2,000) to established personality scales of different length (12, 20, or 29 items) demonstrate that adaptive item presentations significantly reduce the number of items required to make such classification decisions while maintaining a consistent classification accuracy. Furthermore, the simulations highlight the importance of the selected test termination criterion, which has a significant impact on the average test length.

Practitioner’s Approach to Identify Item Drift in CAT - Huijuan Meng; Susan Steinkamp; Paul Jones; Joy Matthews-Lopez (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: CUSUM method; G2 statistic; IPA; item drift; item parameter drift; Lord's chi-square statistic; Raju's NCDIF

Restrictive Stochastic Item Selection Methods in Cognitive Diagnostic Computerized Adaptive Testing - Wang, Chun; Chang, Hua-Hua; Huebner, Alan (Journal of Educational Measurement, 2011)
This paper proposes two new item selection methods for cognitive diagnostic computerized adaptive testing: the restrictive progressive method and the restrictive threshold method. They are built upon the posterior weighted Kullback-Leibler (KL) information index but include additional stochastic components either in the item selection index or in the item selection procedure. Simulation studies show that both methods are successful at simultaneously suppressing overexposed items and increasing the usage of underexposed items. Compared to item selection based upon (1) pure KL information and (2) the Sympson-Hetter method, the two new methods strike a better balance between item exposure control and measurement accuracy. The two new methods are also compared with Barrada et al.'s (2008) progressive method and proportional method.

Small-Sample Shadow Testing - Wallace Judd (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Keywords: CAT; shadow test

A Test Assembly Model for MST - Angela Verschoor; Ingrid Radtke; Theo Eggen (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
This study is just a short exploration in the matter of optimization of a MST. It is extremely hard or maybe impossible to chart influence of item pool and test specifications on optimization process. Simulations are very helpful in finding an acceptable MST.
Keywords: CAT; mst; multistage testing; Rasch; routing; tif

Unproctored Internet test verification: Using adaptive confirmation testing - Makransky, G.; Glas, C. A. W. (Organizational Research Methods, 2011)

The Use of Decision Trees for Adaptive Item Selection and Score Estimation - Barth B. Riley; Rodney Funk; Michael L. Dennis; Richard D. Lennox; Matthew Finkelman (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
Conducted post-hoc simulations comparing the relative efficiency, and precision of decision trees (using CHAID and CART) vs. IRT-based CAT. Measure: Global Appraisal of Individual Needs (GAIN) Substance Problem Scale (16 items) Past-year symptom count (SPSy) Recency of symptom scale (SPSr) Conclusions Decision tree methods were more efficient than CAT CART for dichotomous items (SPSy) CHAID for polytomous items (SPSr) Score bias was low in all conditions, particularly for decision trees using dichotomous items In early stages of administration, decision trees provided slightly higher correlations with the full scale and lower RMSE values. But,... CAT outperformed decision tree methods in later stages of administration. CAT also outperformed decision trees with respect to sensitivity to group differences as measured by effect size. Conclusions CAT selects items based on two criteria: Item location relative to current estimate of theta, Item discrimination Decision Trees select items that best discriminate between groups defined by the total score. CAT is optimal only when trait level is well estimated. Findings suggest that combining decision tree followed by CAT item selection may be advantageous.
Keywords: adaptive item selection; CAT; decision tree

Using Item Response Theory and Adaptive Testing in Online Career Assessment - Betz, Nancy E.; Turner, Brandon M. (Journal of Career Assessment, 2011)
The present article describes the potential utility of item response theory (IRT) and adaptive testing for scale evaluation and for web-based career assessment. The article describes the principles of both IRT and adaptive testing and then illustrates these with reference to data analyses and simulation studies of the Career Confidence Inventory (CCI). The kinds of information provided by IRT are shown to give a more precise look at scale quality across the trait continuum and also to permit the use of adaptive testing, where the items administered are tailored to the individual being tested. Such tailoring can significantly reduce testing time while maintaining high quality of measurement. This efficiency is especially useful when multiscale inventories and/or a large number of scales are to be administered. Readers are encouraged to consider using these advances in career assessment.

Walking the Tightrope: Using Better Content Control to Improve CAT - Kathleen A. Gialluca (Annual Conference of the International Association for Computerized Adaptive Testing, 2011)
All testing involves a balance between measurement precision and content considerations. CAT item-selection algorithms have evolved to accommodate content considerations. Reviews CAT evolution including: Original/”Pure” adaptive exams, Constrained CAT, Weighted-deviations method, Shadow-Test Approach, Testlets instead of fully adapted tests, Administration of one item may preclude the administration of other item(s), and item relationships. Research Questions How do the (new) item-selection algorithms perform in terms of measurement precision? item exposure? item-pool usage? Do we know the best way to incorporate item sets into adaptive exams? Consider exams with multiple item types. Must the item types be treated separately in a CAT, or can we adapt exams across item types?
Keywords: CAT; CAT evolution; test content

Adaptive Mastery Testing Using a Multidimensional IRT Model - Glas, C. A. W.; Vos, H. J. (Elements of Adaptive Testing, 2010)

Adaptive Tests for Measuring Anxiety and Depression - Walter, O. B. (Elements of Adaptive Testing, 2010)

Assembling an Inventory of Multistage Adaptive Testing Systems - Breithaupt, K; Ariel, A.; Hare, D. R. (Elements of Adaptive Testing, 2010)

An automatic online calibration design in adaptive testing - Makransky, G.; Glas, C. A. W. (Journal of Applied Testing Technology, 2010)

Bayesian item selection in constrained adaptive testing - Veldkamp, B. P. (Psicologica, 2010)
Application of Bayesian item selection criteria in computerized adaptive testing might result in improvement of bias and MSE of the ability estimates. The question remains how to apply Bayesian item selection criteria in the context of constrained adaptive testing, where large numbers of specifications have to be taken into account in the item selection process. The Shadow Test Approach is a general purpose algorithm for administering constrained CAT. In this paper it is shown how the approach can be slightly modified to handle Bayesian item selection criteria. No differences in performance were found between the shadow test approach and the modifiedapproach. In a simulation study of the LSAT, the effects of Bayesian item selection criteria are illustrated. The results are compared to item selection based on Fisher Information. General recommendations about the use of Bayesian item selection criteria are provided.
Keywords: computerized adaptive testing

A Comparison of Content-Balancing Procedures for Estimating Multiple Clinical Domains in Computerized Adaptive Testing: Relative Precision, Validity, and Detection of Persons With Misfitting Responses - Barth B. Riley; Michael L. Dennis; Conrad, Kendon J. (Applied Psychological Measurement, 2010)
This simulation study sought to compare four different computerized adaptive testing (CAT) content-balancing procedures designed for use in a multidimensional assessment with respect to measurement precision, symptom severity classification, validity of clinical diagnostic recommendations, and sensitivity to atypical responding. The four content-balancing procedures were (a) no content balancing, (b) screener-based, (c) mixed (screener plus content balancing), and (d) full content balancing. In full content balancing and in mixed content balancing following administration of the screener items, item selection was based on (a) whether the target number of items for the item’s subscale was reached and (b) the item’s information function. Mixed and full content balancing provided the best representation of items from each of the main subscales of the Internal Mental Distress Scale. These procedures also resulted in higher CAT to full-scale correlations for the Trauma and Homicidal/Suicidal Thought subscales and improved detection of atypical responding.

A comparison of content-balancing procedures for estimating multiple clinical domains in computerized adaptive testing: Relative precision, validity, and detection of persons with misfitting responses - Riley, B. B.; Dennis, M. L.; Conrad, K. J. (Applied Psychological Measurement, 2010)
This simulation study sought to compare four different computerized adaptive testing (CAT) content-balancing procedures designed for use in a multidimensional assessment with respect to measurement precision, symptom severity classification, validity of clinical diagnostic recommendations, and sensitivity to atypical responding. The four content-balancing procedures were (a) no content balancing, (b) screener-based, (c) mixed (screener plus content balancing), and (d) full content balancing. In full content balancing and in mixed content balancing following administration of the screener items, item selection was based on (a) whether the target numberof items for the item’s subscale was reached and (b) the item’s information function. Mixed and full content balancing provided the best representation of items from each of the main subscales of the Internal Mental Distress Scale. These procedures also resulted in higher CAT to full-scale correlations for the Trauma and Homicidal/Suicidal Thought subscales and improved detection of atypical responding.Keywords

A Comparison of Item Selection Techniques for Testlets - Murphy, Daniel L.; Dodd, Barbara G.; Vaughn, Brandon K. (Applied Psychological Measurement, 2010)
This study examined the performance of the maximum Fisher’s information, the maximum posterior weighted information, and the minimum expected posterior variance methods for selecting items in a computerized adaptive testing system when the items were grouped in testlets. A simulation study compared the efficiency of ability estimation among the item selection techniques under varying conditions of local-item dependency when the response model was either the three-parameter-logistic item response theory or the three-parameter-logistic testlet response theory. The item selection techniques performed similarly within any particular condition, the practical implications of which are discussed within the article.

Computerized adaptive testing based on decision trees - Ueno, M.; Songmuang, P. (10th IEEE International Conference on Advanced Learning Technologies, 2010)

Constrained Adaptive Testing with Shadow Tests - van der Linden, W. J. (Elements of Adaptive Testing, 2010)

Designing and Implementing a Multistage Adaptive Test: The Uniform CPA Exam - Melican, G.J.; Breithaupt, K; Zhang, Y. (Elements of Adaptive Testing, 2010)

Designing Item Pools for Adaptive Testing - Veldkamp, B. P.; van der Linden, W. J. (Elements of Adaptive Testing, 2010)

Designing item pools to optimize the functioning of a computerized adaptive test - Reckase, M. D. (Psychological Test and Assessment Modeling, 2010)
Computerized adaptive testing (CAT) is a testing procedure that can result in improved precision for a specified test length or reduced test length with no loss of precision. However, these attractive psychometric features of CATs are only achieved if appropriate test items are available for administration. This set of test items is commonly called an “item pool.” This paper discusses the optimal characteristics for an item pool that will lead to the desired properties for a CAT. Then, a procedure is described for designing the statistical characteristics of the item parameters for an optimal item pool within an item response theory framework. Because true optimality is impractical, methods for achieving practical approximations to optimality are described. The results of this approach are shown for an operational testing program including comparisons to the results from the item pool currently used in that testing program.Key

Detecting Person Misfit in Adaptive Testing - Meijer, R. R.; van Krimpen-Stoop, E. M. L. A. (Elements of Adaptive Testing, 2010)

Detection of aberrant item score patterns in computerized adaptive testing: An empirical example using the CUSUM - Egberink, I. J. L.; Meijer, R. R.; Veldkamp, B. P.; Schakel, L.; Smid, N. G. (Personality and Individual Differences, 2010)
The scalability of individual trait scores on a computerized adaptive test (CAT) was assessed through investigating the consistency of individual item score patterns. A sample of N = 428 persons completed a personality CAT as part of a career development procedure. To detect inconsistent item score patterns, we used a cumulative sum (CUSUM) procedure. Combined information from the CUSUM, other personality measures, and interviews showed that similar estimated trait values may have a different interpretation.Implications for computer-based assessment are discussed.
Keywords: CAT; computerized adaptive testing; CUSUM approach; person Fit

Deterioro de parámetros de los ítems en tests adaptativos informatizados: estudio con eCAT [Item parameter drift in computerized adaptive testing: Study with eCAT] - Abad, F. J.; Olea, J.; Aguado, D.; Ponsoda, V.; Barrada, J (Psicothema, 2010)
En el presente trabajo se muestra el análisis realizado sobre un Test Adaptativo Informatizado (TAI) diseñado para la evaluación del nivel de inglés, denominado eCAT, con el objetivo de estudiar el deterioro de parámetros (parameter drift) producido desde la calibración inicial del banco de ítems. Se ha comparado la calibración original desarrollada para la puesta en servicio del TAI (N= 3224) y la calibración actual obtenida con las aplicaciones reales del TAI (N= 7254). Se ha analizado el Funcionamiento Diferencial de los Ítems (FDI) en función de los parámetros utilizados y se ha simulado el impacto que sobre el nivel de rasgo estimado tiene la variación en los parámetros. Los resultados muestran que se produce especialmente un deterioro de los parámetros a y c, que hay unimportante número de ítems del banco para los que existe FDI y que la variación de los parámetros produce un impacto moderado en la estimación de θ de los evaluados con nivel de inglés alto. Se concluye que los parámetros de los ítems se han deteriorado y deben ser actualizados.Item parameter drift in computerized adaptive testing: Study with eCAT. This study describes the parameter drift analysis conducted on eCAT (a Computerized Adaptive Test to assess the written English level of Spanish speakers). The original calibration of the item bank (N = 3224) was compared to a new calibration obtained from the data provided by most eCAT operative administrations (N =7254). A Differential Item Functioning (DIF) study was conducted between the original and the new calibrations. The impact that the new parameters have on the trait level estimates was obtained by simulation. Results show that parameter drift is found especially for a and c parameters, an important number of bank items show DIF, and the parameter change has a moderate impact on high-level-English θ estimates. It is then recommended to replace the original estimates by the new set. by the new set.
Abad, Francisco JOlea, JulioAguado, DavidPonsoda, VicenteBarrada, Juan REnglish AbstractSpainPsicothemaPsicothema. 2010 May;22(2):340-7.
Keywords: *Software; Educational Measurement/*methods/*statistics & numerical data; Humans; Language

Development and evaluation of a confidence-weighting computerized adaptive testing - Yen, Y. C.; Ho, R. G.; Chen, L. J.; Chou, K. Y.; Chen, Y. L. (Educational Technology & Society, 2010)

Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments - Buysse, D. J.; Yu, L.; Moul, D. E.; Germain, A.; Stover, A.; Dodds, N. E.; Johnston, K. L.; Shablesky-Cade, M. A.; Pilkonis, P. A. (Sleep, 2010)
STUDY OBJECTIVES: To develop an archive of self-report questions assessing sleep disturbance and sleep-related impairments (SRI), to develop item banks from this archive, and to validate and calibrate the item banks using classic validation techniques and item response theory analyses in a sample of clinical and community participants. DESIGN: Cross-sectional self-report study. SETTING: Academic medical center and participant homes. PARTICIPANTS: One thousand nine hundred ninety-three adults recruited from an Internet polling sample and 259 adults recruited from medical, psychiatric, and sleep clinics. INTERVENTIONS: None. MEASUREMENTS AND RESULTS: This study was part of PROMIS (Patient-Reported Outcomes Information System), a National Institutes of Health Roadmap initiative. Self-report item banks were developed through an iterative process of literature searches, collecting and sorting items, expert content review, qualitative patient research, and pilot testing. Internal consistency, convergent validity, and exploratory and confirmatory factor analysis were examined in the resulting item banks. Factor analyses identified 2 preliminary item banks, sleep disturbance and SRI. Item response theory analyses and expert content review narrowed the item banks to 27 and 16 items, respectively. Validity of the item banks was supported by moderate to high correlations with existing scales and by significant differences in sleep disturbance and SRI scores between participants with and without sleep disorders. CONCLUSIONS: The PROMIS sleep disturbance and SRI item banks have excellent measurement properties and may prove to be useful for assessing general aspects of sleep and SRI with various groups of patients and interventions.
Buysse, Daniel JYu, LanMoul, Douglas EGermain, AnneStover, AngelaDodds, Nathan EJohnston, Kelly LShablesky-Cade, Melissa APilkonis, Paul AAR052155/AR/NIAMS NIH HHS/United StatesU01AR52155/AR/NIAMS NIH HHS/United StatesU01AR52158/AR/NIAMS NIH HHS/United StatesU01AR52170/AR/NIAMS NIH HHS/United StatesU01AR52171/AR/NIAMS NIH HHS/United StatesU01AR52177/AR/NIAMS NIH HHS/United StatesU01AR52181/AR/NIAMS NIH HHS/United StatesU01AR52186/AR/NIAMS NIH HHS/United StatesResearch Support, N.I.H., ExtramuralValidation StudiesUnited StatesSleepSleep. 2010 Jun 1;33(6):781-92.
Keywords: *Outcome Assessment (Health Care); *Self Disclosure; Adult; Aged; Aged, 80 and over; Cross-Sectional Studies; Factor Analysis, Statistical; Female; Humans; Male; Middle Aged; Psychometrics; Questionnaires; Reproducibility of Results; Sleep Disorders/*diagnosis; Young Adult

Development of computerized adaptive testing (CAT) for the EORTC QLQ-C30 physical functioning dimension - Petersen, M. A.; Groenvold, M.; Aaronson, N. K.; Chie, W. C.; Conroy, T.; Costantini, A.; Fayers, P.; Helbostad, J.; Holzner, B.; Kaasa, S.; Singer, S.; Velikova, G.; et al. (Quality of Life Research, 2010)
PURPOSE: Computerized adaptive test (CAT) methods, based on item response theory (IRT), enable a patient-reported outcome instrument to be adapted to the individual patient while maintaining direct comparability of scores. The EORTC Quality of Life Group is developing a CAT version of the widely used EORTC QLQ-C30. We present the development and psychometric validation of the item pool for the first of the scales, physical functioning (PF). METHODS: Initial developments (including literature search and patient and expert evaluations) resulted in 56 candidate items. Responses to these items were collected from 1,176 patients with cancer from Denmark, France, Germany, Italy, Taiwan, and the United Kingdom. The items were evaluated with regard to psychometric properties. RESULTS: Evaluations showed that 31 of the items could be included in a unidimensional IRT model with acceptable fit and good content coverage, although the pool may lack items at the upper extreme (good PF). There were several findings of significant differential item functioning (DIF). However, the DIF findings appeared to have little impact on the PF estimation. CONCLUSIONS: We have established an item pool for CAT measurement of PF and believe that this CAT instrument will clearly improve the EORTC measurement of PF.
Qual Life Res. 2010 Oct 23.

Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms - Choi, S.; Reise, S. P.; Pilkonis, P. A.; Hays, R. D.; Cella, D. (Quality of Life Research, 2010)

Elements of Adaptive Testing - van der Linden, W. J.; Glas, C. A. W. (Springer, 2010)

Estimation of the Parameters in an Item-Cloning Model for Adaptive Testing - Glas, C. A. W.; van der Linden, W. J.; Geerlings, H. (Elements of Adaptive Testing, 2010)

Features of the sampling distribution of the ability estimate in computerized adaptive testing according to two stopping rules - Blais, J. G.; Raiche, G. (Journal of Applied Measurement, 2010)
Whether paper and pencil or computerized adaptive, tests are usually described by a set of rules managing how they are administered: which item will be first, which should follow any given item, when to administer the last one. This article focus on the latter and looks at the effect of two stopping rules on the estimated sampling distribution of the ability estimate in a CAT: the number of items administered and the a priori determined size of the standard error of the ability estimate.
Blais, Jean-GuyRaiche, GillesUnited StatesJournal of applied measurementJ Appl Meas. 2010;11(4):424-31.

Implementing the Graduate Management Admission Test Computerized Adaptive Test - Rudner, L. M. (Elements of Adaptive Testing, 2010)

Improving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method - Ying Cheng, (Educational and Psychological Measurement, 2010)
This article proposes a new item selection method, namely, the modified maximum global discrimination index (MMGDI) method, for cognitive diagnostic computerized adaptive testing (CD-CAT). The new method captures two aspects of the appeal of an item: (a) the amount of contribution it can make toward adequate coverage of every attribute and (b) the amount of contribution it can make toward recovering the latent cognitive profile. A simulation study shows that the new method ensures adequate coverage of every attribute, which improves the validity of the test scores, and defensibility of the proposed uses of the test. Furthermore, compared with the original global discrimination index method, the MMGDI method improves the recovery rate of each attribute and of the entire cognitive profile, especially the latter. Therefore, the new method improves both the validity and reliability of the test scores from a CD-CAT program.

Innovative Items for Computerized Testing - Parshall, C. G.; Harmes, J. C.; Davey, T.; Pashley, P. J. (Elements of Adaptive Testing, 2010)

The Investigation of Differential Item Functioning in Adaptive Tests - Zwick, R. (Elements of Adaptive Testing, 2010)

Item Parameter Estimation and Item Fit Analysis - Glas, C. A. W. (Elements of Adaptive Testing, 2010)

Item Selection and Ability Estimation in Adaptive Testing - van der Linden, W. J.; Pashley, P. J. (Elements of Adaptive Testing, 2010)

Item Selection and Hypothesis Testing for the Adaptive Measurement of Change - Finkelman, M. D.; Weiss, D. J.; Kim-Kang, G. (Applied Psychological Measurement, 2010)
Assessing individual change is an important topic in both psychological and educational measurement. An adaptive measurement of change (AMC) method had previously been shown to exhibit greater efficiency in detecting change than conventional nonadaptive methods. However, little work had been done to compare different procedures within the AMC framework. This study introduced a new item selection criterion and two new test statistics for detecting change with AMC that were specifically designed for the paradigm of hypothesis testing. In two simulation sets, the new methods for detecting significant change improved on existing procedures by demonstrating better adherence to Type I error rates and substantially better power for detecting relatively small change.
Keywords: change; computerized adaptive testing; individual change; Kullback–Leibler information; likelihood ratio; measuring change

A Japanese Adaptive Test of English as a Foreign Language: Developmental and Operational Aspects - Nogami, Y.; Hayashi, N. (Elements of Adaptive Testing, 2010)

Manual for CATSim: Comprehensive simulation of computerized adaptive testing - Weiss, D. J.; Guyer, R. D. (Assessment Systems Corporation, 2010)

Marginal likelihood inference for a model for item responses and response times - Glas, C. A. W.; van der Linden, W. J. (British Journal of Mathematical and Statistical Psychology, 2010)
Marginal maximum-likelihood procedures for parameter estimation and testing the fit of a hierarchical model for speed and accuracy on test items are presented. The model is a composition of two first-level models for dichotomous responses and response times along with multivariate normal models for their item and person parameters. It is shown how the item parameters can easily be estimated using Fisher's identity. To test the fit of the model, Lagrange multiplier tests of the assumptions of subpopulation invariance of the item parameters (i.e., no differential item functioning), the shape of the response functions, and three different types of conditional independence were derived. Simulation studies were used to show the feasibility of the estimation and testing procedures and to estimate the power and Type I error rate of the latter. In addition, the procedures were applied to an empirical data set from a computerized adaptive test of language comprehension.
Glas, Cees A Wvan der Linden, Wim JResearch Support, Non-U.S. Gov'tEnglandThe British journal of mathematical and statistical psychologyBr J Math Stat Psychol. 2010 Nov;63(Pt 3):603-26. Epub 2010 Jan 28.

MATHCAT: A Flexible Testing System in Mathematics Education for Adults - Verschoor, Angela J.; Straetmans, G. J. J. M. (Elements of Adaptive Testing, 2010)

A Method for the Comparison of Item Selection Rules in Computerized Adaptive Testing - Barrada, Juan Ramón; Olea, Julio; Ponsoda, Vicente; Abad, Francisco José (Applied Psychological Measurement, 2010)
In a typical study comparing the relative efficiency of two item selection rules in computerized adaptive testing, the common result is that they simultaneously differ in accuracy and security, making it difficult to reach a conclusion on which is the more appropriate rule. This study proposes a strategy to conduct a global comparison of two or more selection rules. A plot showing the performance of each selection rule for several maximum exposure rates is obtained and the whole plot is compared with other rule plots. The strategy was applied in a simulation study with fixed-length CATs for the comparison of six item selection rules: the point Fisher information, Fisher information weighted by likelihood, Kullback-Leibler weighted by likelihood, maximum information stratification with blocking, progressive and proportional methods. Our results show that there is no optimal rule for any overlap value or root mean square error (RMSE). The fact that a rule, for a given level of overlap, has lower RMSE than another does not imply that this pattern holds for another overlap rate. A fair comparison of the rules requires extensive manipulation of the maximum exposure rates. The best methods were the Kullback-Leibler weighted by likelihood, the proportional method, and the maximum information stratification method with blocking.

A Monte Carlo Simulation Investigating the Validity and Reliability of Ability Estimation in Item Response Theory with Speeded Computer Adaptive Tests - Schmitt, T. A.; Sass, D. A.; Sullivan, J. R.; Walker, C. M. (International Journal of Testing, 2010)

Multidimensional Adaptive Testing with Kullback–Leibler Information Item Selection - Mulder, J.; van der Linden, W. J. (Elements of Adaptive Testing, 2010)

Multidimensionale adaptive Kompetenzdiagnostik: Ergebnisse zur Messeffizienz [Multidimensional adaptive testing of competencies: Results regarding measurement efficiency]. - Frey, A.; Seitz, N-N. (Zeitschrift für Pädagogik, 2010)

Multistage Testing: Issues, Designs, and Research - Zenisky, A. L.; Hambleton, R. K.; Luecht, RM (Elements of Adaptive Testing, 2010)

A new stopping rule for computerized adaptive testing - Choi, S. W.; Grady, M. W.; Dodd, B. G. (Educational and Psychological Measurement, 2010)
The goal of the current study was to introduce a new stopping rule for computerized adaptive testing. The predicted standard error reduction stopping rule (PSER) uses the predictive posterior variance to determine the reduction in standard error that would result from the administration of additional items. The performance of the PSER was compared to that of the minimum standard error stopping rule and a modified version of the minimum information stopping rule in a series of simulated adaptive tests, drawn from a number of item pools. Results indicate that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant.
U01 AR052177-04/NIAMS NIH HHS/Educ Psychol Meas. 2010 Dec 1;70(6):1-17.

Online calibration via variable length computerized adaptive testing - Chang, Y. I.; Lu, H. Y. (Psychometrika, 2010)
Item calibration is an essential issue in modern item response theory based psychological or educational testing. Due to the popularity of computerized adaptive testing, methods to efficiently calibrate new items have become more important than that in the time when paper and pencil test administration is the norm. There are many calibration processes being proposed and discussed from both theoretical and practical perspectives. Among them, the online calibration may be one of the most cost effective processes. In this paper, under a variable length computerized adaptive testing scenario, we integrate the methods of adaptive design, sequential estimation, and measurement error models to solve online item calibration problems. The proposed sequential estimate of item parameters is shown to be strongly consistent and asymptotically normally distributed with a prechosen accuracy. Numerical results show that the proposed method is very promising in terms of both estimation accuracy and efficiency. The results of using calibrated items to estimate the latent trait levels are also reported.

Principles of Multidimensional Adaptive Testing - Segall, D. O. (Elements of Adaptive Testing, 2010)

A Procedure for Controlling General Test Overlap in Computerized Adaptive Testing - Chen, Shu-Ying (Applied Psychological Measurement, 2010)
To date, exposure control procedures that are designed to control test overlap in computerized adaptive tests (CATs) are based on the assumption of item sharing between pairs of examinees. However, in practice, examinees may obtain test information from more than one previous test taker. This larger scope of information sharing needs to be considered in conducting test overlap control. The purpose of this study is to propose a test overlap control method such that the proportion of overlapping items encountered by an examinee with a group of previous examinees (described as general test overlap rate) can be controlled. Results indicated that item exposure rate and general test overlap rate could be simultaneously controlled by implementing the procedure. In addition, these two indices were controlled on the fly without any iterative simulations conducted prior to operational CATs. Thus, the proposed procedure would be an efficient method for controlling both the item exposure and general test overlap in CATs.

Sequencing an Adaptive Test Battery - van der Linden, W. J. (Elements of Adaptive Testing, 2010)

SimulCAT: Windows application that simulates computerized adaptive test administration - Han, K. T. (2010)

Stratified and Maximum Information Item Selection Procedures in Computer Adaptive Testing - Deng, Hui; Ansley, Timothy; Chang, Hua-Hua (Journal of Educational Measurement, 2010)
In this study we evaluated and compared three item selection procedures: the maximum Fisher information procedure (F), the a-stratified multistage computer adaptive testing (CAT) (STR), and a refined stratification procedure that allows more items to be selected from the high a strata and fewer items from the low a strata (USTR), along with completely random item selection (RAN). The comparisons were with respect to error variances, reliability of ability estimates and item usage through CATs simulated under nine test conditions of various practical constraints and item selection space. The results showed that F had an apparent precision advantage over STR and USTR under unconstrained item selection, but with very poor item usage. USTR reduced error variances for STR under various conditions, with small compromises in item usage. Compared to F, USTR enhanced item usage while achieving comparable precision in ability estimates; it achieved a precision level similar to F with improved item usage when items were selected under exposure control and with limited item selection space. The results provide implications for choosing an appropriate item selection procedure in applied settings.

Stratified and maximum information item selection procedures in computer adaptive testing - Deng, H.; Ansley, T.; Chang, H.-H. (Journal of Educational Measurement, 2010)

Testlet-Based Adaptive Mastery Testing - Vos, H. J.; Glas, C. A. W. (Elements of Adaptive Testing, 2010)

Tests informatizados y otros nuevos tipos de tests [Computerized and other new types of tests] - Olea, J.; Abad, F. J.; Barrada, J (Papeles del Psicólogo, 2010)
Recientemente se ha producido un considerable desarrollo de los tests adaptativos informatizados, en los que el test se adapta progresivamente al rendimiento del evaluando, y de otros tipos de tests: a) los test basados en modelos (se dispone de un modelo o teoría de cómo se responde a cada ítem, lo que permite predecir su dificultad), b) los tests ipsativos (el evaluado ha de elegir entre opciones que tienen parecida deseabilidad social, por lo que pueden resultar eficaces para controlar algunos sesgos de respuestas), c) los tests conductuales (miden rasgos que ordinariamente se han venido midiendo con autoinformes, mediante tareas que requieren respuestas no verbales) y d) los tests situacionales (en los que se presenta al evaluado una situación de conflicto laboral, por ejemplo, con varias posibles soluciones, y ha de elegir la que le parece la mejor descripción de lo que el haría en esa situación). El artículo comenta las características, ventajas e inconvenientes de todos ellos y muestra algunos ejemplos de tests concretos. Palabras clave: Test adaptativo informatizado, Test situacional, Test comportamental, Test ipsativo y generación automática de ítems.The paper provides a short description of some test types that are earning considerable interest in both research and applied areas. The main feature of a computerized adaptive test is that in despite of the examinees receiving different sets of items, their test scores are in the same metric and can be directly compared. Four other test types are considered: a) model-based tests (a model or theory is available to explain the item response process and this makes the prediction of item difficulties possible), b) ipsative tests (the examinee has to select one among two or more options with similar social desirability; so, these tests can help to control faking or other examinee’s response biases), c) behavioral tests (personality traits are measured from non-verbal responses rather than from self-reports), and d) situational tests (the examinee faces a conflictive situation and has to select the option that best describes what he or she will do). The paper evaluates these types of tests, comments on their pros and cons and provides some specific examples. Key words: Computerized adaptive test, Situational test, Behavioral test, Ipsative test and y automatic item generation.

Three-Category Adaptive Classification Testing - Theo Eggen (Elements of Adaptive Testing, 2010)

The use of PROMIS and assessment center to deliver patient-reported outcome measures in clinical research - Gershon, R. C.; Rothrock, N.; Hanrahan, R.; Bass, M.; Cella, D. (Journal of Applied Measurement, 2010)
The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed as one of the first projects funded by the NIH Roadmap for Medical Research Initiative to re-engineer the clinical research enterprise. The primary goal of PROMIS is to build item banks and short forms that measure key health outcome domains that are manifested in a variety of chronic diseases which could be used as a "common currency" across research projects. To date, item banks, short forms and computerized adaptive tests (CAT) have been developed for 13 domains with relevance to pediatric and adult subjects. To enable easy delivery of these new instruments, PROMIS built a web-based resource (Assessment Center) for administering CATs and other self-report data, tracking item and instrument development, monitoring accrual, managing data, and storing statistical analysis results. Assessment Center can also be used to deliver custom researcher developed content, and has numerous features that support both simple and complicated accrual designs (branching, multiple arms, multiple time points, etc.). This paper provides an overview of the development of the PROMIS item banks and details Assessment Center functionality.

Validation of a computer-adaptive test to evaluate generic health-related quality of life - Rebollo, P.; Castejon, I.; Cuervo, J.; Villa, G.; Garcia-Cueto, E.; Diaz-Cuervo, H.; Zardain, P. C.; Muniz, J.; Alonso, J. (Health and Quality of Life Outcomes, 2010)
BACKGROUND: Health Related Quality of Life (HRQoL) is a relevant variable in the evaluation of health outcomes. Questionnaires based on Classical Test Theory typically require a large number of items to evaluate HRQoL. Computer Adaptive Testing (CAT) can be used to reduce tests length while maintaining and, in some cases, improving accuracy. This study aimed at validating a CAT based on Item Response Theory (IRT) for evaluation of generic HRQoL: the CAT-Health instrument. METHODS: Cross-sectional study of subjects aged over 18 attending Primary Care Centres for any reason. CAT-Health was administered along with the SF-12 Health Survey. Age, gender and a checklist of chronic conditions were also collected. CAT-Health was evaluated considering: 1) feasibility: completion time and test length; 2) content range coverage, Item Exposure Rate (IER) and test precision; and 3) construct validity: differences in the CAT-Health scores according to clinical variables and correlations between both questionnaires. RESULTS: 396 subjects answered CAT-Health and SF-12, 67.2% females, mean age (SD) 48.6 (17.7) years. 36.9% did not report any chronic condition. Median completion time for CAT-Health was 81 seconds (IQ range = 59-118) and it increased with age (p < 0.001). The median number of items administered was 8 (IQ range = 6-10). Neither ceiling nor floor effects were found for the score. None of the items in the pool had an IER of 100% and it was over 5% for 27.1% of the items. Test Information Function (TIF) peaked between levels -1 and 0 of HRQoL. Statistically significant differences were observed in the CAT-Health scores according to the number and type of conditions. CONCLUSIONS: Although domain-specific CATs exist for various areas of HRQoL, CAT-Health is one of the first IRT-based CATs designed to evaluate generic HRQoL and it has proven feasible, valid and efficient, when administered to a broad sample of individuals attending primary care settings.
Rebollo, PabloCastejon, IgnacioCuervo, JesusVilla, GuillermoGarcia-Cueto, EduardoDiaz-Cuervo, HelenaZardain, Pilar CMuniz, JoseAlonso, JordiSpanish CAT-Health Research GroupEnglandHealth Qual Life Outcomes. 2010 Dec 3;8:147.

Variations on Stochastic Curtailment in Sequential Mastery Testing - Finkelman, Matthew David (Applied Psychological Measurement, 2010)
In sequential mastery testing (SMT), assessment via computer is used to classify examinees into one of two mutually exclusive categories. Unlike paper-and-pencil tests, SMT has the capability to use variable-length stopping rules. One approach to shortening variable-length tests is stochastic curtailment, which halts examination if the probability of changing classification decisions is low. The estimation of such a probability is therefore a critical component of a stochastically curtailed test. This article examines several variations on stochastic curtailment where the key probability is estimated more aggressively than the standard formulation, resulting in additional savings in average test length (ATL). In two simulation sets, the variations successfully reduced the ATL, and in many cases the average loss, compared with the standard formulation.

Adaptive computer-based tasks under an assessment engineering paradigm - Luecht, RM (2009)
{PDF File, 289 KB}

Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test - Kingsbury, G. G. (2009)
The characteristics of an adaptive test change the characteristics of the field testing that is necessary to add items to an existing measurement scale. The process used to add field-test items to the adaptive test might lead to scale drift or disrupt the test by administering items of inappropriate difficulty. The current study makes use of the transitivity of examinee and item in item response theory to describe a process for adaptive item calibration. In this process an item is successively administered to examinees whose ability levels match the performance of a given field-test item. By treating the item as if it were taking an adaptive test, examinees can be selected who provide the most information about the item at its momentary difficulty level. This should provide a more efficient procedure for estimating item parameters. The process is described within the context of the one-parameter logistic IRT model. The process is then simulated to identify whether it can be more accurate and efficient than random presentation of field-test items to examinees. Results indicated that adaptive item calibration might provide a viable approach to item calibration within the context of an adaptive test. It might be most useful for expanding item pools in settings with small sample sizes or needs for large numbers of items.
{PDF File, 286 KB} {PDF File, 286 KB}

An adaptive testing system for supporting versatile educational assessment - Huang, Y-M.; Lin, Y-T.; Cheng, S-C. (Computers and Education, 2009)
With the rapid growth of computer and mobile technology, it is a challenge to integrate computer based test (CBT) with mobile learning (m-learning) especially for formative assessment and self-assessment. In terms of self-assessment, computer adaptive test (CAT) is a proper way to enable students to evaluate themselves. In CAT, students are assessed through a process that uses item response theory (IRT), a well-founded psychometric theory. Furthermore, a large item bank is indispensable to a test, but when a CAT system has a large item bank, the test item selection of IRT becomes more tedious. Besides the large item bank, item exposure mechanism is also essential to a testing system. However, IRT all lack the above-mentioned points. These reasons have motivated the authors to carry out this study. This paper describes a design issue aimed at the development and implementation of an adaptive testing system. The system can support several assessment functions and different devices. Moreover, the researchers apply a novel approach, particle swarm optimization (PSO) to alleviate the computational complexity and resolve the problem of item exposure. Throughout the development of the system, a formative evaluation was embedded into an integral part of the design methodology that was used for improving the system. After the system was formally released onto the web, some questionnaires and experiments were conducted to evaluate the usability, precision, and efficiency of the system. The results of these evaluations indicated that the system provides an adaptive testing for different devices and supports versatile assessment functions. Moreover, the system can estimate students' ability reliably and validly and conduct an adaptive test efficiently. Furthermore, the computational complexity of the system was alleviated by the PSO approach. By the approach, the test item selection procedure becomes efficient and the average best fitness values are very close to the optimal solutions.
doi: DOI: 10.1016/j.compedu.2008.06.007
Keywords: Architectures for educational technology system; Distance education and telelearning

Adequacy of an item pool measuring proficiency in English language to implement a CAT procedure - Karino, C. A.; Costa, D. R.; Laros, J. A. (2009)
{PDF File, 160 KB}

Applications of CAT in admissions to higher education in Israel: Twenty-two years of experience - Gafni, N.; Cohen, Y.; Roded, K; Baumer, M; Moshinsky, A. (2009)
{PDF file, 326 KB}

An approach to implementing adaptive testing using item response theory both offline and online - Padaki, M.; Natarajan, V. (2009)
{PDF File, 172 KB}

Assessing the equivalence of Internet-based vs. paper-and-pencil psychometric tests. - Baumer, M; Roded, K; Gafni, N. (2009)
PDF File, 142 K

An automatic online calibration design in adaptive testing - Makransky, G. (2009)
{PDF file, 365 KB}

A burdened CAT: Incorporating response burden with maximum Fisher's information for item selection - Swartz, R.J..; Choi, S. W. (2009)
Widely used in various educational and vocational assessment applications, computerized adaptive testing (CAT) has recently begun to be used to measure patient-reported outcomes Although successful in reducing respondent burden, most current CAT algorithms do not formally consider it as part of the item selection process. This study used a loss function approach motivated by decision theory to develop an item selection method that incorporates respondent burden into the item selection process based on maximum Fisher information item selection. Several different loss functions placing varying degrees of importance on respondent burden were compared, using an item bank of 62 polytomous items measuring depressive symptoms. One dataset consisted of the real responses from the 730 subjects who responded to all the items. A second dataset consisted of simulated responses to all the items based on a grid of latent trait scores with replicates at each grid point. The algorithm enables a CAT administrator to more efficiently control the respondent burden without severely affecting the measurement precision than when using MFI alone. In particular, the loss function incorporating respondent burden protected respondents from receiving longer tests when their estimated trait score fell in a region where there were few informative items.
{PDF File, 374 KB}

Comparing methods to recalibrate drifting items in computerized adaptive testing - Masters, J. S.; Muckle, T. J.; Bontempo, B (American Educational Research Association, 2009)

Comparison of ability estimation and item selection methods in multidimensional computerized adaptive testing - Diao, Q.; Reckase, M. (2009)
{PDF File, 342 KB}

Comparison of adaptive Bayesian estimation and weighted Bayesian estimation in multidimensional computerized adaptive testing - Chen, P. H. (2009)
{PDF file, 308KB}

Comparison of CAT Item Selection Criteria for Polytomous Items - Choi, Seung W.; Swartz, Richard J. (Applied Psychological Measurement, 2009)
Item selection is a core component in computerized adaptive testing (CAT). Several studies have evaluated new and classical selection methods; however, the few that have applied such methods to the use of polytomous items have reported conflicting results. To clarify these discrepancies and further investigate selection method properties, six different selection methods are compared systematically. The results showed no clear benefit from more sophisticated selection criteria and showed one method previously believed to be superior—the maximum expected posterior weighted information (MEPWI)—to be mathematically equivalent to a simpler method, the maximum posterior weighted information (MPWI).

Comparison of CAT item selection criteria for polytomous items - Choi, S. W.; Swartz, R.J.. (Applied Psychological Measurement, 2009)

Comparison of methods for controlling maximum exposure rates in computerized adaptive testing - Barrada, J; Abad, F. J.; Veldkamp, B. P. (Psicothema, 2009)
This paper has two objectives: (a) to provide a clear description of three methods for controlling the maximum exposure rate in computerized adaptive testing —the Symson-Hetter method, the restricted method, and the item-eligibility method— showing how all three can be interpreted as methods for constructing the variable sub-bank of items from which each examinee receives the items in his or her test; (b) to indicate the theoretical and empirical limitations of each method and to compare their performance. With the three methods, we obtained basically indistinguishable results in overlap rate and RMSE (differences in the third decimal place). The restricted method is the best method for controlling exposure rate, followed by the item-eligibility method. The worst method is the Sympson-Hetter method. The restricted method presents problems of sequential overlap rate. Our advice is to use the item-eligibility method, as it saves time and satisfies the goals of restricting maximum exposure. Comparación de métodos para el control de tasa máxima en tests adaptativos informatizados. Este artículo tiene dos objetivos: (a) ofrecer una descripción clara de tres métodos para el control de la tasa máxima en tests adaptativos informatizados, el método Symson-Hetter, el método restringido y el métodode elegibilidad del ítem, mostrando cómo todos ellos pueden interpretarse como métodos para la construcción del subbanco de ítems variable, del cual cada examinado recibe los ítems de su test; (b) señalar las limitaciones teóricas y empíricas de cada método y comparar sus resultados. Se obtienen resultados básicamente indistinguibles en tasa de solapamiento y RMSE con los tres métodos (diferencias en la tercera posición decimal). El método restringido es el mejor en el control de la tasa de exposición,seguido por el método de elegibilidad del ítem. El peor es el método Sympson-Hetter. El método restringido presenta un problema de solapamiento secuencial. Nuestra recomendación sería utilizar el método de elegibilidad del ítem, puesto que ahorra tiempo y satisface los objetivos de limitar la tasa máxima de exposición.
Barrada, Juan RamonAbad, Francisco JoseVeldkamp, Bernard PComparative StudySpainPsicothemaPsicothema. 2009 May;21(2):313-20.
Keywords: *Numerical Analysis, Computer-Assisted; Psychological Tests/*standards/*statistics & numerical data

A comparison of three methods of item selection for computerized adaptive testing - Costa, D. R.; Karino, C. A.; Moura, F. A. S.; Andrade, D. F. (2009)
PDF file, 531 K

Computerized adaptive testing by mutual information and multiple imputations - Thissen-Roe, A. (2009)
Over the years, most computerized adaptive testing (CAT) systems have used score estimation procedures from item response theory (IRT). IRT models have salutary properties for score estimation, error reporting, and next-item selection. However, some testing purposes favor scoring approaches outside IRT. Where a criterion metric is readily available and more relevant than the assessed construct, for example in the selection of job applicants, a predictive model might be appropriate (Scarborough & Somers, 2006). In these cases, neither IRT scoring nor a unidimensional assessment structure can be assumed. Yet, the primary benefit of CAT remains desirable: shorter assessments with minimal loss of accuracy due to unasked items. In such a case, it remains possible to create a CAT system that produces an estimated score from a subset of available items, recognizes differential item information given the emerging item response pattern, and optimizes the accuracy of the score estimated at every successive item. The method of multiple imputations (Rubin, 1987) can be used to simulate plausible scores given plausible response patterns to unasked items (Thissen-Roe, 2005). Mutual information can then be calculated in order to select an optimally informative next item (or set of items). Previously observed response patterns to two complete neural network-scored assessments were resampled according to MIMI CAT item selection. The reproduced CAT scores were compared to full-length assessment scores. Approximately 95% accurate assignment of examinees to one of three score categories was achieved with a 70%-80% reduction in median test length. Several algorithmic factors influencing accuracy and computational performance were examined.
{PDF file, 179 KB}

Computerized adaptive testing for cognitive diagnosis - Cheng, Y (2009)
{PDF File, 308 KB}

Computerized adaptive testing using the two parameter logistic model with ability-based guessing - Shih, H.-J.; Wang, W-C. (Paper presented at the International Meeting of the Psychometric Society. Cambridge, 2009)

Computerized classification testing in more than two categories by using stochastic curtailment - Wouda, J. T.; Theo Eggen (2009)
{PDF file, 298 KB}

A conditional exposure control method for multidimensional adaptive testing - Finkelman, M.; Nering, M. L.; Roussos, L. A. (Journal of Educational Measurement, 2009)

A Conditional Exposure Control Method for Multidimensional Adaptive Testing - Matthew Finkelman; Nering, Michael L.; Roussos, Louis A. (Journal of Educational Measurement, 2009)
In computerized adaptive testing (CAT), ensuring the security of test items is a crucial practical consideration. A common approach to reducing item theft is to define maximum item exposure rates, i.e., to limit the proportion of examinees to whom a given item can be administered. Numerous methods for controlling exposure rates have been proposed for tests employing the unidimensional 3-PL model. The present article explores the issues associated with controlling exposure rates when a multidimensional item response theory (MIRT) model is utilized and exposure rates must be controlled conditional upon ability. This situation is complicated by the exponentially increasing number of possible ability values in multiple dimensions. The article introduces a new procedure, called the generalized Stocking-Lewis method, that controls the exposure rate for students of comparable ability as well as with respect to the overall population. A realistic simulation set compares the new method with three other approaches: Kullback-Leibler information with no exposure control, Kullback-Leibler information with unconditional Sympson-Hetter exposure control, and random item selection.

Considerations about expected a posteriori estimation in adaptive testing: adaptive a priori, adaptive correction for bias, and adaptive integration interval - Raiche, G.; Blais, J. G. (Journal of Applied Measurement, 2009)
In a computerized adaptive test, we would like to obtain an acceptable precision of the proficiency level estimate using an optimal number of items. Unfortunately, decreasing the number of items is accompanied by a certain degree of bias when the true proficiency level differs significantly from the a priori estimate. The authors suggest that it is possible to reduced the bias, and even the standard error of the estimate, by applying to each provisional estimation one or a combination of the following strategies: adaptive correction for bias proposed by Bock and Mislevy (1982), adaptive a priori estimate, and adaptive integration interval.
Raiche, GillesBlais, Jean-GuyUnited StatesJournal of applied measurementJ Appl Meas. 2009;10(2):138-56.
Keywords: *Bias (Epidemiology); *Computers; Data Interpretation, Statistical; Models, Statistical

Constrained item selection using a stochastically curtailed SPRT - Wouda, J. T.; Theo Eggen (2009)
{PDF File, 298 KB}{PDF File, 298 KB}

Constraint-Weighted a-Stratification for Computerized Adaptive Testing With Nonstatistical Constraints - Ying Cheng,; Chang, Hua-Hua; Douglas, Jeffrey; Fanmin Guo, (Educational and Psychological Measurement, 2009)
a-stratification is a method that utilizes items with small discrimination (a) parameters early in an exam and those with higher a values when more is learned about the ability parameter. It can achieve much better item usage than the maximum information criterion (MIC). To make a-stratification more practical and more widely applicable, a method for weighting the item selection process in a-stratification as a means of satisfying multiple test constraints is proposed. This method is studied in simulation against an analogous method without stratification as well as a-stratification using descending-rather than ascending-a procedures. In addition, a variation of a-stratification that allows for unbalanced usage of a parameters is included in the study to examine the trade-off between efficiency and exposure control. Finally, MIC and randomized item selection are included as baseline measures. Results indicate that the weighting mechanism successfully addresses the constraints, that stratification helps to a great extent balancing exposure rates, and that the ascending-a design improves measurement precision.

Constraint-weighted a-stratification for computerized adaptive testing with nonstatistical constraints: Balancing measurement efficiency and exposure control - Cheng, Y; Chang, Hua-Hua; Douglas, J.; Guo, F. (Educational and Psychological Measurement, 2009)

Criterion-related validity of an innovative CAT-based personality measure - Schneider, R. J.; McLellan, R. A.; Kantrowitz, T. M.; Houston, J. S.; Borman, W. C. (2009)
This paper describes development and initial criterion-related validation of the PreVisor Computer Adaptive Personality Scales (PCAPS), a computerized adaptive testing-based personality measure that uses an ideal point IRT model based on forced-choice, paired-comparison responses. Based on results from a large consortium study, a composite of six PCAPS scales identified as relevant to the population of interest (first-line supervisors) had an estimated operational validity against an overall job performance criterion of ρ = .25. Uncorrected and corrected criterion-related validity results for each of the six PCAPS scales making up the composite are also reported. Because the PCAPS algorithm computes intermediate scale scores until a stopping rule is triggered, we were able to graph number of statement-pairs presented against criterion-related validities. Results showed generally monotonically increasing functions. However, asymptotic validity levels, or at least a reduction in the rate of increase in slope, were often reached after 5-7 statement-pairs were presented. In the case of the composite measure, there was some evidence that validities decreased after about six statement-pairs. A possible explanation for this is provided.
{PDF File, 163 KB}

Developing item variants: An empirical study - Wendt, A.; Kao, S.; Gorham, J.; Woo, A. (2009)
Large-scale standardized test have been widely used for educational and licensure testing. In computerized adaptive testing (CAT), one of the practical concerns for maintaining large-scale assessments is to ensure adequate numbers of high-quality items that are required for item pool functioning. Developing items at specific difficulty levels and for certain areas of test plans is a wellknown challenge. The purpose of this study was to investigate strategies for varying items that can effectively generate items at targeted difficulty levels and specific test plan areas. Each variant item generation model was developed by decomposing selected source items possessing ideal measurement properties and targeting the desirable content domains. 341 variant items were generated from 72 source items. Data were collected from six pretest periods. Items were calibrated using the Rasch model. Initial results indicate that variant items showed desirable measurement properties. Additionally, compared to an average of approximately 60% of the items passing pretest criteria, an average of 84% of the variant items passed the pretest criteria.
{PDF file, 194 KB}

Development and preliminary testing of a computerized adaptive assessment of chronic pain - Anatchkova, M. D.; Saris-Baglama, R. N.; Kosinski, M.; Bjorner, J. B. (Journal of Pain, 2009)
The aim of this article is to report the development and preliminary testing of a prototype computerized adaptive test of chronic pain (CHRONIC PAIN-CAT) conducted in 2 stages: (1) evaluation of various item selection and stopping rules through real data-simulated administrations of CHRONIC PAIN-CAT; (2) a feasibility study of the actual prototype CHRONIC PAIN-CAT assessment system conducted in a pilot sample. Item calibrations developed from a US general population sample (N = 782) were used to program a pain severity and impact item bank (kappa = 45), and real data simulations were conducted to determine a CAT stopping rule. The CHRONIC PAIN-CAT was programmed on a tablet PC using QualityMetric's Dynamic Health Assessment (DYHNA) software and administered to a clinical sample of pain sufferers (n = 100). The CAT was completed in significantly less time than the static (full item bank) assessment (P < .001). On average, 5.6 items were dynamically administered by CAT to achieve a precise score. Scores estimated from the 2 assessments were highly correlated (r = .89), and both assessments discriminated across pain severity levels (P < .001, RV = .95). Patients' evaluations of the CHRONIC PAIN-CAT were favorable. PERSPECTIVE: This report demonstrates that the CHRONIC PAIN-CAT is feasible for administration in a clinic. The application has the potential to improve pain assessment and help clinicians manage chronic pain.
Anatchkova, Milena DSaris-Baglama, Renee NKosinski, MarkBjorner, Jakob B1R43AR052251-01A1/AR/NIAMS NIH HHS/United StatesEvaluation StudiesResearch Support, N.I.H., ExtramuralUnited StatesThe journal of pain : official journal of the American Pain SocietyJ Pain. 2009 Sep;10(9):932-43.
Keywords: *Computers; *Questionnaires; Activities of Daily Living; Adaptation, Psychological; Chronic Disease; Cohort Studies; Disability Evaluation; Female; Humans; Male; Middle Aged; Models, Psychological; Outcome Assessment (Health Care); Pain Measurement/*methods; Pain, Intractable/*diagnosis/psychology; Psychometrics; Quality of Life; User-Computer Interface

Development of an item bank for the assessment of depression in persons with mental illnesses and physical diseases using Rasch analysis - Forkmann, T.; Boecker, M.; Norra, C.; Eberle, N.; Kircher, T.; Schauerte, P.; Mischke, K.; Westhofen, M.; Gauggel, S.; Wirtz, M. (Rehabilitation Psychology, 2009)
OBJECTIVE: The calibration of item banks provides the basis for computerized adaptive testing that ensures high diagnostic precision and minimizes participants' test burden. The present study aimed at developing a new item bank that allows for assessing depression in persons with mental and persons with somatic diseases. METHOD: The sample consisted of 161 participants treated for a depressive syndrome, and 206 participants with somatic illnesses (103 cardiologic, 103 otorhinolaryngologic; overall mean age = 44.1 years, SD =14.0; 44.7% women) to allow for validation of the item bank in both groups. Persons answered a pool of 182 depression items on a 5-point Likert scale. RESULTS: Evaluation of Rasch model fit (infit < 1.3), differential item functioning, dimensionality, local independence, item spread, item and person separation (>2.0), and reliability (>.80) resulted in a bank of 79 items with good psychometric properties. CONCLUSIONS: The bank provides items with a wide range of content coverage and may serve as a sound basis for computerized adaptive testing applications. It might also be useful for researchers who wish to develop new fixed-length scales for the assessment of depression in specific rehabilitation settings.
Forkmann, ThomasBoecker, MarenNorra, ChristineEberle, NicoleKircher, TiloSchauerte, PatrickMischke, KarlWesthofen, MartinGauggel, SiegfriedWirtz, MarkusResearch Support, Non-U.S. Gov'tUnited StatesRehabilitation psychologyRehabil Psychol. 2009 May;54(2):186-97.
Keywords: Adaptation, Psychological; Adult; Aged; Depressive Disorder/*diagnosis/psychology; Diagnosis, Computer-Assisted; Female; Heart Diseases/*psychology; Humans; Male; Mental Disorders/*psychology; Middle Aged; Models, Statistical; Otorhinolaryngologic Diseases/*psychology; Personality Assessment/statistics & numerical data; Personality Inventory/*statistics & numerical data; Psychometrics/statistics & numerical data; Questionnaires; Reproducibility of Results; Sick Role

Diagnostic classification models and multidimensional adaptive testing: A commentary on Rupp and Templin. - Frey, A.; Carstensen, C. H. (Measurement: Interdisciplinary Research and Perspectives, 2009)

Direct and inverse problems of item pool design for computerized adaptive testing - Belov, D. I.; Armstrong, R. D. (Educational and Psychological Measurement, 2009)

Direct and Inverse Problems of Item Pool Design for Computerized Adaptive Testing - Belov, Dmitry I.; Armstrong, Ronald D. (Educational and Psychological Measurement, 2009)
The recent literature on computerized adaptive testing (CAT) has developed methods for creating CAT item pools from a large master pool. Each CAT pool is designed as a set of nonoverlapping forms reflecting the skill levels of an assumed population of test takers. This article presents a Monte Carlo method to obtain these CAT pools and discusses its advantages over existing methods. Also, a new problem is considered that finds a population ability density function best matching the master pool. An analysis of the solution to this new problem provides testing organizations with effective guidance for maintaining their master pools. Computer experiments with a pool of Law School Admission Test items and its assembly constraints are presented.

Effect of early misfit in computerized adaptive testing on the recovery of theta - Guyer, R. D.; Weiss, D. J. (2009)
{PDF File, 212 KB}

Effekte des adaptiven Testens auf die Moti¬vation zur Testbearbeitung [Effects of adaptive testing on test taking motivation]. - Frey, A.; Hartig, J.; Moosbrugger, H. (Diagnostica, 2009)

Evaluation of a computer-adaptive test for the assessment of depression (D-CAT) in clinical application - Fliege, H.; Becker, J.; Walter, O. B.; Rose, M.; Bjorner, J. B.; Klapp, B. F. (International Journal for Methods in Psychiatric Research, 2009)
In the past, a German Computerized Adaptive Test, based on Item Response Theory (IRT), was developed for purposes of assessing the construct depression [Computer-adaptive test for depression (D-CAT)]. This study aims at testing the feasibility and validity of the real computer-adaptive application.The D-CAT, supplied by a bank of 64 items, was administered on personal digital assistants (PDAs) to 423 consecutive patients suffering from psychosomatic and other medical conditions (78 with depression). Items were adaptively administered until a predetermined reliability (r >/= 0.90) was attained. For validation purposes, the Hospital Anxiety and Depression Scale (HADS), the Centre for Epidemiological Studies Depression (CES-D) scale, and the Beck Depression Inventory (BDI) were administered. Another sample of 114 patients was evaluated using standardized diagnostic interviews [Composite International Diagnostic Interview (CIDI)].The D-CAT was quickly completed (mean 74 seconds), well accepted by the patients and reliable after an average administration of only six items. In 95% of the cases, 10 items or less were needed for a reliable score estimate. Correlations between the D-CAT and the HADS, CES-D, and BDI ranged between r = 0.68 and r = 0.77. The D-CAT distinguished between diagnostic groups as well as established questionnaires do.The D-CAT proved an efficient, well accepted and reliable tool. Discriminative power was comparable to other depression measures, whereby the CAT is shorter and more precise. Item usage raises questions of balancing the item selection for content in the future. Copyright (c) 2009 John Wiley & Sons, Ltd.
Journal articleInternational journal of methods in psychiatric researchInt J Methods Psychiatr Res. 2009 Feb 4.

An evaluation of a new procedure for computing information functions for Bayesian scores from computerized adaptive tests - Ito, K.; Pommerich, M; Segall, D. (2009)
{PDF file, 571 KB}

An evaluation of patient-reported outcomes found computerized adaptive testing was efficient in assessing stress perception - Kocalevent, R. D.; Rose, M.; Becker, J.; Walter, O. B.; Fliege, H.; Bjorner, J. B.; Kleiber, D.; Klapp, B. F. (Journal of Clinical Epidemiology, 2009)
OBJECTIVES: This study aimed to develop and evaluate a first computerized adaptive test (CAT) for the measurement of stress perception (Stress-CAT), in terms of the two dimensions: exposure to stress and stress reaction. STUDY DESIGN AND SETTING: Item response theory modeling was performed using a two-parameter model (Generalized Partial Credit Model). The evaluation of the Stress-CAT comprised a simulation study and real clinical application. A total of 1,092 psychosomatic patients (N1) were studied. Two hundred simulees (N2) were generated for a simulated response data set. Then the Stress-CAT was given to n=116 inpatients, (N3) together with established stress questionnaires as validity criteria. RESULTS: The final banks included n=38 stress exposure items and n=31 stress reaction items. In the first simulation study, CAT scores could be estimated with a high measurement precision (SE<0.32; rho>0.90) using 7.0+/-2.3 (M+/-SD) stress reaction items and 11.6+/-1.7 stress exposure items. The second simulation study reanalyzed real patients data (N1) and showed an average use of items of 5.6+/-2.1 for the dimension stress reaction and 10.0+/-4.9 for the dimension stress exposure. Convergent validity showed significantly high correlations. CONCLUSIONS: The Stress-CAT is short and precise, potentially lowering the response burden of patients in clinical decision making.
Kocalevent, Ruya-DanielaRose, MatthiasBecker, JanineWalter, Otto BFliege, HerbertBjorner, Jakob BKleiber, DieterKlapp, Burghard FEvaluation StudiesUnited StatesJournal of clinical epidemiologyJ Clin Epidemiol. 2009 Mar;62(3):278-87, 287.e1-3. Epub 2008 Jul 18.
Keywords: *Diagnosis, Computer-Assisted; Adolescent; Adult; Aged; Aged, 80 and over; Confidence Intervals; Female; Humans; Male; Middle Aged; Perception; Quality of Health Care/*standards; Questionnaires; Reproducibility of Results; Sickness Impact Profile; Stress, Psychological/*diagnosis/psychology; Treatment Outcome

An examination of decision-theory adaptive testing procedures - Rudner, L. M. (2009)
This research examined three ways to adaptively select items using decision theory: a traditional decision theory sequential testing approach (expected minimum cost), information gain (modeled after Kullback-Leibler), and a maximum discrimination approach, and then compared them all against an approach using maximum IRT Fisher information. It also examined the use of Wald’s (1947) wellknown sequential probability ratio test, SPRT, as a test termination rule in this context. The minimum cost approach was notably better than the best-case possibility for IRT. Information gain, which is based on entropy and comes from information theory, was almost identical to minimum cost. The simple approach using the item that best discriminates between the two most likely classifications also fared better than IRT, but not as well as information gain or minimum cost. Through Wald’s SPRT, large percentages of examinees can be accurately classified with very few items. With only 25 sequentially selected items, for example, approximately 90% of the simulated NAEP examinees were classified with 86% accuracy. The advantages of the decision theory model are many—the model yields accurate mastery state classifications, can use a small item pool, is simple to implement, requires little pretesting, is applicable to criterion-referenced tests, can be used in diagnostic testing, can be adapted to yield classifications on multiple skills, and should be easy to explain to non-statisticians.
{PDF file, 203 KB}

Features of J-CAT (Japanese Computerized Adaptive Test) - Imai, S.; Ito, S.; Nakamura, Y.; Kikuchi, K.; Akagi, Y.; Nakasono, H.; Honda, A.; Hiramura, T. (2009)
{PDF File, 655KB}

Firestar: Computerized adaptive testing simulation program for polytomous IRT models - Choi, S. W. (Applied Psychological Measurement, 2009)

Firestar: Computerized adaptive testing simulation program for polytomous IRT models - Choi, S. W. (Applied Psychological Measurement, 2009)
U01 AR052177-04/NIAMS NIH HHS/United StatesJournal articleApplied psychological measurementAppl Psychol Meas. 2009 Nov 1;33(8):644-645.

A gradual maximum information ratio approach to item selection in computerized adaptive testing - Han, K. T. (2009)
{PDF file, 391 KB}

Gradual maximum information ratio approach to item selection in computerized adaptive testing - Han, K. T. (GMAC Research Reports, 2009)

Guess what? Score differences with rapid replies versus omissions on a computerized adaptive test - Talento-Miller, E.; Guo, F. (2009)
{PDF File, 215 KB}

A hybrid simulation procedure for the development of CATs - Nydick, S. W.; Weiss, D. J. (2009)
{PDF File, 258 KB}

Item response theory and clinical measurement - Reise, S. P.; Waller, N. G. (Annual Review of Clinical Psychology, 2009)
In this review, we examine studies that use item response theory (IRT) to explore the psychometric properties of clinical measures. Next, we consider how IRT has been used in clinical research for: scale linking, computerized adaptive testing, and differential item functioning analysis. Finally, we consider the scale properties of IRT trait scores. We conclude that there are notable differences between cognitive and clinical measures that have relevance for IRT modeling. Future research should be directed toward a better understanding of the metric of the latent trait and the psychological processes that lead to individual differences in item response behaviors.
Reise, Steven PWaller, Niels GU01 AR 52177/AR/NIAMS NIH HHS/United StatesResearch Support, N.I.H., ExtramuralReviewUnited StatesAnnual review of clinical psychologyAnnu Rev Clin Psychol. 2009;5:27-48.
Keywords: *Psychological Theory; Humans; Mental Disorders/diagnosis/psychology; Psychological Tests; Psychometrics; Quality of Life; Questionnaires

Item selection and hypothesis testing for the adaptive measurement of change - Finkelman, M.; Weiss, D. J.; Kim-Kang, G. (2009)
{PDF File, 228 KB}

Item selection in computerized classification testing - Thompson, N. A. (Educational and Psychological Measurement, 2009)
Several alternatives for item selection algorithms based on item response theory in computerized classification testing (CCT) have been suggested, with no conclusive evidence on the substantial superiority of a single method. It is argued that the lack of sizable effect is because some of the methods actually assess items very similarly through different calculations and will usually select the same item. Consideration of methods that assess information across a wider range is often unnecessary under realistic conditions, although it might be advantageous to utilize them only early in a test. In addition, the efficiency of item selection approaches depend on the termination criteria that are used, which is demonstrated through didactic example and Monte Carlo simulation. Item selection at the cut score, which seems conceptually appropriate for CCT, is not always the most efficient option. A broad framework for item selection in CCT is presented that incorporates these points.

Item Selection in Computerized Classification Testing - Thompson, Nathan A. (Educational and Psychological Measurement, 2009)
Several alternatives for item selection algorithms based on item response theory in computerized classification testing (CCT) have been suggested, with no conclusive evidence on the substantial superiority of a single method. It is argued that the lack of sizable effect is because some of the methods actually assess items very similarly through different calculations and will usually select the same item. Consideration of methods that assess information across a wider range is often unnecessary under realistic conditions, although it might be advantageous to utilize them only early in a test. In addition, the efficiency of item selection approaches depend on the termination criteria that are used, which is demonstrated through didactic example and Monte Carlo simulation. Item selection at the cut score, which seems conceptually appropriate for CCT, is not always the most efficient option. A broad framework for item selection in CCT is presented that incorporates these points.

Item selection rules in computerized adaptive testing: Accuracy and security - Barrada, J; Olea, J.; Ponsoda, V.; Abad, F. J. (Methodology, 2009)
(PDF file, 445 KB)

Item selection with biased-coin up-and-down designs - Sheng, Y.; Sheng, Z. (2009)
{PDF file, 748 KB}

I've Fallen and I Can't Get Up: Can High-Ability Students Recover From Early Mistakes in CAT? - Rulison, Kelly L.; Loken, Eric (Applied Psychological Measurement, 2009)
A difficult result to interpret in Computerized Adaptive Tests (CATs) occurs when an ability estimate initially drops and then ascends continuously until the test ends, suggesting that the true ability may be higher than implied by the final estimate. This study explains why this asymmetry occurs and shows that early mistakes by high-ability students can lead to considerable underestimation, even in tests with 45 items. The opposite response pattern, where low-ability students start with lucky guesses, leads to much less bias. The authors show that using Barton and Lord's four-parameter model (4PM) and a less informative prior can lower bias and root mean square error (RMSE) for high-ability students with a poor start, as the CAT algorithm ascends more quickly after initial underperformance. Results also show that the 4PM slightly outperforms a CAT in which less discriminating items are initially used. The practical implications and relevance for psychological measurement more generally are discussed.

I've fallen and I can't get up: can high-ability students recover from early mistakes in CAT? - Rulison, K.,; Loken, E. (Applied Psychological Measurement, 2009)

I've Fallen and I Can't Get Up: Can High-Ability Students Recover From Early Mistakes in CAT? - Rulison, Kelly L.; Loken, Eric (Applied Psychological Measurement, 2009)
A difficult result to interpret in Computerized Adaptive Tests (CATs) occurs when an ability estimate initially drops and then ascends continuously until the test ends, suggesting that the true ability may be higher than implied by the final estimate. This study explains why this asymmetry occurs and shows that early mistakes by high-ability students can lead to considerable underestimation, even in tests with 45 items. The opposite response pattern, where low-ability students start with lucky guesses, leads to much less bias. The authors show that using Barton and Lord's four-parameter model (4PM) and a less informative prior can lower bias and root mean square error (RMSE) for high-ability students with a poor start, as the CAT algorithm ascends more quickly after initial underperformance. Results also show that the 4PM slightly outperforms a CAT in which less discriminating items are initially used. The practical implications and relevance for psychological measurement more generally are discussed.

A Knowledge-Based Approach for Item Exposure Control in Computerized Adaptive Testing - Doong, S. H. (Journal of Educational and Behavioral Statistics, 2009)
The purpose of this study is to investigate a functional relation between item exposure parameters (IEPs) and item parameters (IPs) over parallel pools. This functional relation is approximated by a well-known tool in machine learning. Let P and Q be parallel item pools and suppose IEPs for P have been obtained via a Sympson and Hetter–type simulation. Based on these simulated parameters, a functional relation k = fP (a, b, c) relating IPs to IEPs of P is obtained by an artificial neural network and used to estimate IEPs of Q without tedious simulation. Extensive experiments using real and synthetic pools showed that this approach worked pretty well for many variants of the Sympson and Hetter procedure. It worked excellently for the conditional Stocking and Lewis multinomial selection procedure and the Chen and Lei item exposure and test overlap control procedure. This study provides the first step in an alternative means to estimate IEPs without iterative simulation.

Kullback-Leibler information in multidimensional adaptive testing: theory and application - Wang, C.; Chang, Hua-Hua (2009)
Built on multidimensional item response theory (MIRT), multidimensional adaptive testing (MAT) can, in principle, provide a promising choice to ensuring efficient estimation of each ability dimension in a multidimensional vector. Currently, two item selection procedures have been developed for MAT, one based on Fisher information embedded within a Bayesian framework, and the other powered by Kullback-Leibler (KL) information. It is well-known that in unidimensional IRT that the second derivative of KL information (also termed “global information”) is Fisher information evaluated atθ 0. This paper first generalizes the relationship between these two types of information in two ways—the analytical result is given as well as the graphical representation, to enhance interpretation and understanding. Second, a KL information index is constructed for MAT, which represents the integration of KL nformation over all of the ability dimensions. This paper further discusses how this index correlates with the item discrimination parameters. The analytical results would lay foundation for future development of item selection methods in MAT which can help equalize the item exposure rate. Finally, a simulation study is conducted to verify the above results. The connection between the item parameters, item KL information, and item exposure rate is demonstrated for empirical MAT delivered by an item bank calibrated under two-dimensional IRT.
{PDF file, 316 KB}

Limiting item exposure for target difficulty ranges in a high-stakes CAT - Li, X.; Becker, K.; Gorham, J.; Woo, A. (2009)
MB}

Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples - Rose, M.; Bezjak, A. (Quality of Life Research, 2009)
PURPOSE: Interest in collecting patient-reported outcomes (PROs), such as health-related quality of life (HRQOL), health status reports, and patient satisfaction is on the rise and practical aspects of collecting PROs in clinical practice are becoming more important. The purpose of this paper is to draw the attention to a number of issues relevant for a successful integration of PRO measures into the daily work flow of busy clinical settings. METHODS: The paper summarizes the results from a breakout session held at an ISOQOL special topic conference for PRO measures in clinical practice in 2007. RESULTS: Different methodologies of collecting PROs are discussed, and the support needed for each methodology is highlighted. The discussion is illustrated by practical real-life examples from early adaptors who administered paper-pencil, or electronic PRO assessments (ePRO) for more than a decade. The paper also reports about new experiences with more recent technological developments, such as SmartPens and Computer Adaptive Tests (CATs) in daily practice. CONCLUSIONS: Methodological and logistical issues determine the resources needed for a successful integration of PRO measures into daily work flow procedures and influence significantly the usefulness of PRO data for clinical practice.
Rose, MatthiasBezjak, AndreaNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2009 Feb;18(1):125-36. Epub 2009 Jan 20.

The maximum priority index method for severely constrained item selection in computerized adaptive testing - Cheng, Y; Chang, Hua-Hua (British Journal of Mathematical and Statistical Psychology, 2009)
This paper introduces a new heuristic approach, the maximum priority index (MPI) method, for severely constrained item selection in computerized adaptive testing. Our simulation study shows that it is able to accommodate various non-statistical constraints simultaneously, such as content balancing, exposure control, answer key balancing, and so on. Compared with the weighted deviation modelling method, it leads to fewer constraint violations and better exposure control while maintaining the same level of measurement precision.
Cheng, YingChang, Hua-HuaResearch Support, Non-U.S. Gov'tEnglandThe British journal of mathematical and statistical psychologyBr J Math Stat Psychol. 2009 May;62(Pt 2):369-83. Epub 2008 Jun 2.
Keywords: Aptitude Tests/*statistics & numerical data; Diagnosis, Computer-Assisted/*statistics & numerical data; Educational Measurement/*statistics & numerical data; Humans; Mathematical Computing; Models, Statistical; Personality Tests/*statistics & numerical data; Psychometrics/*statistics & numerical data; Reproducibility of Results; Software

Measuring global physical health in children with cerebral palsy: Illustration of a multidimensional bi-factor model and computerized adaptive testing - Haley, S. M.; Ni, P.; Dumas, H. M.; Fragala-Pinkham, M. A.; Hambleton, R. K.; Montpetit, K.; Bilodeau, N.; Gorton, G. E.; Watson, K.; Tucker, C. A. (Quality of Life Research, 2009)
PURPOSE: The purposes of this study were to apply a bi-factor model for the determination of test dimensionality and a multidimensional CAT using computer simulations of real data for the assessment of a new global physical health measure for children with cerebral palsy (CP). METHODS: Parent respondents of 306 children with cerebral palsy were recruited from four pediatric rehabilitation hospitals and outpatient clinics. We compared confirmatory factor analysis results across four models: (1) one-factor unidimensional; (2) two-factor multidimensional (MIRT); (3) bi-factor MIRT with fixed slopes; and (4) bi-factor MIRT with varied slopes. We tested whether the general and content (fatigue and pain) person score estimates could discriminate across severity and types of CP, and whether score estimates from a simulated CAT were similar to estimates based on the total item bank, and whether they correlated as expected with external measures. RESULTS: Confirmatory factor analysis suggested separate pain and fatigue sub-factors; all 37 items were retained in the analyses. From the bi-factor MIRT model with fixed slopes, the full item bank scores discriminated across levels of severity and types of CP, and compared favorably to external instruments. CAT scores based on 10- and 15-item versions accurately captured the global physical health scores. CONCLUSIONS: The bi-factor MIRT CAT application, especially the 10- and 15-item versions, yielded accurate global physical health scores that discriminated across known severity groups and types of CP, and correlated as expected with concurrent measures. The CATs have potential for collecting complex data on the physical health of children with CP in an efficient manner.
Haley, Stephen MNi, PengshengDumas, Helene MFragala-Pinkham, Maria AHambleton, Ronald KMontpetit, KathleenBilodeau, NathalieGorton, George EWatson, KyleTucker, Carole AK02 HD045354-01A1/HD/NICHD NIH HHS/United StatesK02 HD45354-01A1/HD/NICHD NIH HHS/United StatesResearch Support, N.I.H., ExtramuralResearch Support, Non-U.S. Gov'tNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2009 Apr;18(3):359-70. Epub 2009 Feb 17.
Keywords: *Computer Simulation; *Health Status; *Models, Statistical; Adaptation, Psychological; Adolescent; Cerebral Palsy/*physiopathology; Child; Child, Preschool; Factor Analysis, Statistical; Female; Humans; Male; Massachusetts; Pennsylvania; Questionnaires; Young Adult

The MEDPRO project: An SBIR project for a comprehensive IRT and CAT software system: CAT software - Thompson, N. A. (2009)
Development of computerized adaptive tests (CAT) requires a number of appropriate software tools. This paper describes the development of two new CAT software programs. CATSIM has been designed specifically to conduct several different kinds of simulation studies, which are necessary for planning purposes as well as properly designing live CATs. FastCAT is a software system for banking items and publishing CAT tests as standalone files, to be administered anywhere. Both are available for public use.
{PDF File, 283 KB}

The MEDPRO project: An SBIR project for a comprehensive IRT and CAT software system: IRT software - Thissen, D. (2009)
IRTPRO (Item Response Theory for Patient-Reported Outcomes) is an entirely new application for item calibration and test scoring using IRT. IRTPRO implements algorithms for maximum likelihood estimation of item parameters (item calibration) for several unidimensional and multidimensional item response theory (IRT) models for dichotomous and polytomous item responses. In addition, the software provides computation of goodness-of-fit indices, statistics for the diagnosis of local dependence and for the detection of differential item functioning (DIF), and IRT scaled scores. This paper illustrates the use, and some capabilities, of the software.
PDF File, 817 K

A mixed integer programming model for multiple stage adaptive testing - Edmonds, J.; Armstrong, R. D. (European Journal of Operational Research, 2009)
The last decade has seen paper-and-pencil (P&P) tests being replaced by computerized adaptive tests (CATs) within many testing programs. A CAT may yield several advantages relative to a conventional P&P test. A CAT can determine the questions or test items to administer, allowing each test form to be tailored to a test taker's skill level. Subsequent items can be chosen to match the capability of the test taker. By adapting to a test taker's ability, a CAT can acquire more information about a test taker while administering fewer items. A Multiple Stage Adaptive test (MST) provides a means to implement a CAT that allows review before the administration. The MST format is a hybrid between the conventional P&P and CAT formats. This paper presents mixed integer programming models for MST assembly problems. Computational results with commercial optimization software will be given and advantages of the models evaluated.
doi: DOI: 10.1016/j.ejor.2007.10.047
Keywords: Education; Integer programming; Linear programming

Multidimensional adaptive testing in educational and psychological measurement: Current state and future challenges - Frey, A.; Seitz, N-N. (Studies in Educational Evaluation, 2009)
The paper gives an overview of multidimensional adaptive testing (MAT) and evaluates its applicability in educational and psychological testing. The approach of Segall (1996) is described as a general framework for MAT. The main advantage of MAT is its capability to increase measurement efficiency. In simulation studies conceptualizing situations typical to large scale assessments, the number of presented items was reduced by MAT by about 30–50% compared to unidimensional adaptive testing and by about 70% compared to fixed item testing holding measurement precision constant. Empirical results underline these findings. Before MAT is used routinely some open questions should be answered first. After that, MAT represents a very promising approach to highly efficient simultaneous testing of multiple competencies.

Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection - Mulder, J.; van der Linden, W. J. (Psychometrika, 2009)
Several criteria from the optimal design literature are examined for use with item selection in multidimensional adaptive testing. In particular, it is examined what criteria are appropriate for adaptive testing in which all abilities are intentional, some should be considered as a nuisance, or the interest is in the testing of a composite of the abilities. Both the theoretical analyses and the studies of simulated data in this paper suggest that the criteria of A-optimality and D-optimality lead to the most accurate estimates when all abilities are intentional, with the former slightly outperforming the latter. The criterion of E-optimality showed occasional erratic behavior for this case of adaptive testing, and its use is not recommended. If some of the abilities are nuisances, application of the criterion of A(s)-optimality (or D(s)-optimality), which focuses on the subset of intentional abilities is recommended. For the measurement of a linear combination of abilities, the criterion of c-optimality yielded the best results. The preferences of each of these criteria for items with specific patterns of parameter values was also assessed. It was found that the criteria differed mainly in their preferences of items with different patterns of values for their discrimination parameters.
Journal articlePsychometrikaPsychometrika. 2009 Jun;74(2):273-296. Epub 2008 Dec 23.

Multiple Maximum Exposure Rates in Computerized Adaptive Testing - Barrada, Juan Ramón; Veldkamp, Bernard P.; Olea, Julio (Applied Psychological Measurement, 2009)
Computerized adaptive testing is subject to security problems, as the item bank content remains operative over long periods and administration time is flexible for examinees. Spreading the content of a part of the item bank could lead to an overestimation of the examinees' trait level. The most common way of reducing this risk is to impose a maximum exposure rate (rmax) that no item should exceed. Several methods have been proposed with this aim. All of these methods establish a single value of rmax throughout the test. This study presents a new method, the multiple-rmax method, that defines as many values of rmax as the number of items presented in the test. In this way, it is possible to impose a high degree of randomness in item selection at the beginning of the test, leaving the administration of items with the best psychometric properties to the moment when the trait level estimation is most accurate. The implementation of the multiple-r max method is described and is tested in simulated item banks and in an operative bank. Compared with a single maximum exposure method, the new method has a more balanced usage of the item bank and delays the possible distortion of trait estimation due to security problems, with either no or only slight decrements of measurement accuracy.

The nine lives of CAT-ASVAB: Innovations and revelations - Pommerich, M; Segall, D. O.; Moreno, K. E. (2009)
The Armed Services Vocational Aptitude Battery (ASVAB) is administered annually to more than one million military applicants and high school students. ASVAB scores are used to determine enlistment eligibility, assign applicants to military occupational specialties, and aid students in career exploration. The ASVAB is administered as both a paper-and-pencil (P&P) test and a computerized adaptive test (CAT). CAT-ASVAB holds the distinction of being the first large-scale adaptive test battery to be administered in a high-stakes setting. Approximately two-thirds of military applicants currently take CAT-ASVAB; long-term plans are to replace P&P-ASVAB with CAT-ASVAB at all test sites. Given CAT-ASVAB’s pedigree—approximately 20 years in development and 20 years in operational administration—much can be learned from revisiting some of the major highlights of CATASVAB history. This paper traces the progression of CAT-ASVAB through nine major phases of development including: research and evelopment of the CAT-ASVAB prototype, the initial development of psychometric procedures and item pools, initial and full-scale operational implementation, the introduction of new item pools, the introduction of Windows administration, the introduction of Internet administration, and research and development of the next generation CATASVAB. A background and history is provided for each phase, including discussions of major research and operational issues, innovative approaches and practices, and lessons learned.
{PDF File, 169 KB}

Obtaining reliable diagnostic information through constrained CAT - Wang, C.; Chang, Hua-Hua; Douglas, J. (2009)
{PDF File, 252 KB}

Optimizing item exposure control algorithms for polytomous computerized adaptive tests with restricted item banks - Chajewski, M.; Lewis, C. (2009)
{PDF File, 923 KB}

a posteriori estimation in adaptive testing: Adaptive a priori, adaptive correction for bias, and adaptive integration interval - Raîche, G.; Blais, J-G. (Journal of Applied Measurement, 2009)

Practical issues concerning the application of the DINA model to CAT data - Huebner, A.; Wang, B.; Lee, S. (2009)
{PDF file, 139 KB}

Predictive Control of Speededness in Adaptive Testing - van der Linden, Wim J. (Applied Psychological Measurement, 2009)
An adaptive testing method is presented that controls the speededness of a test using predictions of the test takers' response times on the candidate items in the pool. Two different types of predictions are investigated: posterior predictions given the actual response times on the items already administered and posterior predictions that use the responses on these items as an additional source of information. In a simulation study with an adaptive test modeled after a test from the Armed Services Vocational Aptitude Battery, the effectiveness of the methods in removing differential speededness from the test was evaluated.

Progress in assessing physical function in arthritis: PROMIS short forms and computerized adaptive testing - Fries, J.F.; Cella, D.; Rose, M.; Krishnan, E.; Bruce, B. (Journal of Rheumatology, 2009)
OBJECTIVE: Assessing self-reported physical function/disability with the Health Assessment Questionnaire Disability Index (HAQ) and other instruments has become central in arthritis research. Item response theory (IRT) and computerized adaptive testing (CAT) techniques can increase reliability and statistical power. IRT-based instruments can improve measurement precision substantially over a wider range of disease severity. These modern methods were applied and the magnitude of improvement was estimated. METHODS: A 199-item physical function/disability item bank was developed by distilling 1865 items to 124, including Legacy Health Assessment Questionnaire (HAQ) and Physical Function-10 items, and improving precision through qualitative and quantitative evaluation in over 21,000 subjects, which included about 1500 patients with rheumatoid arthritis and osteoarthritis. Four new instruments, (A) Patient-Reported Outcomes Measurement Information (PROMIS) HAQ, which evolved from the original (Legacy) HAQ; (B) "best" PROMIS 10; (C) 20-item static (short) forms; and (D) simulated PROMIS CAT, which sequentially selected the most informative item, were compared with the HAQ. RESULTS: Online and mailed administration modes yielded similar item and domain scores. The HAQ and PROMIS HAQ 20-item scales yielded greater information content versus other scales in patients with more severe disease. The "best" PROMIS 20-item scale outperformed the other 20-item static forms over a broad range of 4 standard deviations. The 10-item simulated PROMIS CAT outperformed all other forms. CONCLUSION: Improved items and instruments yielded better information. The PROMIS HAQ is currently available and considered validated. The new PROMIS short forms, after validation, are likely to represent further improvement. CAT-based physical function/disability assessment offers superior performance over static forms of equal length.
Fries, James FCella, DavidRose, MatthiasKrishnan, EswarBruce, BonnieU01 AR052158/AR/NIAMS NIH HHS/United StatesU01 AR52177/AR/NIAMS NIH HHS/United StatesConsensus Development ConferenceResearch Support, N.I.H., ExtramuralCanadaThe Journal of rheumatologyJ Rheumatol. 2009 Sep;36(9):2061-6.
Keywords: *Disability Evaluation; *Outcome Assessment (Health Care); Arthritis/diagnosis/*physiopathology; Health Surveys; Humans; Prognosis; Reproducibility of Results

Proposta para a construo de um Teste Adaptativo Informatizado baseado na Teoria da Resposta ao Item (Proposal for the construction of a Computerized Adaptive Testing based on the Item Response Theory) - Moreira Junior, F. J.; Andrade, D. F. (2009)
(In Portguese)

Quantifying the impact of compromised items in CAT - Guo, F. (2009)
{PDF File, 438 KB}

Reduction in patient burdens with graphical computerized adaptive testing on the ADL scale: tool development and simulation - Chien, T. W.; Wu, H. M.; Wang, W-C.; Castillo, R. V.; Chou, W. (Health and Quality of Life Outcomes, 2009)
BACKGROUND: The aim of this study was to verify the effectiveness and efficacy of saving time and reducing burden for patients, nurses, and even occupational therapists through computer adaptive testing (CAT). METHODS: Based on an item bank of the Barthel Index (BI) and the Frenchay Activities Index (FAI) for assessing comprehensive activities of daily living (ADL) function in stroke patients, we developed a visual basic application (VBA)-Excel CAT module, and (1) investigated whether the averaged test length via CAT is shorter than that of the traditional all-item-answered non-adaptive testing (NAT) approach through simulation, (2) illustrated the CAT multimedia on a tablet PC showing data collection and response errors of ADL clinical functional measures in stroke patients, and (3) demonstrated the quality control of endorsing scale with fit statistics to detect responding errors, which will be further immediately reconfirmed by technicians once patient ends the CAT assessment. RESULTS: The results show that endorsed items could be shorter on CAT (M = 13.42) than on NAT (M = 23) at 41.64% efficiency in test length. However, averaged ability estimations reveal insignificant differences between CAT and NAT. CONCLUSION: This study found that mobile nursing services, placed at the bedsides of patients could, through the programmed VBA-Excel CAT module, reduce the burden to patients and save time, more so than the traditional NAT paper-and-pencil testing appraisals.
Chien, Tsair-WeiWu, Hing-ManWang, Weng-ChungCastillo, Roberto VasquezChou, WillyComparative StudyValidation StudiesEnglandHealth and quality of life outcomesHealth Qual Life Outcomes. 2009 May 5;7:39.
Keywords: *Activities of Daily Living; *Computer Graphics; *Computer Simulation; *Diagnosis, Computer-Assisted; Female; Humans; Male; Point-of-Care Systems; Reproducibility of Results; Stroke/*rehabilitation; Taiwan; United States

Replenishing a computerized adaptive test of patient-reported daily activity functioning - Haley, S. M.; Ni, P.; Jette, A. M.; Tao, W.; Moed, R.; Meyers, D.; Ludlow, L. H. (Quality of Life Research, 2009)
PURPOSE: Computerized adaptive testing (CAT) item banks may need to be updated, but before new items can be added, they must be linked to the previous CAT. The purpose of this study was to evaluate 41 pretest items prior to including them into an operational CAT. METHODS: We recruited 6,882 patients with spine, lower extremity, upper extremity, and nonorthopedic impairments who received outpatient rehabilitation in one of 147 clinics across 13 states of the USA. Forty-one new Daily Activity (DA) items were administered along with the Activity Measure for Post-Acute Care Daily Activity CAT (DA-CAT-1) in five separate waves. We compared the scoring consistency with the full item bank, test information function (TIF), person standard errors (SEs), and content range of the DA-CAT-1 to the new CAT (DA-CAT-2) with the pretest items by real data simulations. RESULTS: We retained 29 of the 41 pretest items. Scores from the DA-CAT-2 were more consistent (ICC = 0.90 versus 0.96) than DA-CAT-1 when compared with the full item bank. TIF and person SEs were improved for persons with higher levels of DA functioning, and ceiling effects were reduced from 16.1% to 6.1%. CONCLUSIONS: Item response theory and online calibration methods were valuable in improving the DA-CAT.
Haley, Stephen MNi, PengshengJette, Alan MTao, WeiMoed, RichardMeyers, DougLudlow, Larry HK02 HD45354-01/HD/NICHD NIH HHS/United StatesResearch Support, N.I.H., ExtramuralNetherlandsQuality of life research : an international journal of quality of life aspects of treatment, care and rehabilitationQual Life Res. 2009 May;18(4):461-71. Epub 2009 Mar 14.
Keywords: *Activities of Daily Living; *Disability Evaluation; *Questionnaires; *User-Computer Interface; Adult; Aged; Cohort Studies; Computer-Assisted Instruction; Female; Humans; Male; Middle Aged; Outcome Assessment (Health Care)/*methods

Studying the Equivalence of Computer-Delivered and Paper-Based Administrations of the Raven Standard Progressive Matrices Test - Arce-Ferrer, Alvaro J.; Martínez Guzmán, Elvira (Educational and Psychological Measurement, 2009)
This study investigates the effect of mode of administration of the Raven Standard Progressive Matrices test on distribution, accuracy, and meaning of raw scores. A random sample of high school students take counterbalanced paper-and-pencil and computer-based administrations of the test and answer a questionnaire surveying preferences for computer-delivered test administrations. Administration mode effect is studied with repeated measures multivariate analysis of variance, internal consistency reliability estimates, and confirmatory factor analysis approaches. Results show a lack of test mode effect on distribution, accuracy, and meaning of raw scores. Participants indicate their preferences for the computer-delivered administration of the test. The article discusses findings in light of previous studies of the Raven Standard Progressive Matrices test.

Termination criteria in computerized adaptive tests: Variable-length CATs are not biased. - Babcock, B.; Weiss, D. J. (2009)

Test overlap rate and item exposure rate as indicators of test security in CATs - Barrada, J; Olea, J.; Ponsoda, V.; Abad, F. J. (2009)
PDF File, 261 K

Using automatic item generation to address item demands for CAT - Lai, H.; Alves, C.; Gierl, M. J. (2009)
{PDF File, 320 KB}

Utilizing the generalized likelihood ratio as a termination criterion - Thompson, N. A. (2009)
Computer-based testing can be used to classify examinees into mutually exclusive groups. Currently, the predominant psychometric algorithm for designing computerized classification tests (CCTs) is the sequential probability ratio test (SPRT; Reckase, 1983) based on item response theory (IRT). The SPRT has been shown to be more efficient than confidence intervals around θ estimates as a method for CCT delivery (Spray & Reckase, 1996; Rudner, 2002). More recently, it was demonstrated that the SPRT, which only uses fixed values, is less efficient than a generalized form which tests whether a given examinee’s θ is below θ1or above θ2 (Thompson, 2007). This formulation allows the indifference region to vary based on observed data. Moreover, this composite hypothesis formulation better represents the conceptual purpose of the test, which is to test whether θ is above or below the cutscore. The purpose of this study was to explore the specifications of the new generalized likelihood ratio (GLR; Huang, 2004). As with the SPRT, the efficiency of the procedure depends on the nominal error rates and the distance between θ1 and θ2 (Eggen, 1999). This study utilized a monte-carlo approach, with 10,000 examinees simulated under each condition, to evaluate differences in efficiency and accuracy due to hypothesis structure, nominal error rate, and indifference region size. The GLR was always at least as efficient as the fixed-point SPRT while maintaining equivalent levels of accuracy.
{PDF File, 194 KB}

Validation of the MMPI-2 computerized adaptive version (MMPI-2-CA) in a correctional intake facility - Forbey, J. D.; Ben-Porath, Y. S.; Gartland, D. (Psychological Services, 2009)
Computerized adaptive testing in personality assessment can improve efficiency by significantly reducing the number of items administered to answer an assessment question. The time savings afforded by this technique could be of particular benefit in settings where large numbers of psychological screenings are conducted, such as correctional facilities. In the current study, item and time savings, as well as the test–retest and extratest correlations associated with an audio augmented administration of all the scales of the Minnesota Multiphasic Personality Inventory (MMPI)-2 Computerized Adaptive (MMPI-2-CA) are reported. Participants include 366 men, ages 18 to 62 years (M = 33.04, SD = 10.40), undergoing intake into a large Midwestern state correctional facility. Results of the current study indicate considerable item and corresponding time savings for the MMPI-2-CA compared to conventional administration of the test, as well as comparability in terms of test–retest and correlations with external measures. Future directions of adaptive personality testing are discussed.

When cognitive diagnosis meets computerized adaptive testing: CD-CAT - Cheng, Y (Psychometrika, 2009)

Adaptive measurement of individual change - Kim-Kang, G.; Weiss, D. J. (Zeitschrift für Psychologie / Journal of Psychology, 2008)
{PDF file, 568 KB}

Adaptive models of psychological testing - van der Linden, W. J. (Zeitschrift für Psychologie / Journal of Psychology, 2008)

Adaptive Models of Psychological Testing - van der Linden, W. J. (Zeitschrift für Psychologie / Journal of Psychology, 2008)