Optimal testing with easy items in computerized adaptive testing - Theo Eggen; Verschoor, A. (Paper presented at the conference of the International Association for Educational Assessment, 2003)

Predicting item exposure parameters in computerized adaptive testing - Chen, S-Y.; Doong, H. (Paper presented at the annual meeting of the American Educational Research Association, 2003)
{PDF file, 239 KB}

Psychometric and psychological effects of item selection and review on computerized testing - Revuelta, J.; Ximénez, M. C.; Olea, J. (Educational and Psychological Measurement, 2003)

Psychometric and Psychological Effects of Item Selection and Review on Computerized Testing - Revuelta, Javier; Ximénez, M. Carmen; Olea, Julio (Educational and Psychological Measurement, 2003)
Psychometric properties of computerized testing, together with anxiety and comfort of examinees, are investigated in relation to item selection routine and the opportunity for response review. Two different hypotheses involving examinee anxiety were used to design test properties: perceived control and perceived performance. The study involved three types of administration of a computerized English test for Spanish speakers (adaptive, easy adaptive, and fixed) and four review conditions (no review, review at end, review by blocks of 5 items, and review item-by-item). These were applied to a sample of 557 first-year psychology undergraduate students to examine main and interaction effects of test type and review on psychometric and psychological variables. Statistically significant effects were found in test precision among the different types of test. Response review improved ability estimates and increased testing time. No psychological effects on anxiety were found. Examinees in all review conditions considered more important the possibility of review than those who were not allowed to review. These results concur with previous findings on examinees' preference for item review and raise some issues that should be addressed in the field of tests with item review.

Psychometric properties of several computer-based test designs with ideal and constrained item pools - Jodoin, M. G. (Dissertation Abstracts International: Section B: The Sciences & Engineering, 2003)
The purpose of this study was to compare linear fixed length test (LFT), multi stage test (MST), and computer adaptive test (CAT) designs under three levels of item pool quality, two levels of match between test and item pool content specifications, two levels of test length, and several levels of exposure control expected to be practical for a number of testing programs. This design resulted in 132 conditions that were evaluated using a simulation study with 9000 examinees on several measures of overall measurement precision including reliability, the mean error and root mean squared error between true and estimated ability levels, classification precision including decision accuracy, false positive and false negative rates, and Kappa for cut scores corresponding to 30%, 50%, and 85% failure rates, and conditional measurement precision with the conditional root mean squared error between true and estimated ability levels conditioned on 25 true ability levels. Test reliability, overall and conditional measurement precision, and classification precision increased with item pool quality and test length, and decreased with less adequate match between item pool and test specification match. In addition, as the maximum exposure rate decreased and the type of exposure control implemented became more restrictive, test reliability, overall and conditional measurement precision, and classification precision decreased. Within item pool quality, match between test and item pool content specifications, test length, and exposure control, CAT designs showed superior psychometric properties as compared to MST designs which in turn were superior to LFT designs. However, some caution is warranted in interpreting these results since the ability of the automated test assembly software to construct test that met specifications was limited in conditions where pool usage was high. The practical importance of the differences between test designs on the evaluation criteria studied is discussed with respect to the inferences test users seek to make from test scores and nonpsychometric factors that may be important in some testing programs. (PsycINFO Database Record (c) 2004 APA, all rights reserved).

Recalibration of IRT item parameters in CAT: Sparse data matrices and missing data treatments - Harmes, J. C.; Parshall, C. G.; Kromrey, J. D. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2003)
(PDF file, 626 K

The relationship between item exposure and test overlap in computerized adaptive testing - Chen, S.; Ankenmann, R. D.; Spray, J. A. (Journal of Educational Measurement, 2003)

The relationship between item exposure and test overlap in computerized adaptive testing - Chen, S.; Ankenmann, R. D.; Spray, J. A. (Journal of Educational Measurement, 2003)

The relationship between item exposure and test overlap in computerized adaptive testing - Chen, S-Y.; Ankemann, R. D.; Spray, J. A. (Journal of Educational Measurement, 2003)
The purpose of this article is to present an analytical derivation for the mathematical form of an average between-test overlap index as a function of the item exposure index, for fixed-length computerized adaptive tests (CATs). This algebraic relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. The results indicate that, in fixed-length CATs, control of the average between-test overlap is achieved via the mean and variance of the item exposure rates of the items that constitute the CAT item pool. The mean of the item exposure rates is easily manipulated. Control over the variance of the item exposure rates can be achieved via the maximum item exposure rate (r-sub(max)). Therefore, item exposure control methods which implement a specification of r-sub(max) (e.g., J. B. Sympson and R. D. Hetter, 1985) provide the most direct control at both the item and test levels. (PsycINFO Database Record (c) 2005 APA )
Keywords: (Statistical); Adaptive Testing; Computer Assisted Testing; Human Computer; Interaction computerized adaptive testing; Item Analysis; Item Analysis (Test); Test Items

A sequential Bayes procedure for item calibration in multi-stage testing - van der Linden, W. J.; Alan D Mead (2003)

A simulation study to compare CAT strategies for cognitive diagnosis - Xu, X.; Chang, Hua-Hua; Douglas, J. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2003)
{PDF file, 250 KB}

Small sample estimation in dichotomous item response models: Effect of priors based on judgmental information on the accuracy of item parameter estimates - Swaminathan, H.; Hambleton, R. K.; Sireci, S. G.; Xing, D.; Rizavi, S. M. (Applied Psychological Measurement, 2003)
Large item banks with properly calibrated test items are essential for ensuring the validity of computer-based tests. At the same time, item calibrations with small samples are desirable to minimize the amount of pretesting and limit item exposure. Bayesian estimation procedures show considerable promise with small examinee samples. The purposes of the study were (a) to examine how prior information for Bayesian item parameter estimation can be specified and (b) to investigate the relationship between sample size and the specification of prior information on the accuracy of item parameter estimates. The results of the simulation study were clear: Estimation of item response theory (IRT) model item parameters can be improved considerably. Improvements in the one-parameter model were modest; considerable improvements with the two- and three-parameter models were observed. Both the study of different forms of priors and ways to improve the judgmental data used in forming the priors appear to be promising directions for future research.
Sage Publications, US

Some alternatives to Sympson-Hetter item-exposure control in computerized adaptive testing - van der Linden, W. J. (Journal of Educational and Behavioral Statistics, 2003)
TheHetter and Sympson (1997; 1985) method is a method of probabilistic item-exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic set of values for the examinees’ ability parameter. Formal properties of the method are identified that help us explain why this iterative process can be slow and does not guarantee admissibility. In addition, some alternatives to the SH method are introduced. The behavior of these alternatives was estimated for an adaptive test from an item pool from the Law School Admission Test (LSAT). Two of the alternatives showed attractive behavior and converged smoothly to admissibility for all items in a relatively small number of iteration steps.
Keywords: Adaptive Testing; Computer Assisted Testing; Test Items computerized adaptive testing

Standard-setting issues in computerized-adaptive testing - Gushta, M. M. (Paper Prepared for Presentation at the Annual Conference of the Canadian Society for Studies in Education, 2003)

Statistical detection and estimation of differential item functioning in computerized adaptive testing - Feng, X. (Dissertation Abstracts International: Section B: The Sciences & Engineering, 2003)
Differential item functioning (DIF) is an important issue in large scale standardized testing. DIF refers to the unexpected difference in item performances among groups of equally proficient examinees, usually classified by ethnicity or gender. Its presence could seriously affect the validity of inferences drawn from a test. Various statistical methods have been proposed to detect and estimate DIF. This dissertation addresses DIF analysis in the context of computerized adaptive testing (CAT), whose item selection algorithm adapts to the ability level of each individual examinee. In a CAT, a DIF item may be more consequential and more detrimental be cause fewer items are administered in a CAT than in a traditional paper-and-pencil test and because the remaining sequence of items presented to examinees depends in part on their responses to the DIF item. Consequently, an efficient, stable and flexible method to detect and estimate CAT DIF becomes necessary and increasingly important. We propose simultaneous implementations of online calibration and DIF testing. The idea is to perform online calibration of an item of interest separately in the focal and reference groups. Under any specific parametric IRT model, we can use the (online) estimated latent traits as covariates and fit a nonlinear regression model to each of the two groups. Because of the use of the estimated, not the true , the regression fit has to adjust for the covariate "measurement errors". It turns out that this situation fits nicely into the framework of nonlinear error-in-variable modelling, which has been extensively studied in statistical literature. We develop two bias-correction methods using asymptotic expansion and conditional score theory. After correcting the bias caused by measurement error, one can perform a significance test to detect DIF with the parameter estimates for different groups. This dissertation also discusses some general techniques to handle measurement error modelling with different IRT models, including the three-parameter normal ogive model and polytomous response models. Several methods of estimating DIF are studied as well. Large sample properties are established to justify the proposed methods. Extensive simulation studies show that the resulting methods perform well in terms of Type-I error rate control, accuracy in estimating DIF and power against both unidirectional and crossing DIF. (PsycINFO Database Record (c) 2004 APA, all rights reserved).

Strategies for controlling item exposure in computerized adaptive testing with polytomously scored items - Davis, L. L. (Dissertation Abstracts International: Section B: The Sciences & Engineering, 2003)
Choosing a strategy for controlling the exposure of items to examinees has become an integral part of test development for computerized adaptive testing (CAT). Item exposure can be controlled through the use of a variety of algorithms which modify the CAT item selection process. This may be done through a randomization, conditional selection, or stratification approach. The effectiveness of each procedure as well as the degree to which measurement precision is sacrificed has been extensively studied with dichotomously scored item pools. However, only recently have researchers begun to examine these procedures in polytomously scored item pools. The current study investigated the performance of six different exposure control mechanisms under three polytomous IRT models in terms of measurement precision, test security, and ease of implementation. The three models examined in the current study were the partial credit, generalized partial credit, and graded response models. In addition to a no exposure control baseline condition, the randomesque, within .10 logits, Sympson-Hetter, conditional Sympson-Hetter, a-Stratified, and enhanced a-Stratified procedures were implemented to control item exposure rates. The a-Stratified and enhanced a-Stratified procedures were not evaluated with the partial credit model. Two variations of the randomesque and within .10 logits procedures were also examined which varied the size of the item group from which the next item to be administered was randomly selected. The results of this study were remarkably similar for all three models and indicated that the randomesque and within .10 logits procedures, when implemented with the six item group variation, provide the best option for controlling exposure rates when impact to measurement precision and ease of implementation are considered. The three item group variations of the procedures were, however, ineffective in controlling exposure, overlap, and pool utilization rates to desired levels. The Sympson-Hetter and conditional Sympson-Hetter procedures were difficult and time consuming to implement, and while they did control exposure rates to the target level, their performance in terms of item overlap (for the Sympson-Hetter) and pool utilization were disappointing. The a-Stratified and enhanced a-Stratified procedures both turned in surprisingly poor performances across all variables. (PsycINFO Database Record (c) 2004 APA, all rights reserved).

Strategies for controlling item exposure in computerized adaptive testing with the generalized partial credit model - Davis, L. L. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2003)
PDF file, 620 K

Strategies for controlling testlet exposure rates in computerized adaptive testing systems - Boyd, A. M (2003)
{PDF file, 485 KB}

Student modeling and ab initio language learning - Heift, T.; Schulze, M. (System, 2003)
Provides examples of student modeling techniques that have been employed in computer-assisted language learning over the past decade. Describes two systems for learning German: "German Tutor" and "Geroline." Shows how a student model can support computerized adaptive language testing for diagnostic purposes in a Web-based language learning environment that does not rely on parsing technology. (Author/VWL)

A study of the feasibility of Internet administration of a computerized health survey: The Headache Impact Test (HIT) - Bayliss, M.S.; Dewey, J.E.; Dunlap, I; et. al. (Quality of Life Research, 2003)

Ten recommendations for advancing patient-centered outcomes measurement for older persons - McHorney, C. A. (Annals of Internal Medicine, 2003)
The past 50 years have seen great progress in the measurement of patient-based outcomes for older populations. Most of the measures now used were created under the umbrella of a set of assumptions and procedures known as classical test theory. A recent alternative for health status assessment is item response theory. Item response theory is superior to classical test theory because it can eliminate test dependency and achieve more precise measurement through computerized adaptive testing. Computerized adaptive testing reduces test administration times and allows varied and precise estimates of ability. Several key challenges must be met before computerized adaptive testing becomes a productive reality. I discuss these challenges for the health assessment of older persons in the form of 10 "Ds": things we need to deliberate, debate, decide, and do.
1539-3704Journal ArticleReview
Keywords: *Health Status Indicators; Aged; Geriatric Assessment/*methods; Humans; Patient-Centered Care/*methods; Research Support, U.S. Gov't, Non-P.H.S.

Test information targeting strategies for adaptive multistage testlet designs - Luecht, RM; Burgin, W. L. (Paper presented at the Annual meeting of the National Council on Measurement in Education, 2003)
PDF file, 179 K

Tests adaptativos informatizados (Computerized adaptive testing) - Olea, J.; Ponsoda, V. (2003)
[In Spanish]

Test-score comparability, ability estimation, and item-exposure control in computerized adaptive testing - Chang, Hua-Hua; Ying, Z. (Paper presented at the Annual meeting of the National Council on Measurement in Education, 2003)

Timing behavior in computerized adaptive testing: Response times for correct and incorrect answers are not related to general fluid intelligence/Zum Zeitverhalten beim computergestützten adaptiveb Testen: Antwortlatenzen bei richtigen und falschen Lösun - Rammsayer, Thomas; Brandler, Susanne (Zeitschrift für Differentielle und Diagnostische Psychologie, 2003)
Examined the effects of general fluid intelligence on item response times for correct and false responses in computerized adaptive testing. After performing the CFT3 intelligence test, 80 individuals (aged 17-44 yrs) completed perceptual and cognitive discrimination tasks. Results show that response times were related neither to the proficiency dimension reflected by the task nor to the individual level of fluid intelligence. Furthermore, the false > correct-phenomenon as well as substantial positive correlations between item response times for false and correct responses were shown to be independent of intelligence levels. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Cognitive Ability; Intelligence; Perception; Reaction Time computerized adaptive testing

To stratify or not: An investigation of CAT item selection procedures under practical constraints - Deng, H.; Ansley, T. (Paper presented at the Annual meeting of the National Council on Measurement in Education, 2003)
{PDF file, 186 KB}

Using moving averages to assess test and item security in computer-based testing (Center for Educational Assessment Research Report No 468) - Han, N. (2003)

Using response times to detect aberrant responses in computerized adaptive testing - van der Linden, W. J.; van Krimpen-Stoop, E. M. L. A. (Psychometrika, 2003)
A lognormal model for response times is used to check response times for aberrances in examinee behavior on computerized adaptive tests. Both classical procedures and Bayesian posterior predictive checks are presented. For a fixed examinee, responses and response times are independent; checks based on response times offer thus information independent of the results of checks on response patterns. Empirical examples of the use of classical and Bayesian checks for detecting two different types of aberrances in response times are presented. The detection rates for the Bayesian checks outperformed those for the classical checks, but at the cost of higher false-alarm rates. A guideline for the choice between the two types of checks is offered.
Keywords: Adaptive Testing; Behavior; Computer Assisted Testing; computerized adaptive testing; Models; person Fit; Prediction; Reaction Time

Accuracy of the ability estimate and the item exposure rate under multidimensional adaptive testing with item constraints - Li, Y. H.; Yu, N. Y. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
#LI02-01

Adaptive testing without IRT in the presence of multidimensionality (Research Report 02-09) - Yan, D.; Lewis, C.; Stocking, M. (2002)

Advances in quality of life measurements in oncology patients - Cella, D.; Chang, C-H.; Lai, J. S.; Webster, K. (Seminars in Oncology, 2002)
Accurate assessment of the quality of life (QOL) of patients can provide important clinical information to physicians, especially in the area of oncology. Changes in QOL are important indicators of the impact of a new cytotoxic therapy, can affect a patient's willingness to continue treatment, and may aid in defining response in the absence of quantifiable endpoints such as tumor regression. Because QOL is becoming an increasingly important aspect in the management of patients with malignant disease, it is vital that the instruments used to measure QOL are reliable and accurate. Assessment of QOL involves a multidimensional approach that includes physical, functional, social, and emotional well-being, and the most comprehensive instruments measure at least three of these domains. Instruments to measure QOL can be generic (eg, the Nottingham Health Profile), targeted toward specific illnesses (eg, Functional Assessment of Cancer Therapy - Lung), or be a combination of generic and targeted. Two of the most widely used examples of the combination, or hybrid, instruments are the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 Items and the Functional Assessment of Chronic Illness Therapy. A consequence of the increasing international collaboration in clinical trials has been the growing necessity for instruments that are valid across languages and cultures. To assure the continuing reliability and validity of QOL instruments in this regard, item response theory can be applied. Techniques such as item response theory may be used in the future to construct QOL item banks containing large sets of validated questions that represent various levels of QOL domains. As QOL becomes increasingly important in understanding and approaching the overall management of cancer patients, the tools available to clinicians and researchers to assess QOL will continue to evolve. While the instruments currently available provide reliable and valid measurement, further improvements in precision and application are anticipated.
0093-7754 (Print)Journal ArticleReview
Keywords: *Quality of Life; *Sickness Impact Profile; Cross-Cultural Comparison; Culture; Humans; Language; Neoplasms/*physiopathology; Questionnaires

Applicable adaptive testing models for school teachers - Chang-Hwa, W. A.; Chuang, C-L. (Educational Media International, 2002)
The purpose of this study was to investigate the attitudinal effects on SPRT adaptive testing environment for junior high school students. Subjects were 39 eighth graders from a selected junior high school. Major instruments for the study were the Junior High School Natural Sciences Adaptive Testing System driven by the SPRT algorithm, and a self-developed attitudinal questionnaire, factors examined include: test anxiety, examinee preference, adaptability of the test, and acceptance of the test result. The major findings were that overall, junior high school students" attitudes towards computerized adaptive tests were positive, no significant correlations existed between test attitude and the test length. The results indicated that junior high school students generally have positive attitudes towards adaptive testing.Modèles de tests d"adaptation à l"usage des enseignants. L"objectif de cette étude était d"enquêter sur les effets causés par une passation de tests d"adaptation ( selon l"algorithme "Sequential Probability Radio Test " (SPRT) ) dans une classe de trente-neuf élèves de huitième année du secondaire inférieur. Les principaux instruments utilisés ont été ceux du système de tests d"adaptation (avec le SPRT) et destiné aux classes de sciences naturelles du degré secondaire inférieur. Un questionnaire d"attitude, développé par nos soins, a également été utilisé pour examiner les facteurs suivants: test d"anxiété, préférence des candidats, adaptabilité du test et acceptation des résultats. Les principales conclusions ont été que, dans l"ensemble, l"attitude des élèves du secondaire inférieur face aux tests d"adaptation informatisés a été positive, aucune corrélation significative existant entre cette attitude et la longueur des tests. Les résultats démontrent aussi que les élèves du secondaire ont une attitude généralement positive envers les tests d"adaptation.Test Modelle zur Anwendung durch Klassenlehrer Zweck dieser Untersuchung war, die Auswirkungen über die Einstellung von Jun. High School Schülern im Zusammenhang mit dem SPRT Testumfeld zu untersuchen. 39 Achtklässler einer Jun. High School nahmen an dem Test teil. Die Untersuchung stützte sich hauptsächlich auf das Jun. High School Natural. Sciences Adaptive Testing System, das auf dem SPRT Rechnungsverfahren basiert sowie einem selbst erstellten Fragebogen mit folgenden Faktoren: Testängste, Präferenzen der Testperson, Geeignetheit des Tests, Anerkennung des Testergebnisses. Es stellte sich heraus, dass die Einstellung der Studenten zu den Computer adaptierten Tests im allgemeinen positiv waren; es ergaben sich keine bedeutsamen Wechselwirkungen zwischen persönlicher Testeinstellung und Testlänge. Die Ergebnisse belegen, dass Jun. High School Schüler im allgemeinen eine positive Haltung zu adaptierten Tests haben.

Application of an empirical Bayes enhancement of Mantel-Haenszel differential item functioning analysis to a computerized adaptive test - Zwick, R.; Thayer, D. T. (Applied Psychological Measurement, 2002)

Assessing the efficiency of item selection in computerized adaptive testing - Weissman, A. (2002)

Assessing tobacco beliefs among youth using item response theory models - Panter, A. T.; Reeve, B. B. (Drug and Alcohol Dependence, 2002)
Successful intervention research programs to prevent adolescent smoking require well-chosen, psychometrically sound instruments for assessing smoking prevalence and attitudes. Twelve thousand eight hundred and ten adolescents were surveyed about their smoking beliefs as part of the Teenage Attitudes and Practices Survey project, a prospective cohort study of predictors of smoking initiation among US adolescents. Item response theory (IRT) methods are used to frame a discussion of questions that a researcher might ask when selecting an optimal item set. IRT methods are especially useful for choosing items during instrument development, trait scoring, evaluating item functioning across groups, and creating optimal item subsets for use in specialized applications such as computerized adaptive testing. Data analytic steps for IRT modeling are reviewed for evaluating item quality and differential item functioning across subgroups of gender, age, and smoking status. Implications and challenges in the use of these methods for tobacco onset research and for assessing the developmental trajectories of smoking among youth are discussed.
0376-8716Journal Article
Keywords: *Attitude to Health; *Culture; *Health Behavior; *Questionnaires; Adolescent; Adult; Child; Female; Humans; Male; Models, Statistical; Smoking/*epidemiology

Can examinees use judgments of item difficulty to improve proficiency estimates on computerized adaptive vocabulary tests? - Vispoel, W. P.; Clough, S. J.; Bleiler, T.; Hendrickson, A. B.; Ihrig, D. (Journal of Educational Measurement, 2002)

Comparing three item selection approaches for computerized adaptive testing with content balancing requirement - Leung, C-K..; Chang, Hua-Hua; Hau, K-T. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 226 KB}

A comparison of computer mastery models when pool characteristics vary - Smith, R. L.; Lewis, C. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 692 KB}

A comparison of item selection techniques and exposure control mechanisms in CATs using the generalized partial credit model - Pastor, D. A.; Dodd, B. G.; Chang, Hua-Hua (Applied Psychological Measurement, 2002)
The use of more performance items in large-scale testing has led to an increase in the research investigating the use of polytomously scored items in computer adaptive testing (CAT). Because this research has to be complemented with information pertaining to exposure control, the present research investigated the impact of using five different exposure control algorithms in two sized item pools calibrated using the generalized partial credit model. The results of the simulation study indicated that the a-stratified design, in comparison to a no-exposure control condition, could be used to reduce item exposure and overlap, increase pool utilization, and only minorly degrade measurement precision. Use of the more restrictive exposure control algorithms, such as the Sympson-Hetter and conditional Sympson-Hetter, controlled exposure to a greater extent but at the cost of measurement precision. Because convergence of the exposure control parameters was problematic for some of the more restrictive exposure control algorithms, use of the more simplistic exposure control mechanisms, particularly when the test length to item pool size ratio is large, is recommended. (PsycINFO Database Record (c) 2005 APA ) (journal abstract)
Keywords: (Statistical); Adaptive Testing; Algorithms computerized adaptive testing; Computer Assisted Testing; Item Analysis; Item Response Theory; Mathematical Modeling

A comparison of non-deterministic procedures for the adaptive assessment of knowledge - Hockemeyer, C. (Psychologische Beitrge, 2002)

Comparison of the psychometric properties of several computer-based test designs for credentialing exams - Jodoin, M.; Zenisky, A. L.; Hambleton, R. K. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 261 KB}

Computer adaptive testing: The impact of test characteristics on perceived performance and test takers' reactions - Tonidandel, S. (Dissertation Abstracts International: Section B: the Sciences & Engineering, 2002)
This study examined the relationship between characteristics of adaptive testing and test takers' subsequent reactions to the test. Participants took a computer adaptive test in which two features, the difficulty of the initial item and the difficulty of subsequent items, were manipulated. These two features of adaptive testing determined the number of items answered correctly by examinees and their subsequent reactions to the test. The data show that the relationship between test characteristics and reactions was fully mediated by perceived performance on the test. In addition, the impact of feedback on reactions to adaptive testing was also evaluated. In general, feedback that was consistent with perceptions of performance had a positive impact on reactions to the test. Implications for adaptive test design concerning maximizing test takers' reactions are discussed. (PsycINFO Database Record (c) 2003 APA, all rights reserved).
Keywords: computerized adaptive testing

Computer-adaptive testing: The impact of test characteristics on perceived performance and test takers’ reactions - Tonidandel, S.; Quiñones, M. A.; Adams, A. A. (Journal of Applied Psychology, 2002)

Computerised adaptive testing - Latu, E.; Chapman, E. (British Journal of Educational Technology, 2002)
Considers the potential of computer adaptive testing (CAT). Discusses the use of CAT instead of traditional paper and pencil tests, identifies decisions that impact the efficacy of CAT, and concludes that CAT is beneficial when used to its full potential on certain types of tests. (LRW)
Keywords: computerized adaptive testing

Confirmatory item factor analysis using Markov chain Monte Carlo estimation with applications to online calibration in CAT - Segall, D. O. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)

Constraining item exposure in computerized adaptive testing with shadow tests (Research Report No. 02-06) - van der Linden, W. J.; Veldkamp, B. P. (2002)

Content-stratified random item selection in computerized classification testing - Guille, R. Lipner, R. S.; Norcini, J. J. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
#GU02-01

Controlling item exposure and maintaining item security - Davey, T.; Nering, M. (2002)

Data sparseness and on-line pretest item calibration-scaling methods in CAT - Ban, J-C.; Hanson, B. A.; Yi, Q.; Harris, D. J. (Journal of Educational Measurement, 2002)
Compared and evaluated 3 on-line pretest item calibration-scaling methods (the marginal maximum likelihood estimate with 1 expectation maximization [EM] cycle [OEM] method, the marginal maximum likelihood estimate with multiple EM cycles [MEM] method, and M. L. Stocking's Method B) in terms of item parameter recovery when the item responses to the pretest items in the pool are sparse. Simulations of computerized adaptive tests were used to evaluate the results yielded by the three methods. The MEM method produced the smallest average total error in parameter estimation, and the OEM method yielded the largest total error (PsycINFO Database Record (c) 2005 APA )
Keywords: Computer Assisted Testing; Educational Measurement; Item Response Theory; Maximum Likelihood; Methodology; Scaling (Testing); Statistical Data

Detection of person misfit in computerized adaptive tests with polytomous items - van Krimpen-Stoop, E. M. L. A.; Meijer, R. R. (Applied Psychological Measurement, 2002)
Item scores that do not fit an assumed item response theory model may cause the latent trait value to be inaccurately estimated. For a computerized adaptive test (CAT) using dichotomous items, several person-fit statistics for detecting mis.tting item score patterns have been proposed. Both for paper-and-pencil (P&P) tests and CATs, detection ofperson mis.t with polytomous items is hardly explored. In this study, the nominal and empirical null distributions ofthe standardized log-likelihood statistic for polytomous items are compared both for P&P tests and CATs. Results showed that the empirical distribution of this statistic differed from the assumed standard normal distribution for both P&P tests and CATs. Second, a new person-fit statistic based on the cumulative sum (CUSUM) procedure from statistical process control was proposed. By means ofsimulated data, critical values were determined that can be used to classify a pattern as fitting or misfitting. The effectiveness of the CUSUM to detect simulees with item preknowledge was investigated. Detection rates using the CUSUM were high for realistic numbers ofdisclosed items.

Developing tailored instruments: Item banking and computerized adaptive assessment - Thissen, D. (Paper presented at the conference “Advances in Health Outcomes Measurement, 2002)
{PDF file, 170 KB}

The development and evaluation of a computer-adaptive testing application for English language - Lilley, M; Barker, T (Paper presented at the 2002 Computer-Assisted Testing Conference, 2002)
{PDF file, 308 KB}

Development of an index of physical functional health status in rehabilitation - Hart, D. L.; Wright, B. D. (Archives of Physical Medicine and Rehabilitation, 2002)
OBJECTIVE: To describe (1) the development of an index of physical functional health status (FHS) and (2) its hierarchical structure, unidimensionality, reproducibility of item calibrations, and practical application. DESIGN: Rasch analysis of existing data sets. SETTING: A total of 715 acute, orthopedic outpatient centers and 62 long-term care facilities in 41 states participating with Focus On Therapeutic Outcomes, Inc. PATIENTS: A convenience sample of 92,343 patients (40% male; mean age +/- standard deviation [SD], 48+/-17y; range, 14-99y) seeking rehabilitation between 1993 and 1999. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Patients completed self-report health status surveys at admission and discharge. The Medical Outcomes Study 36-Item Short-Form Health Survey's physical functioning scale (PF-10) is the foundation of the physical FHS. The Oswestry Low Back Pain Disability Questionnaire, Neck Disability Index, Lysholm Knee Questionnaire, items pertinent to patients with upper-extremity impairments, and items pertinent to patients with more involved neuromusculoskeletal impairments were cocalibrated into the PF-10. RESULTS: The final FHS item bank contained 36 items (patient separation, 2.3; root mean square measurement error, 5.9; mean square +/- SD infit, 0.9+/-0.5; outfit, 0.9+/-0.9). Analyses supported empirical item hierarchy, unidimensionality, reproducibility of item calibrations, and content and construct validity of the FHS-36. CONCLUSIONS: Results support the reliability and validity of FHS-36 measures in the present sample. Analyses show the potential for a dynamic, computer-controlled, adaptive survey for FHS assessment applicable for group analysis and clinical decision making for individual patients.
0003-9993 (Print)Journal Article
Keywords: *Health Status Indicators; *Rehabilitation Centers; Adolescent; Adult; Aged; Aged, 80 and over; Female; Health Surveys; Humans; Male; Middle Aged; Musculoskeletal Diseases/*physiopathology/*rehabilitation; Nervous System Diseases/*physiopathology/*rehabilitation; Physical Fitness/*physiology; Recovery of Function/physiology; Reproducibility of Results; Retrospective Studies

The Development of STAR Early Literacy - J. R. McBride (Presentation to the 32rd Annual National Conference on Large-Scale Assessment., 2002)

DEVELOPMENT, RELIABILITY, AND VALIDITY OF A COMPUTERIZED ADAPTIVE VERSION OF THE SCHEDULE FOR NONADAPTIVE AND ADAPTIVE PERSONALITY - Simms, L. J. (2002)

The effect of test characteristics on aberrant response patterns in computer adaptive testing - Rizavi, S. M. (Dissertation Abstracts International Section A: Humanities & Social Sciences, 2002)
The advantages that computer adaptive testing offers over linear tests have been well documented. The Computer Adaptive Test (CAT) design is more efficient than the Linear test design as fewer items are needed to estimate an examinee's proficiency to a desired level of precision. In the ideal situation, a CAT will result in examinees answering different number of items according to the stopping rule employed. Unfortunately, the realities of testing conditions have necessitated the imposition of time and minimum test length limits on CATs. Such constraints might place a burden on the CAT test taker resulting in aberrant response behaviors by some examinees. Occurrence of such response patterns results in inaccurate estimation of examinee proficiency levels. This study examined the effects of test lengths, time limits and the interaction of these factors with the examinee proficiency levels on the occurrence of aberrant response patterns. The focus of the study was on the aberrant behaviors caused by rushed guessing due to restrictive time limits. Four different testing scenarios were examined; fixed length performance tests with and without content constraints, fixed length mastery tests and variable length mastery tests without content constraints. For each of these testing scenarios, the effect of two test lengths, five different timing conditions and the interaction between these factors with three ability levels on ability estimation were examined. For fixed and variable length mastery tests, decision accuracy was also looked at in addition to the estimation accuracy. Several indices were used to evaluate the estimation and decision accuracy for different testing conditions. The results showed that changing time limits had a significant impact on the occurrence of aberrant response patterns conditional on ability. Increasing test length had negligible if not negative effect on ability estimation when rushed guessing occured. In case of performance testing high ability examinees while in classification testing middle ability examinees suffered the most. The decision accuracy was considerably affected in case of variable length classification tests. (PsycINFO Database Record (c) 2003 APA, all rights reserved).
Keywords: computerized adaptive testing

An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model - Stark, S.; F Drasgow (Applied Psychological Measurement, 2002)
Borman et al. recently proposed a computer adaptive performance appraisal system called CARS II that utilizes paired comparison judgments of behavioral stimuli. To implement this approach,the paired comparison ideal point model developed by Zinnes and Griggs was selected. In this article,the authors describe item response and information functions for the Zinnes and Griggs model and present procedures for estimating stimulus and person parameters. Monte carlo simulations were conducted to assess the accuracy of the parameter estimation procedures. The results indicated that at least 400 ratees (i.e.,ratings) are required to obtain reasonably accurate estimates of the stimulus parameters and their standard errors. In addition,latent trait estimation improves as test length increases. The implications of these results for test construction are also discussed.
Keywords: Adaptive Testing; Computer Assisted Testing; Item Response Theory; Maximum Likelihood; Personnel Evaluation; Statistical Correlation; Statistical Estimation

An empirical comparison of achievement level estimates from adaptive tests and paper-and-pencil tests - Kingsbury, G. G. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 134 KB}

An empirical comparison of achievement level estimates from adaptive tests and paper-and-pencil tests - Kingsbury, G. G. (annual meeting of the American Educational Research Association, 2002)
Keywords: computerized adaptive testing

An empirical investigation of selected multi-stage testing design variables on test assembly and decision accuracy outcomes for credentialing exams (Center for Educational Assessment Research Report No 469) - Zenisky, A. L. (2002)

Employing new ideas in CAT to a simulated reading test - Thompson, T. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 216 KB}

Étude de la distribution d'échantillonnage de l'estimateur du niveau d'habileté en testing adaptatif en fonction de deux règles d'arrêt dans le contexte de l'application du modèle de Rasch [Study of the sampling distribution of the proficiecy estima - Raîche, G.; Blais, J-G. (Mesure et évaluation en éducation, 2002)
(In French)

Evaluation of selection procedures for computerized adaptive testing with polytomous items - van Rijn, P. W.; Theo Eggen; Hemker, B. T.; Sanders, P. F. (Applied Psychological Measurement, 2002)
In the present study, a procedure that has been used to select dichotomous items in computerized adaptive testing was applied to polytomous items. This procedure was designed to select the item with maximum weighted information. In a simulation study, the item information function was integrated over a fixed interval of ability values and the item with the maximum area was selected. This maximum interval information item selection procedure was compared to a maximum point information item selection procedure. Substantial differences between the two item selection procedures were not found when computerized adaptive tests were evaluated on bias and the root mean square of the ability estimate.
References .Sage Publications, US
Keywords: computerized adaptive testing

Evaluation of selection procedures for computerized adaptive testing with polytomous items - van Rijn, P. W.; Theo Eggen; Hemker, B. T.; Sanders, P. F. (Applied Psychological Measurement, 2002)

An examination of decision-theory adaptive testing procedures - Rudner, L. M. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 46 KB}

An exploration of potentially problematic adaptive tests - Stocking, M.; Steffen, M. Golub-Smith, M. L.; Eignor, D. R. (2002)
Princeton NJ: Educational Testing Service.

Fairness issues in adaptive tests with strict time limits - Bridgeman, B.; Cline, F. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 1.287 MB}

Feasibility and acceptability of computerized adaptive testing (CAT) for fatigue monitoring in clinical practice - Davis, K. M.; Chang, C-H.; Lai, J-S.; Cella, D. (Quality of Life Research, 2002)

A feasibility study of on-the-fly item generation in adaptive testing (GRE Board Report No 98-12) - Bejar, I. I.; Lawless, R. R; Morley, M. E; Wagner, M. E.; Bennett, R. E.; Revuelta, J. (2002)

A further study on adjusting CAT item selection starting point for individual examinees - Fan, M.; Zhu. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
#FA02-01

Generating abstract reasoning items with cognitive theory - Embretson, S. E. (Item generation for test development, 2002)
(From the chapter) Developed and evaluated a generative system for abstract reasoning items based on cognitive theory. The cognitive design system approach was applied to generate matrix completion problems. Study 1 involved developing the cognitive theory with 191 college students who were administered Set I and Set II of the Advanced Progressive Matrices. Study 2 examined item generation by cognitive theory. Study 3 explored the psychometric properties and construct representation of abstract reasoning test items with 728 young adults. Five structurally equivalent forms of Abstract Reasoning Test (ART) items were prepared from the generated item bank and administered to the Ss. In Study 4, the nomothetic span of construct validity of the generated items was examined with 728 young adults who were administered ART items, and 217 young adults who were administered ART items and the Advanced Progressive Matrices. Results indicate the matrix completion items were effectively generated by the cognitive design system approach. (PsycINFO Database Record (c) 2005 APA )
Using Smart Source ParsingItem generation for test development. (pp. 219-250). Mahwah, NJ : Lawrence Erlbaum Associates, Publishers. xxxii, 412 pp
Keywords: Cognitive Processes; Measurement; Reasoning; Test Construction; Test Items; Test Validity; Theories

Historique et concepts propres au testing adaptatif [Adaptive testing: Historical accounts and concepts] - Blais, J. G. (Presented at the 69th Congress of the Acfas. Sherbrooke: Association canadienne française pour l’avancement des sciences (Acfas). [In French], 2002)

Hypergeometric family and item overlap rates in computerized adaptive testing - Chang, Hua-Hua; Zhang, J. (Psychometrika, 2002)
A computerized adaptive test (CAT) is usually administered to small groups of examinees at frequent time intervals. It is often the case that examinees who take the test earlier share information with examinees who will take the test later, thus increasing the risk that many items may become known. Item overlap rate for a group of examinees refers to the number of overlapping items encountered by these examinees divided by the test length. For a specific item pool, different item selection algorithms may yield different item overlap rates. An important issue in designing a good CAT item selection algorithm is to keep item overlap rate below a preset level. In doing so, it is important to investigate what the lowest rate could be for all possible item selection algorithms. In this paper we rigorously prove that if every item had an equal possibility to be selected from the pool in a fixed-length CAT, the number of overlapping item among any α randomly sampled examinees follows the hypergeometric distribution family for α ≥ 1. Thus, the expected values of the number of overlapping items among any randomly sampled α examinee can be calculated precisely. These values may serve as benchmarks in controlling item overlap rates for fixed-length adaptive tests. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Algorithms; Computer Assisted Testing; Taking; Test; Time On Task computerized adaptive testing

Identify the lower bounds for item sharing and item pooling in computerized adaptive testing - Chang, Hua-Hua; Zhang, J. (Paper presented at the annual meeting of the American Educational Research Association, 2002)

Impact of item quality and item bank size on the psychometric quality of computer-based credentialing exams - Hambleton, R. K. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)

Impact of selected factors on the psychometric quality of credentialing examinations administered with a sequential testlet design - Hambleton, R. K.; Jodoin, M.; Zenisky, A. L. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)

Impact of test design, item quality and item bank size on the psychometric properties of computer-based credentialing exams - Xing, D.; Hambleton, R. K. (Paper presented at the meeting of National Council on Measurement in Education, 2002)
PDF file, 500 K

The implications of the use of non-optimal items in a Computer Adaptive Testing (CAT) environment - Grodenchik, D. J. (Dissertation Abstracts International: Section B: the Sciences & Engineering, 2002)
This study describes the effects of manipulating item difficulty in a computer adaptive testing (CAT) environment. There are many potential benefits when using CATS as compared to traditional tests. These include increased security, shorter tests, and more precise measurement. According to IRT, the theory underlying CAT, as the computer continually recalculates ability, items that match that current estimate of ability are administered. Such items provide maximum information about examinees during the test. Herein, however, lies a potential problem. These optimal CAT items result in an examinee having only a 50% chance of a correct response. Some examinees may consider such items unduly challenging. Further, when test anxiety is a factor, it is possible that test scores may be negatively affected. This research was undertaken to determine the effects of administering easier CAT items on ability estimation and test length using computer simulations. Also considered was the administration of different numbers of initial items prior to the start of the adaptive portion of the test, using three different levels of measurement precision. Results indicate that regardless of the number of initial items administered, the level of precision employed, or the modifications made to item difficulty, the approximation of estimated ability to true ability is good in all cases. Additionally, the standard deviations of the ability estimates closely approximate the theoretical levels of precision used as stopping rules for the simulated CATs. Since optimal CAT items are not used, each item administered provides less information about examinees than optimal CAT items. This results in longer tests. Fortunately, using easier items that provide up to a 66.4% chance of a correct response results in tests that only modestly increase in length, across levels of precision. For larger standard errors, even easier items (up to a 73.5% chance of a correct response) result in only negligible to modest increases in test length. Examinees who find optimal CAT items difficult or examinees with test anxiety may find CATs that implement easier items enhance the already existing benefits of CAT. (PsycINFO Database Record (c) 2003 APA, all rights reserved).
Keywords: computerized adaptive testing

Incorporating the Sympson-Hetter exposure control method into the a-stratified method with content blocking - Yi, Q. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
PDF file, 387 K

Information technology and literacy assessment - Balajthy, E. (Reading and Writing Quarterly, 2002)
This column discusses information technology and literacy assessment in the past and present. The author also describes computer-based assessments today including the following topics: computer-scored testing, computer-administered formal assessment, Internet formal assessment, computerized adaptive tests, placement tests, informal assessment, electronic portfolios, information management, and Internet information dissemination. A model of the major present-day applications of information technologies in reading and literacy assessment is also included. (PsycINFO Database Record (c) 2005 APA )
Keywords: Computer Applications; Computer Assisted Testing; Information; Internet; Literacy; Models; Systems; Technology

Innovative item types for computerized testing - Parshall, C. G.; Davey, T.; Pashley, P. (2002)

An investigation of procedures for estimating error indexes in proficiency estimation in CAT - Shyu, C.-Y.; Fan, M.; Thompson, T,; Hsu, Y. (Paper presented at the annual meeting of the American Educational Research Association, 2002)

An item response model for characterizing test compromise - Segall, D. O. (Journal of Educational and Behavioral Statistics, 2002)
This article presents an item response model for characterizing test-compromise that enables the estimation of item-preview and score-gain distributions observed in on-demand high-stakes testing programs. Model parameters and posterior distributions are estimated by Markov Chain Monte Carlo (MCMC) procedures. Results of a simulation study suggest that when at least some of the items taken by a small sample of test takers are known to be secure (uncompromised), the procedure can provide useful summaries of test-compromise and its impact on test scores. The article includes discussions of operational use of the proposed procedure, possible model violations and extensions, and application to computerized adaptive testing.
References .American Educational Research Assn, US
Keywords: computerized adaptive testing

Item selection in computerized adaptive testing: Improving the a-stratified design with the Sympson-Hetter algorithm - Leung, C-K..; Chang, Hua-Hua; Hau, K-T. (Applied Psychological Measurement, 2002)

Item selection in computerized adaptive testing: Improving the a-stratified design with the Sympson-Hetter algorithm - Leung, C. K.; Chang, Hua-Hua; Hau, K. T. (Applied Psychological Measurement, 2002)

La simulation d’un test adaptatif basé sur le modèle de Rasch [Simulation of a Rasch-based adaptive test] - Raîche, G. (Mesure et évaluation en éducation., 2002)
(In French) {PDF file, 30 KB}

Le testing adaptatif [Adaptive testing] - Raîche, G. (2002)
(In French) {PDF file, 191 KB}

Mapping the Development of Pre-reading Skills with STAR Early Literacy - J. R. McBride; Tardrew, S.P. (Presentation to the Annual Meeting of the Society for the Scientific Study of Reading. Chicago., 2002)

Mathematical-programming approaches to test item pool design - Veldkamp, B. P.; van der Linden, W. J.; Ariel, A. (University of Twente, Faculty of Educational Science and Technology, 2002)
(From the chapter) This paper presents an approach to item pool design that has the potential to improve on the quality of current item pools in educational and psychological testing and hence to increase both measurement precision and validity. The approach consists of the application of mathematical programming techniques to calculate optimal blueprints for item pools. These blueprints can be used to guide the item-writing process. Three different types of design problems are discussed, namely for item pools for linear tests, item pools computerized adaptive testing (CAT), and systems of rotating item pools for CAT. The paper concludes with an empirical example of the problem of designing a system of rotating item pools for CAT.
Using Smart Source ParsingAdvances in psychology research, Vol. ( Hauppauge, NY : Nova Science Publishers, Inc, [URL:http://www.Novapublishers.com]. vi, 228 pp
Keywords: Adaptive Testing; Computer Assisted; Computer Programming; Educational Measurement; Item Response Theory; Mathematics; Psychometrics; Statistical Rotation computerized adaptive testing; Test Items; Testing

Measuring quality of life in chronic illness: the functional assessment of chronic illness therapy measurement system - Cella, D.; Nowinski, C. J. (Archives of Physical Medicine and Rehabilitation, 2002)
We focus on quality of life (QOL) measurement as applied to chronic illness. There are 2 major types of health-related quality of life (HRQOL) instruments-generic health status and targeted. Generic instruments offer the opportunity to compare results across patient and population cohorts, and some can provide normative or benchmark data from which to interpret results. Targeted instruments ask questions that focus more on the specific condition or treatment under study and, as a result, tend to be more responsive to clinically important changes than generic instruments. Each type of instrument has a place in the assessment of HRQOL in chronic illness, and consideration of the relative advantages and disadvantages of the 2 options best drives choice of instrument. The Functional Assessment of Chronic Illness Therapy (FACIT) system of HRQOL measurement is a hybrid of the 2 approaches. The FACIT system combines a core general measure with supplemental measures targeted toward specific diseases, conditions, or treatments. Thus, it capitalizes on the strengths of each type of measure. Recently, FACIT questionnaires were administered to a representative sample of the general population with results used to derive FACIT norms. These normative data can be used for benchmarking and to better understand changes in HRQOL that are often seen in clinical trials. Future directions in HRQOL assessment include test equating, item banking, and computerized adaptive testing.
0003-9993Journal Article
Keywords: *Chronic Disease; *Quality of Life; *Rehabilitation; Adult; Comparative Study; Health Status Indicators; Humans; Psychometrics; Questionnaires; Research Support, U.S. Gov't, P.H.S.; Sensitivity and Specificity

MIRTCAT [computer software] - Li, Y. H. (2002)

Modifications of the Sympson-Hetter method for item-exposure control in computerized adaptive testing - van der Linden, W. J. (2002)

Multidimensional adaptive testing for mental health problems in primary care - Gardner, W.; Kelleher, K. J.; Pajer, K. A. (Medical Care, 2002)
OBJECTIVES: Efficient and accurate instruments for assessing child psychopathology are increasingly important in clinical practice and research. For example, screening in primary care settings can identify children and adolescents with disorders that may otherwise go undetected. However, primary care offices are notorious for the brevity of visits and screening must not burden patients or staff with long questionnaires. One solution is to shorten assessment instruments, but dropping questions typically makes an instrument less accurate. An alternative is adaptive testing, in which a computer selects the items to be asked of a patient based on the patient's previous responses. This research used a simulation to test a child mental health screen based on this technology. RESEARCH DESIGN: Using half of a large sample of data, a computerized version was developed of the Pediatric Symptom Checklist (PSC), a parental-report psychosocial problem screen. With the unused data, a simulation was conducted to determine whether the Adaptive PSC can reproduce the results of the full PSC with greater efficiency. SUBJECTS: PSCs were completed by parents on 21,150 children seen in a national sample of primary care practices. RESULTS: Four latent psychosocial problem dimensions were identified through factor analysis: internalizing problems, externalizing problems, attention problems, and school problems. A simulated adaptive test measuring these traits asked an average of 11.6 questions per patient, and asked five or fewer questions for 49% of the sample. There was high agreement between the adaptive test and the full (35-item) PSC: only 1.3% of screening decisions were discordant (kappa = 0.93). This agreement was higher than that obtained using a comparable length (12-item) short-form PSC (3.2% of decisions discordant; kappa = 0.84). CONCLUSIONS: Multidimensional adaptive testing may be an accurate and efficient technology for screening for mental health problems in primary care settings.
Gardner, WilliamKelleher, Kelly JPajer, Kathleen AMCJ-177022/PHS HHS/MH30915/MH/NIMH NIH HHS/MH50629/MH/NIMH NIH HHS/Med Care. 2002 Sep;40(9):812-23.
Keywords: Adolescent; Child; Child Behavior Disorders/*diagnosis; Child Health Services/*organization & administration; Factor Analysis, Statistical; Female; Humans; Linear Models; Male; Mass Screening/*methods; Parents; Primary Health Care/*organization & administration

Multidimensional adaptive testing with constraints on test content - Veldkamp, B. P.; van der Linden, W. J. (Psychometrika, 2002)
The case of adaptive testing under a multidimensional response model with large numbers of constraints on the content of the test is addressed. The items in the test are selected using a shadow test approach. The 0–1 linear programming model that assembles the shadow tests maximizes posterior expected Kullback-Leibler information in the test. The procedure is illustrated for five different cases of multidimensionality. These cases differ in (a) the numbers of ability dimensions that are intentional or should be considered as ldquonuisance dimensionsrdquo and (b) whether the test should or should not display a simple structure with respect to the intentional ability dimensions.
Psychometric Society, US

Optimum number of strata in the a-stratified adaptive testing design - Wen, J.-B.; Chang, Hua-Hua; Hau, K-T. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 114 KB}

Outlier detection in high-stakes certification testing - Meijer, R. R. (Journal of Educational Measurement, 2002)
Discusses recent developments of person-fit analysis in computerized adaptive testing (CAT). Methods from statistical process control are presented that have been proposed to classify an item score pattern as fitting or misfitting the underlying item response theory model in CAT Most person-fit research in CAT is restricted to simulated data. In this study, empirical data from a certification test were used. Alternatives are discussed to generate norms so that bounds can be determined to classify an item score pattern as fitting or misfitting. Using bounds determined from a sample of a high-stakes certification test, the empirical analysis showed that different types of misfit can be distinguished Further applications using statistical process control methods to detect misfitting item score patterns are discussed. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; computerized adaptive testing; Educational Measurement; Goodness of Fit; Item Analysis (Statistical); Item Response Theory; person Fit; Statistical Estimation; Statistical Power; Test Scores

Practical considerations about expected a posteriori estimation in adaptive testing: Adaptive a prior, adaptive corrections for bias, adaptive integration interval - Raiche, G.; Blais, J. G. (Paper presented at the annual meeting of the International Objective Measurement Workshops-XI, 2002)
{PDF file, 100 KB}

A “rearrangement procedure” for administering adaptive tests with review options - Papanastasiou, E. C. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 410 KB}

Redeveloping the exposure control parameters of CAT items when a pool is modified - Chang, S-W.; Harris, D. J. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 1.113 MB}

Relative precision of ability estimation in polytomous CAT: A comparison under the generalized partial credit model and graded response model - Wang, S; Wang, T. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
#WA02-01 {PDF file, 735 KB}

Reliability and decision accuracy of linear parallel form and multi stage tests with realistic and ideal item pools - Jodoin, M. G. (Paper presented at the International Conference on Computer-Based Testing and the Internet, 2002)

The robustness of the unidimensional 3PL IRT model when applied to two-dimensional data in computerized adaptive testing - Zhao, J. C.; McMorris, R. F.; Pruzek, R. M.; Chen, R. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 1.356 MB}

Self-adapted testing: An overview - Wise, S. L.; Ponsoda, V.; Olea, J. (International Journal of Continuing Engineering Education and Life-Long Learning, 2002)

Some features of the estimated sampling distribution of the ability estimate in computerized adaptive testing according to two stopping rules - Raîche, G.; Blais, J. G. (Communication proposée au 11e Biannual International objective measurement workshop. New-Orleans : International Objective Measurement Workshops., 2002)

Some features of the sampling distribution of the ability estimate in computerized adaptive testing according to two stopping rules - Blais, J-G.; Raiche, G. (Paper presented at the annual meeting of the International Objective Measurement Workshops-XI, 2002)
{PDF file, 38 KB}

STAR Math 2 Computer-Adaptive Math Test and Database: Technical Manual - Renaissance-Learning-Inc. (2002)

Statistical indexes for monitoring item behavior under computer adaptive testing environment - Zhu, R.; Yu, F.; Liu, S. M. ((Original title: Detecting item misfit in computerized adaptive testing.) Paper presented at the annual meeting of the American Educational Research Association, 2002)
{PDF file, 2.287 MB}

Strategies for controlling item exposure in computerized adaptive testing with polytomously scored items - Davis, L. L. (2002)
{PDF file, 1.83 MB}

A strategy for controlling item exposure in multidimensional computerized adaptive testing - Lee, Y. H.; Ip, E.H.; Fuh, C.D. (2002)

A structure-based approach to psychological measurement: Matching measurement models to latent structure - Ruscio, John; Ruscio, Ayelet Meron (Assessment, 2002)
The present article sets forth the argument that psychological assessment should be based on a construct's latent structure. The authors differentiate dimensional (continuous) and taxonic (categorical) structures at the latent and manifest levels and describe the advantages of matching the assessment approach to the latent structure of a construct. A proper match will decrease measurement error, increase statistical power, clarify statistical relationships, and facilitate the location of an efficient cutting score when applicable. Thus, individuals will be placed along a continuum or assigned to classes more accurately. The authors briefly review the methods by which latent structure can be determined and outline a structure-based approach to assessment that builds on dimensional scaling models, such as item response theory, while incorporating classification methods as appropriate. Finally, the authors empirically demonstrate the utility of their approach and discuss its compatibility with traditional assessment methods and with computerized adaptive testing. (PsycINFO Database Record (c) 2005 APA ) (journal abstract)
Keywords: Adaptive Testing; Assessment; Classification (Cognitive Process); Computer Assisted; Item Response Theory; Psychological; Scaling (Testing); Statistical Analysis computerized adaptive testing; Taxonomies; Testing

Technology solutions for testing - Olson, A. (School Administrator, 2002)
Northwest Evaluation Association in Portland, Oregon, consults with state and local educators on assessment issues. Describes several approaches in place at school districts that are using some combination of computer-based tests to measure student growth. The computerized adaptive test adjusts items based on a student's answer in "real time." On-demand testing provides almost instant scoring. (MLF)

Test models for complex computer-based testing - Luecht, RM; Clauser, B. E. (2002)

A testlet assembly design for the uniform CPA examination - Luecht, RM; Brumfield, T.; Breithaupt, K (Paper presented at the Annual Meeting of the National Council on Measurement in Education., 2002)
PDF file 192 KB

To weight or not to weight – balancing influence of initial and later items in CAT - Chang, Hua-Hua; Ying, Z. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 252 KB}

Updated item parameter estimates using sparse CAT data - Smith, R. L.; Rizavi, S.; Paez, R.; Rotou, O. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2002)
{PDF file, 986 KB}

Using judgments of item difficulty to change answers on computerized adaptive vocabulary tests - Vispoel, W. P.; Clough, S. J.; Bleiler, T. (Paper presented at the annual meeting of the American Educational Research Association, 2002)
#VI02-01

Using testlet response theory to evaluate the equivalence of automatically generated multiple-choice items - Williamson, D. M.; Bejar, I. I. (Symposium conducted at the annual meeting of the National Council on Measurement in Eucation, 2002)

Utility of Learning Potential Computerised Adaptive Test (LPCAT) scores in predicting academic performance of bridging students: A comparison with other predictors - De Beer, M. (Paper presented at the 5th Annual Society for Industrial and Organisational Psychology Congress, 2002)

The work ahead: A psychometric infrastructure for computerized adaptive tests - F Drasgow (Computer-based tests: Building the foundation for future assessment, 2002)
(From the chapter) Considers the past and future of computerized adaptive tests and computer-based tests and looks at issues and challenges confronting a testing program as it implements and operates a computer-based test. Recommendations for testing programs from The National Council of Measurement in Education Ad Hoc Committee on Computerized Adaptive Test Disclosure are appended. (PsycINFO Database Record (c) 2005 APA )
Using Smart Source ParsingComputer-based testing: Building the foundation for future assessments. (pp. 1-35). Mahwah, NJ : Lawrence Erlbaum Associates, Publishers. xi, 326 pp
Keywords: Adaptive Testing; Computer Assisted Testing; Educational; Measurement; Psychometrics

Adaptation of a-stratified method in variable length computerized adaptive testing - Wen, J.-B.; Chang, Hua-Hua; Hau, K.-T. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
{PDF file, 384 KB}

Application of data mining to response data in a computerized adaptive test - Mendez, F. A. (2001)

Application of score information for CAT pool development and its connection with "likelihood test information - Krass, I. A. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
#{PDF file, 392 KB}

Assessment in the twenty-first century: A role of computerised adaptive testing in national curriculum subjects - Cowan, P.; Morrison, H. (Teacher Development, 2001)
With the investment of large sums of money in new technologies forschools and education authorities and the subsequent training of teachers to integrate Information and Communications Technology (ICT) into their teaching strategies, it is remarkable that the old outdated models of assessment still remain. This article highlights the current problems associated with pen-and paper-testing and offers suggestions for an innovative and new approach to assessment for the twenty-first century. Based on the principle of the 'wise examiner' a computerised adaptive testing system which measures pupils' ability against the levels of the United Kingdom National Curriculum has been developed for use in mathematics. Using constructed response items, pupils are administered a test tailored to their ability with a reliability index of 0.99. Since the software administers maximally informative questions matched to each pupil's current ability estimate, no two pupils will receive the same set of items in the same order therefore removing opportunities for plagarism and teaching to the test. All marking is automated and a journal recording the outcome of the test and highlighting the areas of difficulty for each pupil is available for printing by the teacher. The current prototype of the system can be used on a school's network however the authors envisage a day when Examination Boards or the Qualifications and Assessment Authority (QCA) will administer Government tests from a central server to all United Kingdom schools or testing centres. Results will be issued at the time of testing and opportunities for resits will become more widespr
Keywords: computerized adaptive testing

a-stratified CAT design with content-blocking - Yi, Q.; Chang, Hua-Hua (Paper presented at the Annual Meeting of the Psychometric Society, 2001)
{PDF file, 410 KB}

a-stratified computerized adaptive testing with unequal item exposure across strata - Deng, H.; Chang, Hua-Hua (Paper presented at the annual meeting of the American Educational Research Association, 2001)
#DE01-01

a-Stratified multistage computerized adaptive testing with b blocking - Chang, Hua-Hua; Qian, J.; Ying, Z. (Applied Psychological Measurement, 2001)
Chang & Ying’s (1999) computerized adaptive testing item-selection procedure stratifies the item bank according to a parameter values and requires b parameter values to be evenly distributed across all strata. Thus, a and b parameter values must be incorporated into how strata are formed. A refinement is proposed, based on Weiss’ (1973) stratification of items according to b values. Simulation studies using a retired item bank of a Graduate Record Examination test indicate that the new approach improved control of item exposure rates and reduced mean squared errors.

a-stratified multistage computerized adaptive testing with b blocking - Chang, Hua-Hua; Qian, J.; Yang, Z. (Applied Psychological Measurement, 2001)
Proposed a refinement, based on the stratification of items developed by D. Weiss (1973), of the computerized adaptive testing item selection procedure of H. Chang and Z. Ying (1999). Simulation studies using an item bank from the Graduate Record Examination show the benefits of the new procedure. (SLD)
Keywords: computerized adaptive testing

Can examinees use judgments of item difficulty to improve proficiency estimates on computerized adaptive vocabulary tests? - Vispoel, W. P.; Clough, S. J.; Bleiler, T. Hendrickson, A. B.; Ihrig, D. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
#VI01-01

CATSIB: A modified SIBTEST procedure to detect differential item functioning in computerized adaptive tests (Research report) - Nandakumar, R.; Roussos, L. (Law School Admission Council, 2001)

CB BULATS: Examining the reliability of a computer based test using test-retest method - Geranpayeh, A. (2001)
#GE01-01 14-16. {PDF file, 456 KB}

A comparative study of on line pretest item—Calibration/scaling methods in computerized adaptive testing - Ban, J. C.; Hanson, B. A.; Wang, T.; Yi, Q.; Harris, D. J. (Journal of Educational Measurement, 2001)
The purpose of this study was to compare and evaluate five on-line pretest item-calibration/scaling methods in computerized adaptive testing (CAT): marginal maximum likelihood estimate with one EM cycle (OEM), marginal maximum likelihood estimate with multiple EM cycles (MEM), Stocking's Method A, Stocking's Method B, and BILOG/Prior. The five methods were evaluated in terms ofitem-parameter recovery, using three different sample sizes (300, 1000 and 3000). The MEM method appeared to be the best choice among these, because it produced the smallest parameter-estimation errors for all sample size conditions. MEM and OEM are mathematically similar, although the OEM method produced larger errors. MEM also was preferable to OEM, unless the amount of timeinvolved in iterative computation is a concern. Stocking's Method B also worked very well, but it required anchor items that either would increase test lengths or require larger sample sizes depending on test administration design. Until more appropriate ways of handling sparse data are devised, the BILOG/Prior method may not be a reasonable choice for small sample sizes. Stocking's Method A hadthe largest weighted total error, as well as a theoretical weakness (i.e., treating estimated ability as true ability); thus, there appeared to be little reason to use it

Comparison of the SPRT and CMT procedures in computerized adaptive testing - Yi, Q.; Hanson, B.; Widiatmo, H.; Harris, D. J. (Paper presented at the annual meeting of the American Educational Research Association, 2001)

Computerized adaptive testing with equated number-correct scoring - van der Linden, W. J. (Applied Psychological Measurement, 2001)
A constrained computerized adaptive testing (CAT) algorithm is presented that can be used to equate CAT number-correct (NC) scores to a reference test. As a result, the CAT NC scores also are equated across administrations. The constraints are derived from van der Linden & Luecht’s (1998) set of conditions on item response functions that guarantees identical observed NC score distributions on two test forms. An item bank from the Law School Admission Test was used to compare the results of the algorithm with those for equipercentile observed-score equating, as well as the prediction of NC scores on a reference test using its test response function. The effects of the constraints on the statistical properties of the θ estimator in CAT were examined.
Sage Publications, US

Computerized adaptive testing with the generalized graded unfolding model - Roberts, J. S.; Lin, Y.; Laughlin, J. E. (Applied Psychological Measurement, 2001)
Examined the use of the generalized graded unfolding model (GGUM) in computerized adaptive testing. The objective was to minimize the number of items required to produce equiprecise estimates of person locations. Simulations based on real data about college student attitudes toward abortion and on data generated to fit the GGUM were used. It was found that as few as 7 or 8 items were needed to produce accurate and precise person estimates using an expected a posteriori procedure. The number items in the item bank (20, 40, or 60 items) and their distribution on the continuum (uniform locations or item clusters in moderately extreme locations) had only small effects on the accuracy and precision of the estimates. These results suggest that adaptive testing with the GGUM is a good method for achieving estimates with an approximately uniform level of precision using a small number of items. (PsycINFO Database Record (c) 2005 APA )
Keywords: Attitude Measurement; College Students computerized adaptive testing; Computer Assisted Testing; Item Response; Models; Statistical Estimation; Theory

Computerized-adaptive versus paper-and-pencil testing environments: An experimental analysis of examinee experience - Bringsjord, E. L. (2001)

Concerns with computerized adaptive oral proficiency assessment. A commentary on "Comparing examinee attitudes Toward computer-assisted and other oral proficient assessments": Response to the Norris Commentary - Norris, J. M.; Kenyon, D. M.; Malabonga, V. (Language Learning and Technology, 2001)
Responds to an article on computerized adaptive second language (L2) testing, expressing concerns about the appropriateness of such tests for informing language educators about the language skills of L2 learners and users and fulfilling the intended purposes and achieving the desired consequences of language test use.The authors of the original article respond. (Author/VWL)

CUSUM-based person-fit statistics for adaptive testing - van Krimpen-Stoop, E. M. L. A.; Meijer, R. R. (Journal of Educational and Behavioral Statistics, 2001)

Data sparseness and online pretest calibration/scaling methods in CAT - Ban, J; Hanson, B. A.; Yi, Q.; Harris, D. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
Also ACT Research Report 2002-1)

Deriving a stopping rule for sequential adaptive tests - Grabovsky, I.; Chang, Hua-Hua; Ying, Z. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
{PDF file, 111 KB}

Detection of misfitting item-score patterns in computerized adaptive testing - Stoop, E. M. L. A. (2001)
#ST01-01 V.

Development and evaluation of test assembly procedures for computerized adaptive testing - Robin, F. (2001)

Development of an adaptive multimedia program to collect patient health data - Sutherland, L. A.; Campbell, M.; Ornstein, K.; Wildemuth, B.; Lobach, D. (American Journal of Preventative Medicine, 2001)

The Development of STAR Early Literacy: A report of the School Renaissance Institute. - School-Renaissance-Institute (2001)

Developments in measurement of persons and items by means of item response models - Sijtsma, K. (Behaviormetrika, 2001)
This paper starts with a general introduction into measurement of hypothetical constructs typical of the social and behavioral sciences. After the stages ranging from theory through operationalization and item domain to preliminary test or questionnaire have been treated, the general assumptions of item response theory are discussed. The family of parametric item response models for dichotomous items is introduced and it is explained how parameters for respondents and items are estimated from the scores collected from a sample of respondents who took the test or questionnaire. Next, the family of nonparametric item response models is explained, followed by the 3 classes of item response models for polytomous item scores (e.g., rating scale scores). Then, to what degree the mean item score and the unweighted sum of item scores for persons are useful for measuring items and persons in the context of item response theory is discussed. Methods for fitting parametric and nonparametric models to data are briefly discussed. Finally, the main applications of item response models are discussed, which include equating and item banking, computerized and adaptive testing, research into differential item functioning, person fit research, and cognitive modeling. (PsycINFO Database Record (c) 2005 APA )
Keywords: Cognitive; Computer Assisted Testing; Item Response Theory; Models; Nonparametric Statistical Tests; Processes

Differences between self-adapted and computerized adaptive tests: A meta-analysis - Pitkin, A. K.; Vispoel, W. P. (Journal of Educational Measurement, 2001)
Self-adapted testing has been described as a variation of computerized adaptive testing that reduces test anxiety and thereby enhances test performance. The purpose of this study was to gain a better understanding of these proposed effects of self-adapted tests (SATs); meta-analysis procedures were used to estimate differences between SATs and computerized adaptive tests (CATs) in proficiency estimates and post-test anxiety levels across studies in which these two types of tests have been compared. After controlling for measurement error the results showed that SATs yielded proficiency estimates that were 0.12 standard deviation units higher and post-test anxiety levels that were 0.19 standard deviation units lower than those yielded by CATs. The authors speculate about possible reasons for these differences and discuss advantages and disadvantages of using SATs in operational settings. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Computer Assisted Testing; Scores computerized adaptive testing; Test; Test Anxiety

The effect of test and examinee characteristics on the occurrence of aberrant response patterns in a computerized adaptive test - Rizavi, S.; Swaminathan, H. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
#RI01-01

Effective use of simulated data in an on-line item calibration in practical situations of computerized adaptive testing - Samejima, F. (Paper presented at the annual meeting of the American Educational Research Association, 2001)

Effects of changes in the examinees’ ability distribution on the exposure control methods in CAT - Chang, S-W.; Twu, B.-Y. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
#CH01-02 {PDF file, 695 KB}

Efficient on-line item calibration using a nonparametric method adjusted to computerized adaptive testing - Samejima, F. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)

Evaluation of an MMPI-A short form: Implications for adaptive testing - Archer, R. P.; Tirrell, C. A.; Elkins, D. E. (Journal of Personality Assessment, 2001)
Reports some psychometric properties of an MMPI-Adolescent version (MMPI-A; J. N. Butcher et al, 1992) short form based on administration of the 1st 150 items of this test instrument. The authors report results for both the MMPI-A normative sample of 1,620 adolescents (aged 14-18 yrs) and a clinical sample of 565 adolescents (mean age 15.2 yrs) in a variety of treatment settings. The authors summarize results for the MMPI-A basic scales in terms of Pearson product-moment correlations generated between full administration and short-form administration formats and mean T score elevations for the basic scales generated by each approach. In this investigation, the authors also examine single-scale and 2-point congruences found for the MMPI-A basic clinical scales as derived from standard and short-form administrations. The authors present the relative strengths and weaknesses of the MMPI-A short form and discuss the findings in terms of implications for attempts to shorten the item pool through the use of computerized adaptive assessment approaches. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Mean; Minnesota Multiphasic Personality Inventory; Psychometrics; Statistical Correlation; Statistical Samples; Test Forms

An examination of conditioning variables used in computer adaptive testing for DIF analyses - Walker, C. M.; Beretvas, S. N; Ackerman, T. A. (Applied Measurement in Education, 2001)

An examination of item review on a CAT using the specific information item selection algorithm - Bowles, R; Pommerich, M (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
PDF file, 325 K

An examination of item review on a CAT using the specific information item selection algorithm - Bowles, R; Pommerich, M (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
PDF file, 325 KB}

An examination of item review on computer adaptive tests - Bowles, R (2001)

An examination of item selection rules by stratified CAT designs integrated with content balancing methods - Leung, C-K..; Chang, Hua-Hua; Hau, K-T. (Paper presented at the Annual Meeting of the American Educational Research Association, 2001)
{PDF file, 296 KB}

An examination of testlet scoring and item exposure constraints in the verbal reasoning section of the MCAT - Davis, L. L.; Dodd, B. G. (2001)

An examination of testlet scoring and item exposure constraints in the Verbal Reasoning section of the MCAT - Davis, L. L.; Dodd, B. G. (2001)
{PDF file, 653 KB}

An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales - Borman, W. C.; Buck, D. E.; Hanson, M. A.; Motowidlo, S. J.; Stark, S.; F Drasgow (Journal of Applied Psychology, 2001)
This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.
214803450021-9010Journal ArticleValidation Studies
Keywords: *Computer Simulation; *Employee Performance Appraisal; *Personnel Selection; Adult; Automatic Data Processing; Female; Human; Male; Reproducibility of Results; Sensitivity and Specificity; Support, U.S. Gov't, Non-P.H.S.; Task Performance and Analysis; Video Recording

The FastTEST Professional Testing System, Version 1.6 [Computer software] - Assessment-Systems-Corporation (2001)

Final answer? - Coyle, J. (American School Board Journal, 2001)
The Northwest Evaluation Association helped an Indiana school district develop a computerized adaptive testing system that was aligned with its curriculum and geared toward measuring individual student growth. Now the district can obtain such information from semester to semester and year to year, get immediate results, and test students on demand. (MLH)
Keywords: computerized adaptive testing

Impact of item location effects on ability estimation in CAT - Liu, M.; Zhu, R.; Guo, F. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
#LI01-01

Impact of scoring options for not reached items in CAT - Yi, Q.; Widiatmo, H.; Ban, J-C.; Harris, D. J. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
PDF file, 232 K

Impact of several computer-based testing variables on the psychometric properties of credentialing examinations (Laboratory of Psychometric and Evaluative Research Report No 393) - Xing, D.; Hambleton, R. K. (2001)

Impact of several computer-based testing variables on the psychometric properties of credentialing examinations - Xing, D.; Hambleton, R. K. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)

Implementing constrained CAT with shadow tests for large item pools - Veldkamp, B. P. (2001)

Implementing content constraints in a-stratified adaptive testing using a shadow test approach (Research Report 01-001) - Chang, Hua-Hua; van der Linden, W. J. (2001)

The influence of item characteristics and administration position on CAT Scores - Wang, L.; Gawlick, L. (Paper presented at the 33rd annual meeting of the Northeastern Educational Research Association, 2001)

Integrating stratification and information approaches for multiple constrained CAT - Leung, C.-I.; Chang, Hua-Hua; Hau, K-T. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)
{PDF file, 322 KB}

An investigation of procedures for estimating error indexes in proficiency estimation in a realistic second-order equitable CAT environment - Shyu, C.-Y.; Fan, M.; Thompson, T,; Hsu. (Paper presented at the annual meeting of the American Educational Research Association, 2001)

An investigation of the impact of items that exhibit mild DIF on ability estimation in CAT - Jennings, J. A.; Dodd, B. G.; Fitzpatrick, S. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)

Item and passage selection algorithm simulations for a computerized adaptive version of the verbal section of the Medical College Admission Test (MCAT) - Smith, R. W.; Plake, B. S.; De Ayala, R. J., (2001)

Item pool design for computerized adaptive tests - Reckase, M. D. (Invited small group session at the 6th Conference of the European Association of Psychological Assessment, 2001)

Item response theory applied to combinations of multiple-choice and constructed-response items--approximation methods for scale scores - Thissen, D.; Nelson, L. A.; Swygert, K. A. (Test scoring, 2001)
(From the chapter) The authors develop approximate methods that replace the scoring tables with weighted linear combinations of the component scores. Topics discussed include: a linear approximation for the extension to combinations of scores; the generalization of two or more scores; potential applications of linear approximations to item response theory in computerized adaptive tests; and evaluation of the pattern-of-summed-scores, and Gaussian approximation, estimates of proficiency. (PsycINFO Database Record (c) 2005 APA )
Using Smart Source ParsingTest scoring. (pp. 293-341). Mahwah, NJ : Lawrence Erlbaum Associates, Publishers. xii, 422 pp
Keywords: Adaptive Testing; Item Response Theory; Method); Multiple Choice (Testing; Scoring (Testing); Statistical Estimation; Statistical Weighting; Test Items; Test Scores

Item selection in computerized adaptive testing: Should more discriminating items be used first? - Hau, Kit-Tai; Chang, Hua-Hua (Journal of Educational Measurement, 2001)
During computerized adaptive testing (CAT), items are selected continuously according to the test-taker's estimated ability. Test security has become a problem because high-discrimination items are more likely to be selected and become overexposed. So, there seems to be a tradeoff between high efficiency in ability estimations and balanced usage of items. This series of four studies addressed the dilemma by focusing on the notion of whether more or less discriminating items should be used first in CAT. The first study demonstrated that the common maximum information method with J. B. Sympson and R. D. Hetter (1985) control resulted in the use of more discriminating items first. The remaining studies showed that using items in the reverse order, as described in H. Chang and Z. Yings (1999) stratified method had potential advantages: (a) a more balanced item usage and (b) a relatively stable resultant item pool structure with easy and inexpensive management. This stratified method may have ability-estimation efficiency better than or close to that of other methods. It is argued that the judicious selection of items, as in the stratified method, is a more active control of item exposure. (PsycINFO Database Record (c) 2005 APA )
Keywords: ability; Adaptive Testing; Computer Assisted Testing; Estimation; Statistical; Test Items computerized adaptive testing

On maximizing item information and matching difficulty with ability - Bickel, P.; Buyske, S.; Chang, Hua-Hua; Ying, Z. (Psychometrika, 2001)

Measurement efficiency of multidimensional computerized adaptive testing - Wang, W-C.; Chen, B.-H. (Paper presented at the annual meeting of the American Psychological Association, 2001)

Measuring test compromise in high-stakes computerized adaptive testing: A Bayesian Strategy for surrogate test-taker detection - Segall, D. O. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
{PDF file, 275 KB}

A method for building a realistic model of test taker behavior for computerized adaptive testing (Research Report 01-22) - Stocking, M. L.; Steffen, M.; Eignor, D. R. (2001)

Methods to test invariant ability across subgroups of items in CAT - Meijer, R. R. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)

A minimax procedure in the context of sequential testing problems in psychodiagnostics - Vos, H. J. (British Journal of Mathematical and Statistical Psychology, 2001)
#VO01139 Vos, H J (2001) A minimax procedure in the context of sequential testing problems

Modeling variability in item parameters in CAT - Glas, C. A. W.; van der Linden, W. J. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)

Monitoring items for changes in performance in computerized adaptive tests - Smith, R. L.; Wang, M.M.; Wingersky, M.; Zhao, C. (Paper presented at the annual conference of the National Council on Measurement in Education, 2001)

A monte carlo study of the feasibility of on-the-fly assessment - Revuelta, J.; Bejar, I. I.; Stocking, M. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)

Multidimensional adaptive testing using the weighted likelihood estimation - Tseng, F-L. (Dissertation Abstracts International Section A: Humanities & Social Sciences, 2001)
This study extended Warm's (1989) weighted likelihood estimation (WLE) to a multidimensional computerized adaptive test (MCAT) setting. WLE was compared with the maximum likelihood estimation (MLE), expected a posteriori (EAP), and maximum a posteriori (MAP) using a three-dimensional 3PL IRT model under a variety of computerized adaptive testing conditions. The dependent variables included bias, standard error of ability estimates (SE), square root of mean square error (RMSE), and test information. The independent variables were ability estimation methods, intercorrelation levels between dimensions, multidimensional structures, and ability combinations. Simulation results were presented in terms of descriptive statistics, such as figures and tables. In addition, inferential procedures were used to analyze bias by conceptualizing this Monte Carlo study as a statistical sampling experiment. The results of this study indicate that WLE and the other three estimation methods yield significantly more accurate ability estimates under an approximate simple test structure with one dominant dimension and several secondary dimensions. All four estimation methods, especially WLE, yield very large SEs when a three equally dominant multidimensional structure was employed. Consistent with previous findings based on unidimensional IRT model, MLE and WLE are less biased in the extreme of the ability scale; MLE and WLE yield larger SEs than the Bayesian methods; test information-based SEs underestimate actual SEs for both MLE and WLE in MCAT situations, especially at shorter test lengths; WLE reduced the bias of MLE under the approximate simple structure; test information-based SEs underestimates the actual SEs of MLE and WLE estimators in the MCAT conditions, similar to the findings of Warm (1989) in the unidimensional case. The results from the MCAT simulations did show some advantages of WLE in reducing the bias of MLE under the approximate simple structure with a fixed test length of 50 items, which was consistent with the previous research findings based on different unidimensional models. It is clear from the current results that all four methods perform very poorly when the multidimensional structures with multiple dominant factors were employed. More research efforts are urged to investigate systematically how different multidimensional structures affect the accuracy and reliability of ability estimation. Based on the simulated results in this study, there is no significant effect found on the ability estimation from the intercorrelation between dimensions. (PsycINFO Database Record (c) 2003 APA, all rights reserved).
Keywords: computerized adaptive testing

Multidimensional adaptive testing using weighted likelihood estimation: A comparison of estimation methods - Tseng, F.-E.; Hsu, T.-C. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)
{PDF file, 988 KB}

Multidimensional IRT-based adaptive sequential mastery testing - Vos, H. J.; Glas, C. E. W. (Paper presented at the Annual Meeting of the National Council on Measurement in Education, 2001)

NCLEX-RN performance: predicting success on the computerized examination - Beeman, P. B.; Waterhouse, J. K. (Journal of Professional Nursing, 2001)
Since the adoption of the Computerized Adaptive Testing (CAT) format of the National Certification Licensure Examination for Registered Nurses (NCLEX-RN), no studies have been reported in the literature on predictors of successful performance by baccalaureate nursing graduates on the licensure examination. In this study, a discriminant analysis was used to identify which of 21 variables can be significant predictors of success on the CAT NCLEX-RN. The convenience sample consisted of 289 individuals who graduated from a baccalaureate nursing program between 1995 and 1998. Seven significant predictor variables were identified. The total number of C+ or lower grades earned in nursing theory courses was the best predictor, followed by grades in several individual nursing courses. More than 93 per cent of graduates were correctly classified. Ninety-four per cent of NCLEX "passes" were correctly classified, as were 92 per cent of NCLEX failures. This degree of accuracy in classifying CAT NCLEX-RN failures represents a marked improvement over results reported in previous studies of licensure examinations, and suggests the discriminant function will be helpful in identifying future students in danger of failure. J Prof Nurs 17:158-165, 2001.
8755-7223Journal Article
Keywords: *Education, Nursing, Baccalaureate; *Educational Measurement; *Licensure; Adult; Female; Humans; Male; Predictive Value of Tests; Software

Nearest neighbors, simple strata, and probabilistic parameters: An empirical comparison of methods for item exposure control in CATs - Parshall, C. G.; Kromrey, J. D.; Harmes, J. C.; Sentovich, C. (Paper presented at the annual meeting of the National Council on Measurement in Education, 2001)

A new approach to simulation studies in computerized adaptive testing - Chen, S-Y. (Paper presented at the annual meeting of the American Educational Research Association, 2001)
{PDF file, 251 KB}

A new computer algorithm for simultaneous test construction of two-stage and multistage testing - Wu, I. L. (Journal of Educational and Behavioral Statistics, 2001)

Nouveaux développements dans le domaine du testing informatisé [New developments in the area of computerized testing] - Meijer, R. R.; Grégoire, J. (Psychologie Française, 2001)
L'usage de l'évaluation assistée par ordinateur s'est fortement développé depuis la première formulation de ses principes de base dans les années soixante et soixante-dix. Cet article offre une introduction aux derniers développements dans le domaine de l'évaluation assistée par ordinateur, en particulier celui du testing adaptative informatisée (TAI). L'estimation de l'aptitude, la sélection des items et le développement d'une base d'items dans le cas du TAI sont discutés. De plus, des exemples d'utilisations innovantes de l'ordinateur dans des systèmes intégrés de testing et de testing via Internet sont présentés. L'article se termine par quelques illustrations de nouvelles applications du testing informatisé et des suggestions pour des recherches futures.Discusses the latest developments in computerized psychological assessment, with emphasis on computerized adaptive testing (CAT). Ability estimation, item selection, and item pool development in CAT are described. Examples of some innovative approaches to CAT are presented. (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Computer Applications; Computer Assisted; Diagnosis; Psychological Assessment computerized adaptive testing

On-line Calibration Using PARSCALE Item Specific Prior Method: Changing Test Population and Sample Size - Guo, F.; Stone, E.; Cruz, D. (Paper presented at National Council on Measurement in Education Annual Meeting, 2001)

Online item parameter recalibration: Application of missing data treatments to overcome the effects of sparse data conditions in a computerized adaptive version of the MCAT - Harmes, J. C.; Kromrey, J. D.; Parshall, C. G. (2001)
{PDF file, 406 KB}

Outlier measures and norming methods for computerized adaptive tests - Bradlow, E. T.; Weiss, R. E. (Journal of Educational and Behavioral Statistics, 2001)
Notes that the problem of identifying outliers has 2 important aspects: the choice of outlier measures and the method to assess the degree of outlyingness (norming) of those measures. Several classes of measures for identifying outliers in Computerized Adaptive Tests (CATs) are introduced. Some of these measures are constructed to take advantage of CATs' sequential choice of items; other measures are taken directly from paper and pencil (P&P) tests and are used for baseline comparisons. Assessing the degree of outlyingness of CAT responses, however, can not be applied directly from P&P tests because stopping rules associated with CATs yield examinee responses of varying lengths. Standard outlier measures are highly correlated with the varying lengths which makes comparison across examinees impossible. Therefore, 4 methods are presented and compared which map outlier statistics to a familiar probability scale (a p value). The methods are explored in the context of CAT data from a 1995 Nationally Administered Computerized Examination (NACE). (PsycINFO Database Record (c) 2005 APA )
Keywords: Adaptive Testing; Computer Assisted Testing; Statistical Analysis; Test Norms

Overexposure and underexposure of items in computerized adaptive testing (Measurement and Research Department Reports 2001-1) - Theo Eggen (2001)
{PDF file, 276 KB}

Pasado, presente y futuro de los test adaptativos informatizados: Entrevista con Isaac I. Béjar [Past, present and future of computerized adaptive testing: Interview with Isaac I. Béjar] - Tejada, R.; Antonio, J. (Psicothema, 2001)
En este artículo se presenta el resultado de una entrevista con Isaac I. Bejar. El Dr. Bejar es actualmente Investigador Científico Principal y Director del Centro para el Diseño de Evaluación y Sistemas de Puntuación perteneciente a la División de Investigación del Servicio de Medición Educativa (Educa - tional Testing Service, Princeton, NJ, EE.UU.). El objetivo de esta entrevista fue conversar sobre el pasado, presente y futuro de los Tests Adaptativos Informatizados. En la entrevista se recogen los inicios de los Tests Adaptativos y de los Tests Adaptativos Informatizados y últimos avances que se desarrollan en el Educational Testing Service sobre este tipo de tests (modelos generativos, isomorfos, puntuación automática de ítems de ensayo…). Se finaliza con la visión de futuro de los Tests Adaptativos Informatizados y su utilización en España.Past, present and future of Computerized Adaptive Testing: Interview with Isaac I. Bejar. In this paper the results of an interview with Isaac I. Bejar are presented. Dr. Bejar is currently Principal Research Scientist and Director of Center for Assessment Design and Scoring, in Research Division at Educational Testing Service (Princeton, NJ, U.S.A.). The aim of this interview was to review the past, present and future of the Computerized Adaptive Tests. The beginnings of the Adaptive Tests and Computerized Adaptive Tests, and the latest advances developed at the Educational Testing Service (generative response models, isomorphs, automated scoring…) are reviewed. The future of Computerized Adaptive Tests is analyzed, and its utilization in Spain commented.
Keywords: computerized adaptive testing

Polytomous modeling of cognitive errors in computer adaptive testing - Wang, L.-S.; Chun-Shan Li. (Journal of Applied Measurement, 2001)
Used Monte Carlo simulation to compare the relative measurement efficiency of polytomous modeling and dichotomous modeling under different scoring schemes and termination criteria. Results suggest that polytomous computerized adaptive testing (CAT) yields marginal gains over dichotomous CAT when termination criteria are more stringent. Discusses conditions under which polytomous CAT cannot prevent the nonuniform gain in test information. (SLD)

Pour une évaluation sur mesure des étudiants : défis et enjeux du testing adaptatif - Raîche, G. (Commnication présentée à l’intérieur de la 23e session d’études de l’Association pour le dévelopement de la mesure et de l’évaluation en éducation, 2001)
Québec: ADMÉÉ.

Pour une évaluation sur mesure pour chaque étudiant : défis et enjeux du testing adaptatif par ordinateur en éducation [Tailored testing for each student : Principles and stakes of computerized adaptive testing in education] - Raîche, G, Blais, J.G.; Boiteau, N. (Presented at the 23th Study Session of the ADMÉÉ. Québec: Association pour le développement de la mesure et de l’évaluation en éducation (ADMÉÉ)., 2001)

Practical issues in setting standards on computerized adaptive tests - Sireci, S. G.; Clauser, B. E. (Setting performance standards: Concepts, methods, and perspectives, 2001)
(From the chapter) Examples of setting standards on computerized adaptive tests (CATs) are hard to find. Some examples of CATs involving performance standards include the registered nurse exam and the Novell systems engineer exam. Although CATs do not require separate standard setting-methods, there are special issues to be addressed by test specialist who set performance standards on CATs. Setting standards on a CAT will typical require modifications on the procedures used with more traditional, fixed-form, paper-and -pencil examinations. The purpose of this chapter is to illustrate why CATs pose special challenges to the standard setter. (PsycINFO Database Record (c) 2005 APA )
Using Smart Source ParsingSetting performance standards: Concepts, methods, and perspectives. (pp. 355-369). Mahwah, NJ : Lawrence Erlbaum Associates, Publishers. xiii, 510 pp
Keywords: Adaptive Testing; Computer Assisted Testing; Performance Tests; Testing Methods