%0 Journal Article %J Educational and Psychological Measurement %D 2009 %T Direct and Inverse Problems of Item Pool Design for Computerized Adaptive Testing %A Belov, Dmitry I. %A Armstrong, Ronald D. %X

The recent literature on computerized adaptive testing (CAT) has developed methods for creating CAT item pools from a large master pool. Each CAT pool is designed as a set of nonoverlapping forms reflecting the skill levels of an assumed population of test takers. This article presents a Monte Carlo method to obtain these CAT pools and discusses its advantages over existing methods. Also, a new problem is considered that finds a population ability density function best matching the master pool. An analysis of the solution to this new problem provides testing organizations with effective guidance for maintaining their master pools. Computer experiments with a pool of Law School Admission Test items and its assembly constraints are presented.

%B Educational and Psychological Measurement %V 69 %P 533-547 %U http://epm.sagepub.com/content/69/4/533.abstract %R 10.1177/0013164409332224 %0 Journal Article %J Applied Psychological Measurement %D 2008 %T A Monte Carlo Approach for Adaptive Testing With Content Constraints %A Belov, Dmitry I. %A Armstrong, Ronald D. %A Weissman, Alexander %X

This article presents a new algorithm for computerized adaptive testing (CAT) when content constraints are present. The algorithm is based on shadow CAT methodology to meet content constraints but applies Monte Carlo methods and provides the following advantages over shadow CAT: (a) lower maximum item exposure rates, (b) higher utilization of the item pool, and (c) more robust ability estimates. Computer simulations with Law School Admission Test items demonstrated that the new algorithm (a) produces similar ability estimates as shadow CAT but with half the maximum item exposure rate and 100% pool utilization and (b) produces more robust estimates when a high- (or low-) ability examinee performs poorly (or well) at the beginning of the test.

%B Applied Psychological Measurement %V 32 %P 431-446 %U http://apm.sagepub.com/content/32/6/431.abstract %R 10.1177/0146621607309081 %0 Journal Article %J Applied Psychological Measurement %D 2008 %T A Monte Carlo Approach to the Design, Assembly, and Evaluation of Multistage Adaptive Tests %A Belov, Dmitry I. %A Armstrong, Ronald D. %X

This article presents an application of Monte Carlo methods for developing and assembling multistage adaptive tests (MSTs). A major advantage of the Monte Carlo assembly over other approaches (e.g., integer programming or enumerative heuristics) is that it provides a uniform sampling from all MSTs (or MST paths) available from a given item pool. The uniform sampling allows a statistically valid analysis for MST design and evaluation. Given an item pool, MST model, and content constraints for test assembly, three problems are addressed in this study. They are (a) the construction of item response theory (IRT) targets for each MST path, (b) the assembly of an MST such that each path satisfies content constraints and IRT constraints, and (c) an analysis of the pool and constraints to increase the number of nonoverlapping MSTs that can be assembled from the pool. The primary intent is to produce reliable measurements and enhance pool utilization.

%B Applied Psychological Measurement %V 32 %P 119-137 %U http://apm.sagepub.com/content/32/2/119.abstract %R 10.1177/0146621606297308 %0 Journal Article %J Applied Psychological Measurement %D 2005 %T Monte Carlo Test Assembly for Item Pool Analysis and Extension %A Belov, Dmitry I. %A Armstrong, Ronald D. %X

A new test assembly algorithm based on a Monte Carlo random search is presented in this article. A major advantage of the Monte Carlo test assembly over other approaches (integer programming or enumerative heuristics) is that it performs a uniform sampling from the item pool, which provides every feasible item combination (test) with an equal chance of being built during an assembly. This allows the authors to address the following issues of pool analysis and extension: compare the strengths and weaknesses of different pools, identify the most restrictive constraint(s) for test assembly, and identify properties of the items that should be added to a pool to achieve greater usability of the pool. Computer experiments with operational pools are given.

%B Applied Psychological Measurement %V 29 %P 239-261 %U http://apm.sagepub.com/content/29/4/239.abstract %R 10.1177/0146621605275413 %0 Journal Article %J Applied Psychological Measurement %D 2004 %T Computerized Adaptive Testing With Multiple-Form Structures %A Armstrong, Ronald D. %A Jones, Douglas H. %A Koppel, Nicole B. %A Pashley, Peter J. %X

A multiple-form structure (MFS) is an orderedcollection or network of testlets (i.e., sets of items).An examinee’s progression through the networkof testlets is dictated by the correctness of anexaminee’s answers, thereby adapting the test tohis or her trait level. The collection of pathsthrough the network yields the set of all possibletest forms, allowing test specialists the opportunityto review them before they are administered. Also,limiting the exposure of an individual MFS to aspecific period of time can enhance test security.This article provides an overview of methods thathave been developed to generate parallel MFSs.The approach is applied to the assembly of anexperimental computerized Law School Admission Test (LSAT).

%B Applied Psychological Measurement %V 28 %P 147-164 %U http://apm.sagepub.com/content/28/3/147.abstract %R 10.1177/0146621604263652