02111nas a2200193 4500008003900000245007100039210006900110300000900179490000700188520153100195653000701726653003201733653001701765653002301782100001501805700001501820700001901835856006301854 2020 d00aItem Calibration Methods With Multiple Subscale Multistage Testing0 aItem Calibration Methods With Multiple Subscale Multistage Testi a3-280 v573 aAbstract Many large-scale educational surveys have moved from linear form design to multistage testing (MST) design. One advantage of MST is that it can provide more accurate latent trait (θ) estimates using fewer items than required by linear tests. However, MST generates incomplete response data by design; hence, questions remain as to how to calibrate items using the incomplete data from MST design. Further complication arises when there are multiple correlated subscales per test, and when items from different subscales need to be calibrated according to their respective score reporting metric. The current calibration-per-subscale method produced biased item parameters, and there is no available method for resolving the challenge. Deriving from the missing data principle, we showed when calibrating all items together the Rubin's ignorability assumption is satisfied such that the traditional single-group calibration is sufficient. When calibrating items per subscale, we proposed a simple modification to the current calibration-per-subscale method that helps reinstate the missing-at-random assumption and therefore corrects for the estimation bias that is otherwise existent. Three mainstream calibration methods are discussed in the context of MST, they are the marginal maximum likelihood estimation, the expectation maximization method, and the fixed parameter calibration. An extensive simulation study is conducted and a real data example from NAEP is analyzed to provide convincing empirical evidence.10aEM10amarginal maximum likelihood10amissing data10amultistage testing1 aWang, Chun1 aChen, Ping1 aJiang, Shengyu uhttps://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.1224101835nas a2200157 4500008003900000022001400039245010200053210006900155300001400224490000700238520133800245100001501583700001901598700001901617856004101636 2014 d a1745-398400aAn Enhanced Approach to Combine Item Response Theory With Cognitive Diagnosis in Adaptive Testing0 aEnhanced Approach to Combine Item Response Theory With Cognitive a358–3800 v513 a

Computerized adaptive testing offers the possibility of gaining information on both the overall ability and cognitive profile in a single assessment administration. Some algorithms aiming for these dual purposes have been proposed, including the shadow test approach, the dual information method (DIM), and the constraint weighted method. The current study proposed two new methods, aggregate ranked information index (ARI) and aggregate standardized information index (ASI), which appropriately addressed the noncompatibility issue inherent in the original DIM method. More flexible weighting schemes that put different emphasis on information about general ability (i.e., θ in item response theory) and information about cognitive profile (i.e., α in cognitive diagnostic modeling) were also explored. Two simulation studies were carried out to investigate the effectiveness of the new methods and weighting schemes. Results showed that the new methods with the flexible weighting schemes could produce more accurate estimation of both overall ability and cognitive profile than the original DIM. Among them, the ASI with both empirical and theoretical weights is recommended, and attribute-level weighting scheme is preferred if some attributes are considered more important from a substantive perspective.

1 aWang, Chun1 aZheng, Chanjin1 aChang, Hua-Hua uhttp://dx.doi.org/10.1111/jedm.1205701984nas a2200121 4500008003900000245008900039210006900128300001200197490000700209520157400216100001501790856005701805 2014 d00aImproving Measurement Precision of Hierarchical Latent Traits Using Adaptive Testing0 aImproving Measurement Precision of Hierarchical Latent Traits Us a452-4770 v393 a

Many latent traits in social sciences display a hierarchical structure, such as intelligence, cognitive ability, or personality. Usually a second-order factor is linearly related to a group of first-order factors (also called domain abilities in cognitive ability measures), and the first-order factors directly govern the actual item responses. Because only a subtest of items is used to measure each domain, the lack of sufficient reliability becomes the primary impediment for generating and reporting domain abilities. In recent years, several item response theory (IRT) models have been proposed to account for hierarchical factor structures, and these models are also shown to alleviate the low reliability issue by using in-test collateral information to improve measurement precision. This article advocates using adaptive item selection together with a higher order IRT model to further increase the reliability of hierarchical latent trait estimation. Two item selection algorithms are proposed—the constrained D-optimal method and the sequencing domain method. Both are shown to yield improved measurement precision as compared to the unidimensional item selection (by treating each dimension separately). The improvement is more prominent when the test length is short and when the correlation between dimensions is high (e.g., higher than .64). Moreover, two reliability indices for hierarchical latent traits are discussed and their use for quantifying the reliability of hierarchical traits measured by adaptive testing is demonstrated.

1 aWang, Chun uhttp://jeb.sagepub.com/cgi/content/abstract/39/6/45202198nas a2200145 4500008003900000245007900039210006900118300001100187490000700198520173800205100001501943700001901958700002301977856005202000 2013 d00aDeriving Stopping Rules for Multidimensional Computerized Adaptive Testing0 aDeriving Stopping Rules for Multidimensional Computerized Adapti a99-1220 v373 a

Multidimensional computerized adaptive testing (MCAT) is able to provide a vector of ability estimates for each examinee, which could be used to provide a more informative profile of an examinee’s performance. The current literature on MCAT focuses on the fixed-length tests, which can generate less accurate results for those examinees whose abilities are quite different from the average difficulty level of the item bank when there are only a limited number of items in the item bank. Therefore, instead of stopping the test with a predetermined fixed test length, the authors use a more informative stopping criterion that is directly related to measurement accuracy. Specifically, this research derives four stopping rules that either quantify the measurement precision of the ability vector (i.e., minimum determinant rule [D-rule], minimum eigenvalue rule [E-rule], and maximum trace rule [T-rule]) or quantify the amount of available information carried by each item (i.e., maximum Kullback–Leibler divergence rule [K-rule]). The simulation results showed that all four stopping rules successfully terminated the test when the mean squared error of ability estimation is within a desired range, regardless of examinees’ true abilities. It was found that when using the D-, E-, or T-rule, examinees with extreme abilities tended to have tests that were twice as long as the tests received by examinees with moderate abilities. However, the test length difference with K-rule is not very dramatic, indicating that K-rule may not be very sensitive to measurement precision. In all cases, the cutoff value for each stopping rule needs to be adjusted on a case-by-case basis to find an optimal solution.

1 aWang, Chun1 aChang, Hua-Hua1 aBoughton, Keith, A uhttp://apm.sagepub.com/content/37/2/99.abstract01865nas a2200121 4500008003900000245012200039210006900161300001400230490000700244520142300251100001501674856005401689 2013 d00aMutual Information Item Selection Method in Cognitive Diagnostic Computerized Adaptive Testing With Short Test Length0 aMutual Information Item Selection Method in Cognitive Diagnostic a1017-10350 v733 a

Cognitive diagnostic computerized adaptive testing (CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. Cognitive diagnosis models aim at classifying examinees into the correct mastery profile group so as to pinpoint the strengths and weakness of each examinee whereas CAT algorithms choose items to determine those strengths and weakness as efficiently as possible. Most of the existing CD-CAT item selection algorithms are evaluated when test length is relatively long whereas several applications of CD-CAT, such as in interim assessment, require an item selection algorithm that is able to accurately recover examinees’ mastery profile with short test length. In this article, we introduce the mutual information item selection method in the context of CD-CAT and then provide a computationally easier formula to make the method more amenable in real time. Mutual information is then evaluated against common item selection methods, such as Kullback–Leibler information, posterior weighted Kullback–Leibler information, and Shannon entropy. Based on our simulations, mutual information consistently results in nearly the highest attribute and pattern recovery rate in more than half of the conditions. We conclude by discussing how the number of attributes, Q-matrix structure, correlations among the attributes, and item quality affect estimation accuracy.

1 aWang, Chun uhttp://epm.sagepub.com/content/73/6/1017.abstract01630nas a2200157 4500008003900000245010100039210006900140300001200209490000700221520111200228100001501340700001601355700001901371700002501390856005701415 2013 d00aA Semiparametric Model for Jointly Analyzing Response Times and Accuracy in Computerized Testing0 aSemiparametric Model for Jointly Analyzing Response Times and Ac a381-4170 v383 a

The item response times (RTs) collected from computerized testing represent an underutilized type of information about items and examinees. In addition to knowing the examinees’ responses to each item, we can investigate the amount of time examinees spend on each item. Current models for RTs mainly focus on parametric models, which have the advantage of conciseness, but may suffer from reduced flexibility to fit real data. We propose a semiparametric approach, specifically, the Cox proportional hazards model with a latent speed covariate to model the RTs, embedded within the hierarchical framework proposed by van der Linden to model the RTs and response accuracy simultaneously. This semiparametric approach combines the flexibility of nonparametric modeling and the brevity and interpretability of the parametric modeling. A Markov chain Monte Carlo method for parameter estimation is given and may be used with sparse data obtained by computerized adaptive testing. Both simulation studies and real data analysis are carried out to demonstrate the applicability of the new model.

1 aWang, Chun1 aFan, Zhewen1 aChang, Hua-Hua1 aDouglas, Jeffrey, A. uhttp://jeb.sagepub.com/cgi/content/abstract/38/4/38101369nas a2200157 4500008003900000022001400039245010400053210006900157300001400226490000700240520085700247100001501104700001901119700001801138856005501156 2011 d a1745-398400aRestrictive Stochastic Item Selection Methods in Cognitive Diagnostic Computerized Adaptive Testing0 aRestrictive Stochastic Item Selection Methods in Cognitive Diagn a255–2730 v483 a

This paper proposes two new item selection methods for cognitive diagnostic computerized adaptive testing: the restrictive progressive method and the restrictive threshold method. They are built upon the posterior weighted Kullback-Leibler (KL) information index but include additional stochastic components either in the item selection index or in the item selection procedure. Simulation studies show that both methods are successful at simultaneously suppressing overexposed items and increasing the usage of underexposed items. Compared to item selection based upon (1) pure KL information and (2) the Sympson-Hetter method, the two new methods strike a better balance between item exposure control and measurement accuracy. The two new methods are also compared with Barrada et al.'s (2008) progressive method and proportional method.

1 aWang, Chun1 aChang, Hua-Hua1 aHuebner, Alan uhttp://dx.doi.org/10.1111/j.1745-3984.2011.00145.x