05519nas a2200193 4500008004100000245009000041210006900131260005500200520480200255653002305057653001905080653002105099653002605120100002205146700002005168700001605188700001905204856010205223 2017 eng d00aAdaptive Item and Feedback Selection in Personalized Learning with a Network Approach0 aAdaptive Item and Feedback Selection in Personalized Learning wi aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Personalized learning is a term used to describe educational systems that adapt student-specific curriculum sequencing, pacing, and presentation based on their unique backgrounds, knowledge, preferences, interests, and learning goals. (Chen, 2008; Netcoh, 2016). The technological approach to personalized learning provides data-driven models to incorporate these adaptations automatically. Examples of applications include online learning systems, educational games, and revision-aid systems. In this study we introduce Bayesian networks as a methodology to implement an adaptive framework within a personalized learning environment. Existing ideas from Computerized Adaptive Testing (CAT) with Item Response Theory (IRT), where choices about content provision are based on maximizing information, are related to the goals of personalized learning environments. Personalized learning entails other goals besides efficient ability estimation by maximizing information, such as an adaptive configuration of preferences and feedback to the student. These considerations will be discussed and their application in networks will be illustrated.

Adaptivity in Personalized Learning.In standard CAT’s there is a focus on selecting items that provide maximum information about the ability of an individual at a certain point in time (Van der Linden & Glas, 2000). When learning is the main goal of testing, alternative adaptive item selection methods were explored by Eggen (2012). The adaptive choices made in personalized learning applications require additional adaptivity with respect to the following aspects; the moment of feedback, the kind of feedback, and the possibility for students to actively influence the learning process.

Bayesian Networks and Personalized Learning.Personalized learning aims at constructing a framework to incorporate all the aspects mentioned above. Therefore, the goal of this framework is not only to focus on retrieving ability estimates by choosing items on maximum information, but also to construct a framework that allows for these other factors to play a role. Plajner and Vomlel (2016) have already applied Bayesian Networks to adaptive testing, selecting items with help of entropy reduction. Almond et al. (2015) provide a reference work on Bayesian Networks in Educational Assessment. Both acknowledge the potential of the method in terms of features such as modularity options to build finer-grained models. IRT does not allow to model sub-skills very easily and to gather information on fine-grained level, due to its dependency on the assumption of generally one underlying trait. The local independence assumption in IRT implies being interested in mainly the student’s overall ability on the subject of interest. When the goal is to improve student’s learning, we are not just interested in efficiently coming to their test score on a global subject. One wants a model that is able to map educational problems and talents in detail over the whole educational program, while allowing for dependency between items. The moment in time can influence topics to be better mastered than others, and this is exactly what we can to get out of a model. The possibility to model flexible structures, estimate abilities on a very detailed level for sub-skills and to easily incorporate other variables such as feedback in Bayesian Networks makes it a very promising method for making adaptive choices in personalized learning. It is shown in this research how item and feedback selection can be performed with help of the promising Bayesian Networks. A student involvement possibility is also introduced and evaluated.

References

Almond, R. G., Mislevy, R. J., Steinberg, L. S., Yan, D., & Williamson, D. M. (2015). Bayesian Networks in Educational Assessment. Test. New York: Springer Science+Business Media. http://doi.org/10.1007/978-0-387-98138-3

Eggen, T.J.H.M. (2012) Computerized Adaptive Testing Item Selection in Computerized Adaptive Learning Systems. In: Eggen. TJHM & Veldkamp, BP.. (Eds). Psychometrics in Practice at RCEC. Enschede: RCEC

Netcoh, S. (2016, March). “What Do You Mean by ‘Personalized Learning?’. Croscutting Conversations in Education – Research, Reflections & Practice. Blogpost.

Plajner, M., & Vomlel, J. (2016). Student Skill Models in Adaptive Testing. In Proceedings of the Eighth International Conference on Probabilistic Graphical Models (pp. 403-414).

Van der Linden, W. J., & Glas, C. A. (2000). Computerized adaptive testing: Theory and practice. Dordrecht: Kluwer Academic Publishers.

Session Video

10afeedback selection10aitem selection10anetwork approach10apersonalized learning1 avan Buuren, Nikky1 aStraat, Hendrik1 aEggen, Theo1 aFox, Jean-Paul uhttp://mail.iacat.org/adaptive-item-and-feedback-selection-personalized-learning-network-approach03162nas a2200181 4500008004100000245010600041210006900147260005500216520249300271653000802764653001502772653002702787100002202814700002602836700001602862700001502878856008702893 2017 eng d00aEfficiency of Targeted Multistage Calibration Designs under Practical Constraints: A Simulation Study0 aEfficiency of Targeted Multistage Calibration Designs under Prac aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Calibration of an item bank for computer adaptive testing requires substantial resources. In this study, we focused on two related research questions. First, we investigated whether the efficiency of item calibration under the Rasch model could be enhanced by calibration designs that optimize the match between item difficulty and student ability (Berger, 1991). Therefore, we introduced targeted multistage calibration designs, a design type that refers to a combination of traditional targeted calibration designs and multistage designs. As such, targeted multistage calibration designs consider ability-related background variables (e.g., grade in school), as well as performance (i.e., outcome of a preceding test stage) for assigning students to suitable items.

Second, we explored how limited a priori knowledge about item difficulty affects the efficiency of both targeted calibration designs and targeted multistage calibration designs. When arranging items within a given calibration design, test developers need to know the item difficulties to locate items optimally within the design. However, usually, no empirical information about item difficulty is available before item calibration. Owing to missing empirical data, test developers might fail to assign all items to the most suitable location within a calibration design.

Both research questions were addressed in a simulation study in which we varied the calibration design, as well as the accuracy of item distribution across the different booklets or modules within each design (i.e., number of misplaced items). The results indicated that targeted multistage calibration designs were more efficient than ordinary targeted designs under optimal conditions. Especially, targeted multistage calibration designs provided more accurate estimates for very easy and 52 IACAT 2017 ABSTRACTS BOOKLET very difficult items. Limited knowledge about item difficulty during test construction impaired the efficiency of all designs. The loss of efficiency was considerably large for one of the two investigated targeted multistage calibration designs, whereas targeted designs were more robust.

References

Berger, M. P. F. (1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement, 15(3), 293–306. doi:10.1177/014662169101500310

Session Video

10aCAT10aEfficiency10aMultistage Calibration1 aBerger, Stephanie1 aVerschoor, Angela, J.1 aEggen, Theo1 aMoser, Urs uhttps://drive.google.com/file/d/1ko2LuiARKqsjL_6aupO4Pj9zgk6p_xhd/view?usp=sharing03184nas a2200157 4500008004100000245010000041210006900141260005500210520260800265653002002873653001602893653001102909100001902920700001602939856007102955 2017 eng d00aThe Implementation of Nationwide High Stakes Computerized (adaptive) Testing in the Netherlands0 aImplementation of Nationwide High Stakes Computerized adaptive T aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

In this presentation the challenges of implementation of (adaptive) digital testing in the Facet system in the Netherlands is discussed. In the Netherlands there is a long tradition of implementing adaptive testing in educational settings. Already since the late nineties of the last century adaptive testing was used mostly in low stakes testing. Several CATs were implemented in student monitoring systems for primary education and in the general subjects language and arithmetic in vocational education. The only nationwide implemented high stakes CAT is the WISCAT-pabo: an arithmetic test for students in the first year of primary school teacher colleges. The psychometric advantages of item based adaptive testing are obvious. For example efficiency and high measurement precision. But there are also some disadvantages such as the impossibility of reviewing items during and after the test. During the test the student is not in control of his own test; e.q . he can only navigate forward to the next item. This is one of the reasons other methods of testing, such as multistage-testing, with adaptivity not on the item level but on subtest level, has become more popular to use in high stakes testing.

A main challenge of computerized (adaptive) testing is the implementation of the item bank and the test workflow in a digital system. Since 2014 a nationwide new digital system (Facet) was introduced in the Netherlands, with connections to the digital systems of different parties based on international standards (LTI and QTI). The first nationwide tests in the Facet-system were flexible exams Dutch and arithmetic for vocational (and secondary) education, taken as item response theory-based equated linear multiple forms tests, which are administered during 5 periods in a year. Nowadays there are some implementations of different methods of (multistage) adaptive testing in the same Facet system (DTT en Acet).

In this conference, other presenters of Cito will elaborate on the psychometric characteristics of this other adaptive testing methods. In this contribution, the system architecture and interoperability of the Facet system will be explained. The emphasis is on the implementation and the problems to be solved by using this digital system in all phases of the (adaptive) testing process: item banking, test construction, designing, publication, test taking, analyzing and reporting to the student. An evaluation of the use of the system will be presented.

Session Video

10aHigh stakes CAT10aNetherlands10aWISCAT1 avan Boxel, Mia1 aEggen, Theo uhttps://drive.google.com/open?id=1Kn1PvgioUYaOJ5pykq-_XWnwDU15rRsf01252nas a2200181 4500008004100000245012100041210006900162260001200231520055600243653003300799653000800832653001900840653003500859100001800894700001600912700002200928856012000950 2011 eng d00aItem Selection Methods based on Multiple Objective Approaches for Classification of Respondents into Multiple Levels0 aItem Selection Methods based on Multiple Objective Approaches fo c10/20113 a

Is it possible to develop new item selection methods which take advantage of the fact that we want to classify into multiple categories? New methods: Taking multiple points on the ability scale into account; Based on multiple objective approaches.

Conclusions

10aadaptive classification test10aCAT10aitem selection10asequential classification test1 aGroen, Maaike1 aEggen, Theo1 aVeldkamp, Bernard uhttp://mail.iacat.org/content/item-selection-methods-based-multiple-objective-approaches-classification-respondents00782nas a2200205 4500008004100000245003400041210003200075260001200107520027300119653000800392653000800400653002300408653001000431653001200441653000800453100002200461700001900483700001600502856005800518 2011 eng d00aA Test Assembly Model for MST0 aTest Assembly Model for MST c10/20113 a

This study is just a short exploration in the matter of optimization of a MST. It is extremely hard or maybe impossible to chart influence of item pool and test specifications on optimization process. Simulations are very helpful in finding an acceptable MST.

10aCAT10amst10amultistage testing10aRasch10arouting10atif1 aVerschoor, Angela1 aRadtke, Ingrid1 aEggen, Theo uhttp://mail.iacat.org/content/test-assembly-model-mst00349nas a2200097 4500008004100000245005100041210005000092300001200142100001600154856008100170 2010 eng d00aThree-Category Adaptive Classification Testing0 aThreeCategory Adaptive Classification Testing a373-3871 aEggen, Theo uhttp://mail.iacat.org/content/three-category-adaptive-classification-testing00567nas a2200109 4500008004100000245010000041210006900141260009700210100001500307700001600322856011900338 2009 eng d00aComputerized classification testing in more than two categories by using stochastic curtailment0 aComputerized classification testing in more than two categories  aD. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.1 aWouda, J T1 aEggen, Theo uhttp://mail.iacat.org/content/computerized-classification-testing-more-two-categories-using-stochastic-curtailment00514nas a2200109 4500008004100000245006900041210006900110260009700179100001500276700001600291856009700307 2009 eng d00aConstrained item selection using a stochastically curtailed SPRT0 aConstrained item selection using a stochastically curtailed SPRT aD. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.1 aWouda, J T1 aEggen, Theo uhttp://mail.iacat.org/content/constrained-item-selection-using-stochastically-curtailed-sprt00461nas a2200097 4500008004100000245006400041210006400105260009700169100001600266856008100282 2007 eng d00aChoices in CAT models in the context of educational testing0 aChoices in CAT models in the context of educational testing aD. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing.1 aEggen, Theo uhttp://mail.iacat.org/content/choices-cat-models-context-educational-testing00437nas a2200097 4500008004100000245006500041210006500106260007000171100001600241856008200257 2007 eng d00aChoices in CAT models in the context of educattional testing0 aChoices in CAT models in the context of educattional testing aSt. Paul, MNbGraduate Management Admission CouncilcJune 7, 20071 aEggen, Theo uhttp://mail.iacat.org/content/choices-cat-models-context-educattional-testing00560nas a2200121 4500008004100000245011100041210006900152260003400221100001800255700002500273700001600298856012400314 2007 eng d00aA multiple objective test assembly approach for exposure control problems in computerized adaptive testing0 amultiple objective test assembly approach for exposure control p aArnhem, The NetherlandsbCito1 aVeldkamp, B P1 aVerschoor, Angela, J1 aEggen, Theo uhttp://mail.iacat.org/content/multiple-objective-test-assembly-approach-exposure-control-problems-computerized-adaptive01622nas a2200217 4500008004100000020002200041245008200063210006900145260002600214300001200240490000700252520088700259653002801146653002501174653002501199653001901224653001601243100001601259700002501275856010401300 2006 eng d a0146-6216 (Print)00aOptimal testing with easy or difficult items in computerized adaptive testing0 aOptimal testing with easy or difficult items in computerized ada bSage Publications: US a379-3930 v303 aComputerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, it is shown that maximum information item selection in CATs using an item bank that is calibrated with the one- or the two-parameter logistic model results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the one-parameter logistic model and not with the two-parameter logistic model. An alternative selection procedure, based on maximum information at a shifted ability level, gives satisfactory results with both models. (PsycINFO Database Record (c) 2007 APA, all rights reserved)10acomputer adaptive tests10aindividualized tests10aItem Response Theory10aitem selection10aMeasurement1 aEggen, Theo1 aVerschoor, Angela, J uhttp://mail.iacat.org/content/optimal-testing-easy-or-difficult-items-computerized-adaptive-testing01352nas a2200133 4500008003900000245008200039210006900121300001200190490000700202520091500209100001601124700002501140856005301165 2006 d00aOptimal Testing With Easy or Difficult Items in Computerized Adaptive Testing0 aOptimal Testing With Easy or Difficult Items in Computerized Ada a379-3930 v303 a

Computerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, it is shown that maximum information item selection in CATs using an item bank that is calibrated with the one or the two-parameter logistic model results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the one-parameter logistic model and not with the two-parameter logistic model. An alternative selection procedure, based on maximum information at a shifted ability level, gives satisfactory results with both models. Index terms: computerized adaptive testing, item selection, item response theory

1 aEggen, Theo1 aVerschoor, Angela, J uhttp://apm.sagepub.com/content/30/5/379.abstract00437nam a2200097 4500008003900000245007800039210006900117260003900186100001600225856009800241 2004 d00aContributions to the theory and practice of computerized adaptive testing0 aContributions to the theory and practice of computerized adaptiv aArnhem, The Netherlands: Citogroep1 aEggen, Theo uhttp://mail.iacat.org/content/contributions-theory-and-practice-computerized-adaptive-testing00543nas a2200109 4500008004100000245012100041210006900162260004000231100001600271700001900287856012700306 2004 eng d00aOptimal testing with easy items in computerized adaptive testing (Measurement and Research Department Report 2004-2)0 aOptimal testing with easy items in computerized adaptive testing aArnhem, The Netherlands: Cito Group1 aEggen, Theo1 aVerschoor, A J uhttp://mail.iacat.org/content/optimal-testing-easy-items-computerized-adaptive-testing-measurement-and-research-department00431nas a2200109 4500008004100000245006900041210006900110260001800179100001600197700001700213856009100230 2003 eng d00aOptimal testing with easy items in computerized adaptive testing0 aOptimal testing with easy items in computerized adaptive testing aManchester UK1 aEggen, Theo1 aVerschoor, A uhttp://mail.iacat.org/content/optimal-testing-easy-items-computerized-adaptive-testing00548nas a2200145 4500008004100000245009500041210006900136300001200205490000700217100001400224700001600238700001600254700001700270856011500287 2002 eng d00aEvaluation of selection procedures for computerized adaptive testing with polytomous items0 aEvaluation of selection procedures for computerized adaptive tes a393-4110 v261 aRijn, P W1 aEggen, Theo1 aHemker, B T1 aSanders, P F uhttp://mail.iacat.org/content/evaluation-selection-procedures-computerized-adaptive-testing-polytomous-items-001311nas a2200169 4500008004100000245009500041210006900136300001200205490000700217520070700224653003400931100001400965700001600979700001600995700001701011856011301028 2002 eng d00aEvaluation of selection procedures for computerized adaptive testing with polytomous items0 aEvaluation of selection procedures for computerized adaptive tes a393-4110 v263 aIn the present study, a procedure that has been used to select dichotomous items in computerized adaptive testing was applied to polytomous items. This procedure was designed to select the item with maximum weighted information. In a simulation study, the item information function was integrated over a fixed interval of ability values and the item with the maximum area was selected. This maximum interval information item selection procedure was compared to a maximum point information item selection procedure. Substantial differences between the two item selection procedures were not found when computerized adaptive tests were evaluated on bias and the root mean square of the ability estimate. 10acomputerized adaptive testing1 aRijn, P W1 aEggen, Theo1 aHemker, B T1 aSanders, P F uhttp://mail.iacat.org/content/evaluation-selection-procedures-computerized-adaptive-testing-polytomous-items00520nas a2200097 4500008004100000245013000041210006900171260004000240100001600280856012600296 2001 eng d00aOverexposure and underexposure of items in computerized adaptive testing (Measurement and Research Department Reports 2001-1)0 aOverexposure and underexposure of items in computerized adaptive aArnhem, The Netherlands: CITO Groep1 aEggen, Theo uhttp://mail.iacat.org/content/overexposure-and-underexposure-items-computerized-adaptive-testing-measurement-and-research01599nas a2200157 4500008004100000245008200041210006900123300001100192490000700203520101400210653003401224653004001258100001601298700002401314856010301338 2000 eng d00aComputerized adaptive testing for classifying examinees into three categories0 aComputerized adaptive testing for classifying examinees into thr a713-340 v603 aThe objective of this study was to explore the possibilities for using computerized adaptive testing in situations in which examinees are to be classified into one of three categories.Testing algorithms with two different statistical computation procedures are described and evaluated. The first computation procedure is based on statistical testing and the other on statistical estimation. Item selection methods based on maximum information (MI) considering content and exposure control are considered. The measurement quality of the proposed testing algorithms is reported. The results of the study are that a reduction of at least 22% in the mean number of items can be expected in a computerized adaptive test (CAT) compared to an existing paper-and-pencil placement test. Furthermore, statistical testing is a promising alternative to statistical estimation. Finally, it is concluded that imposing constraints on the MI selection strategy does not negatively affect the quality of the testing algorithms10acomputerized adaptive testing10aComputerized classification testing1 aEggen, Theo1 aStraetmans, G J J M uhttp://mail.iacat.org/content/computerized-adaptive-testing-classifying-examinees-three-categories00595nas a2200133 4500008004100000245013300041210006900174260003400243100000900277700001600286700001600302700001700318856012600335 2000 eng d00aA selection procedure for polytomous items in computerized adaptive testing (Measurement and Research Department Reports 2000-5)0 aselection procedure for polytomous items in computerized adaptiv aArnhem, The Netherlands: Cito1 aRijn1 aEggen, Theo1 aHemker, B T1 aSanders, P F uhttp://mail.iacat.org/content/selection-procedure-polytomous-items-computerized-adaptive-testing-measurement-and-research00437nas a2200109 4500008004100000245008200041210006900123300001200192490000700204100001600211856010000227 1999 eng d00aItem selection in adaptive testing with the sequential probability ratio test0 aItem selection in adaptive testing with the sequential probabili a249-2610 v231 aEggen, Theo uhttp://mail.iacat.org/content/item-selection-adaptive-testing-sequential-probability-ratio-test00880nas a2200133 4500008004100000245006300041210006200104300001000166490000700176520043400183100002400617700001600641856008900657 1998 eng d00aComputerized adaptive testing: What it is and how it works0 aComputerized adaptive testing What it is and how it works a45-520 v383 aDescribes the workings of computerized adaptive testing (CAT). Focuses on the key concept of information and then discusses two important components of a CAT system: the calibrated item bank and the testing algorithm. Describes a CAT that was designed for making placement decisions on the basis of two typical test administrations and notes the most significant differences between traditional paper-based testing and CAT. (AEF)1 aStraetmans, G J J M1 aEggen, Theo uhttp://mail.iacat.org/content/computerized-adaptive-testing-what-it-and-how-it-works00517nas a2200097 4500008004100000245013300041210006900174260003500243100001600278856012500294 1998 eng d00aItem selection in adaptive testing with the sequential probability ratio test (Measurement and Research Department Report, 98-1)0 aItem selection in adaptive testing with the sequential probabili aArnhem, The Netherlands: Cito.1 aEggen, Theo uhttp://mail.iacat.org/content/item-selection-adaptive-testing-sequential-probability-ratio-test-measurement-and-research00542nas a2200109 4500008004100000245012900041210006900170260003400239100001600273700002400289856011900313 1996 eng d00aComputerized adaptive testing for classifying examinees into three categories (Measurement and Research Department Rep 96-3)0 aComputerized adaptive testing for classifying examinees into thr aArnhem, The Netherlands: Cito1 aEggen, Theo1 aStraetmans, G J J M uhttp://mail.iacat.org/content/computerized-adaptive-testing-classifying-examinees-three-categories-measurement-and