DFIT8 for Windows User’s Manual Differential Functioning of Items and Tests T. C. Oshima, Steve Kushubar, John C. Scott, and Nambury S. Raju DFIT8 for Windows – User’s Manual System Requirements Microsoft Windows 2000/XP/Vista 7 Mb free disk space 2 Mb free RAM For Further Information Assessment Systems Corporation 2233 University Avenue, Suite 200 St. Paul, Minnesota 55114, U.S.A. Voice: 651/647-9220 Fax: 651/647-0412 E-Mail: support@assess.com World Wide Web page: www.assess.com User’s Manual for DFIT8: Differential Functioning of Items and Tests First Edition, Windows Version (DFIT8.04) DFIT8 for Windows – User’s Manual Your DFIT8 License Unless you have purchased multiple licenses for DFIT8, your license is a single-user license. You may install DFIT8 on up to two computers (e.g., a desktop and a laptop) so long as there is no possibility that the two copies will be used simultaneously. Technical Assistance If you need technical assistance using DFIT8, please visit the Support section of our Web site, www.assess.com. If the answer to your question is not posted, please email us at support@assess.com. Technical assistance for DFIT8 is provided for one year from the date you purchase your license. Please provide us with the invoice number for your license purchase when you request technical assistance. Citation Oshima, T. C., Kushubar, S., Scott, J.C. & Raju N.S. (2009). DFIT8 for Window User’s Manual: Differential functioning of items and tests. St. Paul MN: Assessment Systems Corporation. DFIT8 for Windows – User’s Manual Contents 1 Introduction .............................................................................................................1 Item Response Theory Models ................................................................................................................. 1 Differential Functioning of Items and Tests ............................................................................................. 3 The DFIT Framework for DIF and DTF ................................................................................................... 4 Significance Tests in DFIT ....................................................................................................................... 6 References ................................................................................................................................................. 7 2 Formatting Input Data for Analysis ........................................................................8 Dichotomous IRT Calibration................................................................................................................... 8 Polytomous IRT Calibration ..................................................................................................................... 8 Linking ...................................................................................................................................................... 9 References ............................................................................................................................................... 10 3 Definition File .......................................................................................................11 Running DFIT ......................................................................................................................................... 11 Dichotomous Run Definition File ........................................................................................................... 11 Polytomous Graded Response Model Definition File ............................................................................ 14 4 Output File ............................................................................................................17 Viewing Output....................................................................................................................................... 17 Content of the Output File ...................................................................................................................... 17 Appendix A. Examples ............................................................................................18 1. A dichotomous example (Example 1)................................................................................................. 18 2. A polytomous example (Example 2) .................................................................................................. 25 Appendix B. Troubleshooting ..................................................................................33 DFIT8 for Windows – User’s Manual 1 Introduction Nambury S. Raju (1937 – 2005) developed a framework for assessing the differential functioning of items and tests (DFIT) based on item response theory (IRT). In DFIT, differential item functioning (DIF) and differential test functioning (DTF) are assessed by using item and score parameter estimates obtained from an IRT calibration. The framework can be applied to test data that are either dichotomous (Raju, van der Linden, & Fleer, 1995) or polytomous (Flowers, Oshima, & Raju, 1999). Although both unidimensional and multidimensional DFIT models have been developed (Oshima, Raju, & Flowers, 1997), this program implements only the unidimensional models. Item Response Theory Models For dichotomously scored items, the three-parameter logistic model describes the probability of an individual answering item i correctly as a function of ability (θ) as follows. Pi (θ s ) = ci + (1 − ci ) exp Dai (θ s − bi ) 1 + exp Dai (θ s − bi ) . (1) The item characteristics are represented by the ai, bi, and ci parameters. The location parameter, bi reflects the difficulty of the item. The discrimination parameter, ai, relates to the steepness of the curve. The ci parameter reflects the lower asymptote, or the probability that a person with extremely low θ would correctly answer the item. D is a scaling constant typically set at 1.702. Polytomous IRT models require the estimation of multiple item response functions (IRF) representing the different response categories. Although various polytomous models have been developed in the IRT literature, the current version of DFIT8 uses Samejima’s (1969) graded Page 1 DFIT8 for Windows – User’s Manual response model (GRM) which is designed for ordered response categories. Theoretically, however, DFIT can be applied to different types of polytomous models. According to GRM, for an item with m response categories, there will be m-1 boundary response functions (BRF). A BRF represents the probability of person s responding above response category k on item i, Pik* (θ s ) = exp Dai (θ s − bik ) 1 + exp Dai (θ s − bik ) , (2) where bik is a location parameter that designates the boundary between response categories k and k+1, and ai is the item discrimination parameter. The probability of responding in a particular response category can be computed from the difference between adjacent BRFs. This function is referred to as the category response function (CRF): Pik (θ ) = Pi*( k −1) (θ ) − Pik* (θ ) . (3) Because the first and last response categories lack an adjacent boundary, Samejima (1969) defined Pi*0 (θ ) = 1, and Pim* (θ ) = 0. There will be as many CRFs for an item as there are response categories. The expected score of individual s on item i, ESsi(θs), can be defined as a weighted average of the category values, where the weights reflect the probability of the individual selecting each category (i.e, the CRFs), m ES i (θ s ) = ∑ Pik (θ s ) X ik , (4) k =1 where Xik is the value assigned to category k on item i. For a dichotomously scored item, the expected score is equal to the probability of answering an item correctly (Equation 1). Page 2 DFIT8 for Windows – User’s Manual The total score on a test can be defined as the sum of the scores on the individual items. This total test score can also be modeled as a function of θ and the resulting curve is called the test response function (TRF). The TRF is defined as sum of expected score functions across n items, n T (θ s ) = ∑ ES i (θ s ) . (5) i =1 Differential Functioning of Items and Tests The DFIT analysis starts with two sets of item parameter estimates (one from a reference group and another from a focal group) which are placed on the scale of the focal group along with the focal group’s θ estimates. According to IRT, the item response functions (IRFs; also called item characteristic curves or item characteristic functions) are invariant over subgroups of examinees. Therefore, DIF is conceptualized by measuring the differences between the two IRFs . Figure 1 graphically depicts the differences between a reference group IRF and a focal group IRF in the dichotomous model, by plotting probability (Y axis) against θ (X axis). The curve plotted is the IRF. According to IRT, the two curves should be invariant after the item parameter estimates from each group are placed on the common scale. Thus, any gap between the two IRFs indicates DIF. The gap can be measured in various ways. For example Raju (1988) developed a DIF index based on the area of the gap. In the DFIT framework, on the other hand, the gap is measured by the average squared distance of the probability difference at θ levels based on the focal group. This approach, compared to the area approach, offers substantial advantages as it can be easily expanded to different models (unidimensional or multidimensional models) in various scoring formats (dichotomous or polytomous data). The DFIT approach also makes several applications possible that are useful in DIF research (e.g., DTF, differential bundle functioning (Oshima, Raju, Flower, & Slinde, 1998)). Page 3 DFIT8 for Windows – User’s Manual Figure 1. Two IRFs/ICFs The DFIT Framework for DIF and DTF The DFIT framework offers two types of DIF, non-compensatory DIF (NCDIF) and compensatory DIF (CDIF). NCDIF is defined as 2 NCDIFi = EF di (θ ) , (6) where d i (θ ) = PiF (θ ) − PiR (θ ) (7) for the dichotomous model (see Figure 1), and d i (θ ) = ES iF (θ ) − ES iR (θ ) (8) for the polytomous model. EF denotes the expectation taken over the θ distribution from the focal group. Page 4 DFIT8 for Windows – User’s Manual DTF is similarly defined but at the test level. As shown in Figure 2, the two curves compared are the TRFs and the distance is defined as, D(θ ) = TF (θ ) − TR (θ ) . (9) DTF is then defined as the expected value of the squared difference between focal and reference groups, where the expectation is taken across the θ distribution from the focal group, 2 DTF = EF D (θ ) . (10) Figure 2. Two TRFs/TCFs Despite the mathematical similarity between NCDIF and DTF, the relationship between the two is not straightforward as NCDIF does not take the item covariances into account. CDIF, on the other hand, has a straightforward relationship with DTF, n DTF = ∑ CDIFi . (11) i =1 CDIF is defined as CDIFi = E F (d i D ) = Cov(d i , D ) + µ di µ D , (12) where Cov stands for covariance. Page 5 DFIT8 for Windows – User’s Manual NCDIF and CDIF both relate to DIF, but address different issues. Like most item-level DIF indices, NCDIF assumes that all items other than the studied item are DIF free, and can be used like any other DIF indices. CDIF, on the other hand, takes into account the pattern of DIF across items, and can be used to investigate the effect of removing the DIF items on DTF. Significance Tests in DFIT DFIT8 employs a recently developed significance test for NCDIF and DTF. The test is called the item parameter replication (IPR) method (Oshima, Raju & Nanda, 2006; Raju, Fortmann, Kim, Morris, Nering, & Oshima, in press) and provides a means of deriving cutoff values that are tailored to a particular data set. The IPR method begins with estimates of item parameters for the focal group and the sampling variances and covariances of these estimates. Based on these initial estimates, a large number of replications (typically 1,000 pairs) of item parameters are simulated. Then the cutoff value for each alpha level is determined by the empirical sampling distribution of NCDIF/DTF obtained under the null hypothesis that focal and reference groups have identical parameters. The IPR method produces cutoff values for each item. The significance of CDIF is not tested directly. Instead, items with large CDIF are removed one by one until DTF reaches non-significance by a chi-square test. Those removed CDIF items are then considered significant. Page 6 DFIT8 for Windows – User’s Manual References More detailed information on DFIT can be found in the following references. Researchers new to DFIT might find the instructional module developed for the National Council on Measurement in Education (Oshima & Morris, 2008) useful. Flowers, C. P., Oshima, T. C., & Raju, N. S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326. Oshima, T. C., & Morris, S. B. (2008). An NCME Instructional Module on Raju’s Differential Functioning of Items and Tests (DFIT). Educational Measurement: Issues and Practice, 27, 4350. Oshima, T. C., Raju, N. S., & Flowers, C. P. (1997). Development and demonstration of multidimensional IRT-based internal measures of differential functioning of items and tests. Journal of Educational Measurement, 34, 253-272. Oshima, T. C., Raju, N. S., Flowers, C. P., & Slinde, J. (1998). Differential bundle functioning (DBF) using the DFIT framework: Procedures for identifying possible sources of differential functioning. Applied Measurement in Education, 11, 353-369. Oshima, T. C., Raju, N. S., & Nanda (2006). A new method for assessing the statistical significance in the differential functioning of items and tests (DFIT) framework. Journal of Educational Measurement, 43, 1-17. Raju, N. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502. Raju, N. S., K. A. Fortmann, Kim, W., Morris, S. B., Nering, M., & Oshima, T. C. (in press). The item parameter replication method for detecting differential functioning in the DFIT framework. Applied Psychological Measurement. Raju, N. S., van der Linden, W. J., & Fleer, P. F. (1995). An IRT-based internal measure of test bias with applications for differential item functioning. Applied Psychological Measurement, 19, 353-368. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, No. 17. Page 7 DFIT8 for Windows – User’s Manual 2 Formatting Input Data for Analysis Dichotomous IRT Calibration Any dichotomous IRT calibration program can be used to obtain item parameter estimates, their variance and covariance for each group (focal or reference). BILOG-MG3 (Zimowski, Muraki, Mislevy, & Bock, 2002), for example, provides an estimated covariance file (*.cov) which contains item parameter estimates, error variances and covariances. The *.cov file needs to be rearranged to be readily read into DFIT8. The newly created file should contain the following information in the specific order for each item for the focal group: b, a, c, V(b), V(a), V(c), Cov(b,a), Cov(b,c) and Cov(a,c), where V is the variance and Cov is the covariance. BILOGMG3 also provides the ability file (*.sco) which contains the θ estimates. DFIT depends on the accurate estimation of item parameters, and accurate parameter estimation typically requires large samples of examinees. Therefore, DFIT is not recommended for application with small samples. For dichotomous IRT models, the required sample size for each group will depend on the number of parameters estimated: Conservative sample sizes (N) are N > 200 for the oneparameter model, N > 500 for the two-parameter model, and N > 1,000 for the three-parameter model (Crocker & Algina, 1986). Polytomous IRT Calibration Any polytomous IRT calibration program can be used to obtain item parameter and variance estimates. The covariance information may not be readily available, however. For example, PARSCALE (Muraki & Bock, 2003) does not provide the covariance information. A Fortran program (Polycov; Morris 2007) is available to calculate the item variances and covariances from Parscale output and to create the input data file for DFIT8. It is not recommended to assume that the covariances are equal to 0. The input data file to DFIT8 has to provide focal group information in a specific order. Using a 5-category item as an example, it will be a, b1, b2, Page 8 DFIT8 for Windows – User’s Manual b3, b4, V(a), V(b1), V(b2), V(b3), V(b4), Cov(a,b1), Cov(a,b2), Cov(a,b3), Cov(a,b4), Cov(b1,b2), Cov(b1,b3), Cov(b1,b4), Cov(b2,b3), Cov(b2,b4), and Cov(b3,b4) in that order for each item for the focal group. As with the dichotomous case, large sample sizes are required for polytomous DFIT. A sample size recommendation for accurate estimation for the polytomous models should be followed (e.g., N > 500 is recommended by Reise & Yu, 1990). Linking In the DFIT framework, the item parameter estimates from the reference group are placed on the scale of the focal group. Linking coefficients (a multiplicative coefficient and an additive coefficient) can be obtained by using linking programs such as IPLINK (Lee & Oshima, 1996) or PIE for dichotomous models, IRTEQ (Han, 2007) for either dichotomous or polytomous models, or POLYEQUATE for polytomous models. Iterative linking, or two-stage linking is recommended. Iterative linking is a process by which only items that are considered to be DIFfree are used as linking items. The steps for conducting two-stage linking are as follows: 1. Obtain linking coefficients using all items (the first stage), 2. Run DFIT to identify large DIF items, 3. Remove those large DIF items from the test, 4. Obtain revised linking coefficients (the second stage), and finally 5. Run DFIT for all items again using those revised linking coefficients. Page 9 DFIT8 for Windows – User’s Manual References Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston. Han, K. T. (2007). IRTEQ: Windows application that implements IRT scaling and equating [computer program]. Amherst, MA: University of Massachusetts Amherst, Center for Educational Assessment. Available for download at http://www.umass.edu/remp/software/irteq/ Lee, K., & Oshima, T. C. (1996). IPLINK: Multidimensional and unidimensional item parameter linking in item response theory. Applied Psychological Measurement, 20, 230. Available for download at http://education.gsu.edu/eps/4493.html Morris, B. S. (2008). Polycov [Computer program]. Chicago: Illinois Institute of Technology. Available for download at http://www.iit.edu/~morris/polycov/polycov.html Muraki, E., & Bock, R. D. (2003). PARSCALE 4: IRT Item Analysis and Test Scoring for Rating-Scale Data. Scientific Software: Chicago, IL. Reise, S. P. & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144. Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (2002). BILOG-MG3 [Computer software]. St.Paul, MN: Assessment Systems Corporation. Page 10 DFIT8 for Windows – User’s Manual 3 Definition File Running DFIT In order to perform a DFIT analysis, the user must create a DFIT run definition file (*.dft). The run definition file provides all the information necessary to read the necessary data files, perform the analysis and generate the output file. Definition files are created by using the File New menu option. The user can create the definition file by using the Normal View or Raw View (choices under the View pull-down menu). In the Normal View, the Windows frames assist the user to fill out the necessary information. In the Raw View, the user can type the syntax directly in text mode. After the definition file is saved, the user can run the program by clicking Run in the pull-down menu. Dichotomous Run Definition File Here is a sample dichotomous run definition file: Each line in the dichotomous run definition file is described below: Page 11 DFIT8 for Windows – User’s Manual Line 1 Any title up to 80 characters long. Line 2 The first number represents the number of items (I5 format – i.e., an integer field five characters wide). The maximum number of Items allowed is 100. The second number represents the IRT model. This number should be 1, 2, or 3 (I5 format). The third number represents the number of examinees or subjects in the focal group. The maximum number of examinees allowed is 5,000 (I5 format). The fourth number is the number of replications for generating (using the IPR method) itemlevel and test/scale-level cut-off scores for assessing the significance of NCDIF and DTF indices (I5 format). The maximum is 99999. The fifth number is the number for generating random item parameters in the Monte Carlo procedure and it MUST be a negative integer (I5 format). If the Windows interface is used to make the present file, the number for generating random item parameters is created by the program. The sixth number is D (F8.4 format. i.e., a 8-character field with a decimal point and four significant digits after the decimal point), the IRT constant (see Equations 1 and 2). It must be set 1.0 if the IRT model is logistic or 1.7 if the IRT model is normal ogive. The seventh and eighth numbers are the multiplicative and additive constants (F8.4 format for both). These are used for placing the reference group item parameters on the same metric as the focal group item parameters. In the case of the 1-paramter logistic or Rasch Model, the multiplicative constant must be set to 1.0. If the focal and reference group parameters are already on a common metric, the multiplicative and additive constants must be 1.0 and 0.0 respectively. Page 12 DFIT8 for Windows – User’s Manual Line 3 A ‘1’ tells the program to include the item in the DIF/DTF analysis. A ‘0’ excludes the item from the analysis. For example, if there are 30 items in a test and you want to exclude Item 5 from the DIF/DTF analysis, then Line 3 should be as follows: 111101111111111111111111111111 If you want to include all 30 items in the DIF/DTF analysis, Line 3 should read as follows: 111111111111111111111111111111 Line 4 The format for reading the item parameters and the variance-covariance matrix as a vector [b, a, c, V(b), V(a), V(c), Cov(b,a), Cov(b,c) and Cov(a,c) in that order] for each item in the focal group. The specific format could vary from one application to the next. In the example above, 9F8.4 means nine fields of 8 characters each, including a decimal point and four significant digits after the decimal point. Formats must be enclosed in parentheses. Line 5 The format for reading item parameters as a vector (b, a, and c in that order) for each item in the reference group. Information about the variances and covariances is not needed for the reference group. The specific item format could vary from one application to the next. Line 6 The format for reading the θ values for the focal group. The specific θ format could vary from one application to the next. In the example above. 12x means skip the first 12 columns and begin reading in column 13. Line 7 The file path to the focal group item data file. If no path is specified (i.e., just a file name, as in the above example) , the program will look for the files in the same folder in which the program exists Page 13 DFIT8 for Windows – User’s Manual Line 8 The file path to the reference group item data file. Line 9 The file path to the θ values file. Line 10 The file path to the output file. Polytomous Graded Response Model Definition File Here is a sample polytomous run definition file: Line 1 Any title up to 80 characters long Line 2 The first number represents the number of items (I5 format). The maximum number of items allowed is 100. The second number represents the number of examinees or subjects in the focal group. The maximum number of examinees allowed is 5,000 (I5 format). Page 14 DFIT8 for Windows – User’s Manual The third number is the number of replications for generating (via Monte Carlo procedures) item-level and test/scale-level cut-off scores for assessing the significance of NCDIF and DTF indices (I5 format). The fourth number is the number for generating random item parameters in the Monte Carlo procedure and it MUST be a negative integer (I5 format). If the Windows interface is used to make the present file, the number for generating random item parameters is created by the program. The fifth number is D (F8.4 format). It must be set to 1.0 if the IRT model is logistic or 1.7 if the IRT model is normal ogive. The sixth and seventh numbers are the multiplicative and additive constants (F8.4 format for both). These are used for placing the reference group item parameters on the same scale as the focal group item parameters. In the case of one-paramter logistic or Rasch model, the multiplicative constant must be set to 1.0. If the focal and reference group parameters are already on a common metric, the multiplicative and additive constants must be 1.0 and 0.0, respectively. Line 3 A ‘1’ tells the program to include the item in the DIF/DTF analysis. A ‘0’ excludes the item from the analysis. For example, if there 25 items in a test and you want to exclude Item 5 from the DIF/DTF analysis, then Line 3 should be as follows: 1111011111111111111111111 If all 25 items are to be included in the DIF/DTF analysis, Line 3 should read as follows: 1111111111111111111111111 Line 4 Each number represents the number of categories for a given item. The maximum number of categories allowed is 9. If you have a scale with 25 items with 5 categories per item, Line 4 should read: 5555555555555555555555555 Page 15 DFIT8 for Windows – User’s Manual Line 5 The format for reading item parameters and the variance-covariance matrix as a vector [For a 5category item, it will be a, b1, b2, b3, b4, V(a), V(b1), V(b2), V(b3), V(b4), Cov(a,b1), Cov(a,b2), Cov(a,b3), Cov(a,b4), Cov(b1,b2), Cov(b1,b3), Cov(b1,b4), Cov(b2,b3), Cov(b2,b4), and Cov(b3,b4) in that order] for each item in the focal group. The specific format could vary from one application to the next. Line 6 The format for reading item parameters and their variances as a vector [for an item with 5 categories: a, b1, b2, b3, b4, V(a), V(b1), V(b2), V(b3) and V(b4) in that order] for each item in the Reference Group. The specific format could vary from one application to the next. Line 7 The format for reading the θ values for the focal group. The specific θ format could vary from one application to the next. Line 8 The file path to the focal group item data file. Line 9 The file path to the reference group item data file. Line 10 The file path to the θ values file. Line 11 The file path to the output file. Line 12 and beyond: Each row indicates the initial score values associated with the categories for a given item. For example, if you have 25 items in a scale, you need 25 lines (Lines 12-36). Page 16 DFIT8 for Windows – User’s Manual 4 Output File Viewing Output The output file is created after a successful run of the definition file and presented on the screen. It can be also viewed later using File Open in the program, or using any other programs that display text data. Content of the Output File Items 1 – 5 listed below are input information to verify that the input files are read into DFIT8 correctly. Items 6 – 10 present the results from the DFIT analysis. See Appendix A for examples. 1. Title 2. Input variables 3. Focal group item parameters 4. Reference group (unequated) item parameters 5. Reference group (equated) item parameters 6. Monte Carlo generated item-level and test-level cut-offs (from the IPR method). Cutoff values for alpha = .001, .005, .01, 05, .10, and .50 are provided. DTF cutoff values are listed at the bottom. 7. Main output where CDIF, NCDIF, and NCDIF significance (ns indicates not significant) are reported. DIF category is for future use and it is blank. 8. Summary statistics for true scores 9. DTF related statistics, including the chi-square test 10. CDIF items to be removed to achieve non-significant DTF. Page 17 DFIT8 for Windows – User’s Manual Appendix A. Examples 1. A dichotomous example (Example 1) This is a 40-item simulated dichotomous test with 1,000 examinees in each group (reference and focal). DIF was embedded in Items 5, 10, 15, and 20. Items were calibrated separately for each group using BILOG-MG3 (2-parameter model). The item parameter estimates and their variance covariance estimates are listed in EX1R.PAR and EX1F.PAR for the reference group and focal group, respectively. In each file, there are five columns (b, a, V(b), V(a), and Cov(b,a) in that order). The linking coefficients were calculated using IPLINK (1.0240 and −0.0180 for the multiplicative and additive coefficients, respectively, after the second-stage linking). In the score file (EX1F.THT), the third column contains the θ estimates. For the IPR method, 1,000 replications were used to conduct the significance test. Definition File in the Normal View Page 18 DFIT8 for Windows – User’s Manual Definition File in the Raw View Page 19 DFIT8 for Windows – User’s Manual EX1F.PAR EX1R.PAR EX1F.THT Page 20 DFIT8 for Windows – User’s Manual EX1.TXT (Selected output) 1. Title 2. Input variables 3. Focal group item parameters 4. Reference group (unequated) item parameters Page 21 DFIT8 for Windows – User’s Manual 5. Reference group (equated) item parameters 6. Monte Carlo generated item-level and test-level cut-offs (the IPR method) Page 22 DFIT8 for Windows – User’s Manual 7. Main output where CDIF, NCDIF, and NCDIF significance (ns indicates not-significant) are reported. DIF category is for future use and it is left blank. 8. Summary statistics for true scores Page 23 DFIT8 for Windows – User’s Manual 9. DTF related statistics including the chi-square test 10. CDIF items to be removed to achieve non-significant DTF. Results indicate that Items 5, 10, 15, and 20 show NCDIF at the .001 level. Item 13 shows NCDIF at the .05 level. The DTF of .51 is significant at the .001 level. After removing two significant CDIF items (Items 15 and 5), DTF is no longer significant. Page 24 DFIT8 for Windows – User’s Manual 2. A polytomous example (Example 2) This is a 20-item simulated polytomous test (5-category responses scored 0, 1, 2, 3, 4) with 1,000 examinees in each group (reference and focal). DIF was embedded in Items 3, 8, 13, and 18. Items were calibrated separately for each group using PARSCALE (Muraki & Bock, 2003). The item parameter estimates and their variance covariance estimates are listed in EX2R.PAR and EX2F.PAR for the reference group and focal group, respectively. Those *.PAR files were created by POLYCOV which read the PARSCALE parameter files, added covariances, and arranged the information as required for DFIT input. In each file, there are 20 variables [a, b1, b2, b3, b4, V(a), V(b1), V(b2), V(b3), V(b4), Cov(a,b1), Cov(a,b2), Cov(a,b3), Cov(a,b4), Cov(b1,b2), Cov(b1,b3), Cov(b1,b4), Cov(b2,b3), Cov(b2,b4), and Cov(b3,b4) in that order]. The linking coefficients were calculated using EQUATE (1.0335 and 0.0492 for the multiplicative and additive coefficients, respectively, after the second-stage linking). In the score file (EX2F.SCO), the θ can be found at (/58x,F10.4). For the IPR method, 1000 replications were used to conduct the significance test. Definition File in the Normal View Page 25 DFIT8 for Windows – User’s Manual Page 26 DFIT8 for Windows – User’s Manual Definition File in the Raw View Page 27 DFIT8 for Windows – User’s Manual EX2F.PAR EX2R.PAR EX2F.SCO Page 28 DFIT8 for Windows – User’s Manual EX2.TXT (Selected output) 1. Title 2. Input variables 3. Focal group item parameters Page 29 DFIT8 for Windows – User’s Manual 4. Reference group (unequated) item parameters 5. Reference group (equated) item parameters Page 30 DFIT8 for Windows – User’s Manual 6. Monte Carlo generated item-level and test-level cut-offs (the IPR method) 7. Main output where CDIF, NCDIF, and NCDIF significance (ns indicates not-significant) are reported. DIF category is for future use and it is left blank. Page 31 DFIT8 for Windows – User’s Manual 8. Summary statistics for true scores 9. DTF related statistics including the chi-square test 10. CDF items to be removed to achieve non-significant DTF. Results indicate that Items 3, 8, 13, and 15 show NCDIF at the .001 level. Item 14 shows NCDIF at the .01 level. The DTF of .92 is significant at the .001 level. After removing two significant CDIF items (Items 3 and 8), DTF is no longer significant. Page 32 DFIT8 for Windows – User’s Manual Appendix B. Troubleshooting Most problems that occur when running DFIT are a result of incorrect or poorly formatted data. If you open the file in Normal View, when you start the run DFIT will perform validation of the definition file. The most likely error is an end of file error. The figure below shows a typical end of file error. This can be caused by several different errors in the data files: 1. Insufficient data in the data file 2. Incorrect format specification in the run definition file 3. Incorrect number of items or examinees specified on line 2 of the run definition file Page 33

Download PDF

- Similar pages
- external card, Please boot
- Mixed-Format Tests: Psychometric Properties with a Primary Focus on Equating (Volume 1)
- Mixed-Format Tests: Psychometric Properties with a Primary Focus on Equating (Volume 2)
- Differential item functioning analysis system User`s Manual
- Assessing Equating Results Based on First-order and Second
- Wave® Antenna User Guide - NeWave Sensor Solutions
- CATSIM Manual - International Association for Computerized
- Differential item functioning analysis system User`s Manual
- BuySpeed Seller Manual/Seller Administrator Manual
- AvSAP Manual
- null 10602973
- Manual
- 2010, odyssée des modèles de classification diagnostique (MCD)
- Manual de Usuario PARTES DE AVERIA